JSON API

[gsnw-sebast]

Introduction

The new JSON API for fping will be discussed in this section.

There are currently 3 versions, which are partly based on each other

• JSON output support with --json/-J option #160
• New option -j / --json for the JSON output in the functions print_per_system_stats() and print_global_stats() #379
• Issue/164-alternate - JSON functionality #380

The best version is #380 on which the discussion should be based.

Topics

@schweikert

1. We might want to differentiate between output lines that contain summaries and those that contain details.
2. The output should be readable but as compact as possible (this is probably already fine)

More content to follow

[schweikert]

I suggest the following:

• The JSON output would be streamable and one "event" per line (except when pretty-printing is on).
• Every event can be of different types, with different fields. The event line would look like this:

  { "event type": { "field1": value1, "field2": value2, ... }}
  

• Unknown fields should be ignored by clients (so that we can extend events)
• In general every "event" represents a line in the current output.

Here are a couple of examples of the current output, and how it would look like in JSON:

1. fping -c3 1.1.1.1

   1.1.1.1 : [0], 64 bytes, 13.5 ms (13.5 avg, 0% loss)
   1.1.1.1 : [1], 64 bytes, 13.5 ms (13.5 avg, 0% loss)
   1.1.1.1 : [2], 64 bytes, 13.5 ms (13.5 avg, 0% loss)
   
   1.1.1.1 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 13.5/13.5/13.5
   

   JSON:

   {"resp": {"host": "1.1.1.1", "seq": 0, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
   {"resp": {"host": "1.1.1.1", "seq": 1, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
   {"resp": {"host": "1.1.1.1", "seq": 2, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
   {"summary": {"host": "1.1.1.1", "xmt": 3, "rcv": 3, "loss"=0, "rttMin": 13.5, "rttAvg": 13.5, "rttMax": 13.5}}
   

2. fping -Q1 -C2 1.1.1.1 8.8.8.8 -p1500

    [15:57:35]
    1.1.1.1 : xmt/rcv/%loss = 1/1/0%, min/avg/max = 13.5/13.5/13.5
    8.8.8.8 : xmt/rcv/%loss = 1/1/0%, min/avg/max = 3.61/3.61/3.61
    1.1.1.1 : 13.5 13.5
    8.8.8.8 : 3.61 3.64
   

   JSON:

   {"intSum": { "time": 1739631700, "host": "1.1.1.1", "xmt": 1, "rcv": 1, "loss"=0, "rttMin": 13.5, "rttAvg": 13.5, "rttMax": 13.5}}
   {"intSum": { "time": 1739631700, "host": "8.8.8.8", "xmt": 1, "rcv": 1, "loss"=0, "rttMin": 3.61, "rttAvg": 3.61, "rttMax": 3.61}}
   {"vSum": {"host": "1.1.1.1", "values": [ 13.5, 13.5]}}
   {"vSum": {"host": "8.8.8.8", "values": [3.61, 3.64]}}
   

   Here "intSum" would be for intermediate host summary, and "vSum" would be for "verbose host summary".

[schweikert]

I don't think we need to have everything moved to JSON format immediately, and it is fine to use stderr. One advantage of using this sort of "event" or object based approach is that it is extensible, so not everything needs to be supported.

[auerswal]

I'd prefer if unsupported combinations of options with --json would result in an error exit of fping. Thus adding support would not break existing scripts.

[gsnw-sebast]

I would currently keep the JSON format as small as possible and agree with @schweikert.
The reason for this is the current implementation in the code, which is very bloated due to the many individual outputs.

It already looks quite confusing. Especially as I also remember that JSON is only one format and after JSON someone else will want to use XML or other formats, for example.

[gsnw-sebast]

What do you think about simply duplicating print_netdata() and modifying it for the JSON output?
print_netdata() has the same purpose, which might be considered for json first.

[schweikert]

Yes. I think that's a good idea. Maybe we can have a lightweight abstraction of the output function, which calls the appropriate function based on the wanted format. Something like:

void print_per_system_stats(void) {
    if (netdata_flag) {
            print_per_system_stats_netdata()
    }
    else if(json_flag) {
            print_per_system_stats_json()
    }
    else {
            print_per_system_stats_legacy()
    }
}

[auerswal]

I'd think it could be helpful to have some use case descriptions. How would you like to use fping --json? (I do not (yet) have a use case for JSON output from fping. That makes it hard for me to review and evaluate pull requests for this feature.)

[schweikert]

The main use case I think is integration into tools that want to do some probing using fping.

[AthenaNetworks]

I think we need to discuss how we should build the JSON, do we push each update (ping reply) to the console as its own JSON object and expect the receiving script/program to process that as an independent JSON chunk, or do we write it as a simple JSON object that is streamed to the user.

I'll explain the differences

as a whole
{"resp": {"host": "1.1.1.1", "seq": 0, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
{"resp": {"host": "1.1.1.1", "seq": 1, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
{"resp": {"host": "1.1.1.1", "seq": 2, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}}
{"summary": {"host": "1.1.1.1", "xmt": 3, "rcv": 3, "loss"=0, "rttMin": 13.5, "rttAvg": 13.5, "rttMax": 13.5}}

Is not parsable as a JSON object. However, each line is.

The alternative is to stream it as a JSON array, for example

[{"resp": {"host": "1.1.1.1", "seq": 0, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}},
{"resp": {"host": "1.1.1.1", "seq": 1, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}},
{"resp": {"host": "1.1.1.1", "seq": 2, "size": 64, "rtt": 13.5, "rttAvg": 13.5, "loss": 0}},
{"summary": {"host": "1.1.1.1", "xmt": 3, "rcv": 3, "loss": 0, "rttMin": 13.5, "rttAvg": 13.5, "rttMax": 13.5}}]

Which is absolutely parsable, however! Its only parsable as a complete object, so once it finishes.

Food for thought.

[schweikert]

In my opinion the output should not only be parseable at the end. I think that streamable json would be preferable. Even if you are not interested in streaming the output, parsing it line-by-line should be easy enough. In other words, the streaming output.

[AthenaNetworks]

I'm in agreeance here. I didn't want to put forward my preference, in the hopes of seeing other opinions and not wanting to sway them.

In my experience, I read these things until the next line feed, then parse the response. So I'd be inclined to suggest that a each "update" should be its own object.

Otherwise, the alternative is, not pushing updates at all, and instead only providing a single JSON object at the very end, and just make that either a limitation of the implementation or a feature that is toggle-able?

[schweikert]

By the way another use case is errors. Those could also be surfaced as "error" events.

[AthenaNetworks]

In my implementation, I just check for is "does array key exist 'error'" then consider it an error, we could definitely handle it more gracefully with specific return codes, because right now I just "return", but we would also need to go through every instance of perror() and handle it as JSON also.

So yeah, lookiung back at my code, its very much incomplete, despite passing tests.

I think maybe we need to settle on the JSON format, then I'm happy to implement the changes and also cover off any error events. Or you can, I don't mind.

[auerswal]

I'd think that it might be simpler for tools using a new JSON output format from fping to only have to read one data stream, i.e., if errors were sent on standard output with a key called error.

[AthenaNetworks]

Just FYI, here is my JSON implementation working in production :)