Incorrect TTFB and request duration

[bajtos]

The PR #51 introduced a bug: we initialise stats.startAt before querying the indexer, and we don't measure how long the indexer query took.

As a result, our dashboards include "time to query IPNI" in the values for "time to first byte" and "request duration".

I discovered this problem while looking at #44.

[bajtos]

I am proposing the following changes:

In Spark checker (this repo):

• Modify queryTheIndex to produce a new stat field: indexerLookupTime
• Modify fetchCAR to produce two new stat fields: timeToFirstByte and timeToLastByte
• Move the code changing stats.endAt up to nextRetrieval - the same function that is setting stats.startAt.

In spark-api & spark-publish:

• Store the new fields in the DB and publish them to IPFS
• Keep support for firstByteAt for now, but add a code comment with a deprecation notice

In spark-evaluate:

• Rework the code calculating TTFB, request duration and job duration to use the new fields

@juliangruber WDYT?

[juliangruber]

I think it's a good idea to rename the ttfb field, to make it clear that it has new semantics now. Otoh, "firstByteAt" and "timeToFirstByte" are so similar that the renaming seems arbitrary. Therefore I'm +-0.

I definitely agree with adding indexerLookupTime, and starting the TTFB measurement after the indexer lookup finished (unless we want to include indexer lookup in the ttfb time, just like dns too, since it would be a normal part of every request).

[bajtos]

TTFB & retrieval duration is now calculated correctly. I am closing this issue as done.

Let's open new issues for recording the duration of other requests performed as part of the retrieval checks if we ever need those metrics.