Add support for getting data OONI Data Kraken API

[hellais]

The current target is only the MAT.

We support an environment flag for getting the data from the aggregation API implement in ooni/data, see: ooni/data#45.

Here is a working example on how to run this:

1. Edit .env.local
2. Place the following in .env.local

# Used only in local environments
# Refer: https://nextjs.org/docs/basic-features/environment-variables

NEXT_PUBLIC_OONI_API=https://ams-pg-test.ooni.org
NEXT_PUBLIC_OONI_DATA_API=https://data.ooni.org
NEXT_PUBLIC_USER_FEEDBACK_API=https://ams-pg-test.ooni.org
NEXT_PUBLIC_EXPLORER_URL=http://localhost:3100

RUN_GIT_COMMIT_SHA_SHORT=yarn --silent git:getCommitSHA:short
RUN_GIT_COMMIT_SHA=yarn --silent git:getCommitSHA
RUN_GIT_COMMIT_REF=yarn --silent git:getCommitRef
RUN_GIT_COMMIT_TAGS=yarn --silent git:getReleasesAndTags

3. Run yarn run dev
4. Visit http://localhost:3100/chart/mat?probe_cc=ID&since=2023-10-23&until=2023-11-23&time_grain=day&axis_x=measurement_start_day&axis_y=domain&test_name=web_connectivity

Example live preview link: https://explorer-3ljbnbf5b-ooni1.vercel.app/chart/mat?probe_cc=ID&since=2023-10-23&until=2023-11-23&time_grain=day&axis_x=measurement_start_day&axis_y=domain&test_name=web_connectivity

Notes about the implementation

For backward compatibility we implement an aggregation API which is compliant with the current API. The only difference is that it also supports a new field called anomaly_threshold which is set by default to 0.7 and used to determine if a particular measurement is to be considered anomalous or not based on the loni values.

It does however already support returning extra data which could eventually be used in an alternative future MAT view.

The schema of the returned data format can be inspected here: https://github.com/ooni/data/blob/5dd30c472da7a6eaa19071c5a49a510693e03f19/oonidata/fastapi/routers/aggregation.py#L43.

The interesting stuff is inside of the loni_* keys.

These are aggregate loni based likelyhood of blocking estimates calculated using the new analysis engine and are values that go from 0-1, where 1 indicates our belief in that particular blocking method being implemented.

Other new interesting keys are:

• observation_count: the unique count of observations used to determine a particular aggregate loni value
• vantage_point_count : a guess of the number of unique vantage points that contributed to this experiment result (estimated via COUNT(DISTINCT location_network_type, location_network_asn, location_network_cc, location_resolver_asn))

Please note that the experiment results tables only contain data from the last few months and contrary to observation tables is not being updated regularly.

🆕 you can now view the LoNI values by clicking on a bar in the MAT

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
explorer	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 22, 2023 2:15pm