status.json wishlist

[drkane]

Following conversation with @robredpath - this is my wishlist for the contents of the json output after completion of #4.

For each file record the following information:

(NB - there's a question for me over whether this should be "for every fundingOrganization in every file" - theoretically a publisher could include more than one funder in a file?)

• list of the funders included

• list of the currencies used

• earliest awardDate

• latest awardDate

• earliest lastModified

• latest lastModified

• total number of rows (& split by currency)

• total value of grants (split by currency)

• for each field in the file:

  • field name
  • number of null/blank values
  • (future development) - number of values matching the required format (eg conform to regex)
  • (future development) - for fields that can be externally validated (eg identifiers) - number of values that are externally verified
  • whether the field appears in the 360 giving standard (plus guess at the field type if a non-standard field)

• required fields that do not appear in file

• recommended fields that do not appear in file

• something on location fields, giving a breakdown of the number of location fields found (eg X records have 0 beneficiary location records, Y records have 1 beneificiary location, Z records have 2 beneficiary location)

[Bjwebb]

Thanks for putting together this list!

We've been doing some work on status.json at ThreeSixtyGiving/datagetter#6. It is now accessible at
https://storage.googleapis.com/datagetter-360giving-output/branch/master/status.json

I think all the fields you want will be in under the datagetter_aggregates key in each dataset:

We have the following:

• earliest awardDate: min_award_date
• latest awardDate: max_award_date
• list of the currencies used: they keys within currencies
• total number of rows (& split by currency): count and currencies/XXX/count
• total value of grants (split by currency): currencies/XXX/total_amount

We don't yet have:

• list of the funders included Mw/update requires datagetter#8

• earliest lastModified Bump lxml from 4.5.2 to 4.6.2 datagetter#9

• latest lastModified Bump lxml from 4.5.2 to 4.6.2 datagetter#9

• for each field in the file: requirements: Update flatten-tool datagetter#11

  • field name
  • number of null/blank values
  • (future development) - number of values matching the required format (eg conform to regex)
  • (future development) - for fields that can be externally validated (eg identifiers) - number of values that are externally verified
  • whether the field appears in the 360 giving standard (plus guess at the field type if a non-standard field)

• required fields that do not appear in file flattentool calls fetch the schema for each download datagetter#10

• recommended fields that do not appear in file flattentool calls fetch the schema for each download datagetter#10

• something on location fields, giving a breakdown of the number of location fields found (eg X records have 0 beneficiary location records, Y records have 1 beneificiary location, Z records have 2 beneficiary location) get: Allow ods file types datagetter#13

As I've added new issues for each point, I'm closing this issue. Also, work has moved to the https://github.com/ThreeSixtyGiving/datagetter repo, rather than this one, so please report any additional issues there.