Exclusion of json files with non-standard or compliant META prefixes

[M-DOREL]

Hello Maggot Team,

I tried to push files from Dataverse into Filebrowser and prefixed them with META to test if Maggot can handle them.

The result is an error parsing the file in the pdg_data/ directory (line with regex.sub(’ ’, value) in db-insert.py).

I understand that this is not the way to name the files. If the files have not been generated by Maggot, then there will be no ‘META_’ prefix.

But you can avoid this type of error for users in Filebrowser as I have done. If a user renames their file with the ‘META_’ prefix, there will be no files scanned and there will be an error.

I suggest creating :

• a "scan.log" file in the directory to track inconveniences.
  (something with the logging module in python)
• In the scan script (db-insert.py), check whether the ‘profile’ key is present in the json file scanned and check whether it ends with ‘maggot-schema.json’.

I hope this helps