10 9 2023 Tech Team Report

10-9-2023

Logged Tasks

Date	Task	Hours (Main)	Hours (EOLS)	Hours (PII)	Hours (QDAS)
2-Oct-2023	Investigate/coord PID failures for updateCitations call, reporting, meeting, review ORCID review requirements	3
3-Oct-2023	Investigate problem viewing html, merge/deploy rich html viewer, start debugging errors, add ORCID email messages/logo/err responses on dev/stage, develop sql process to delete datasets, delete corrupt datasets/files on dev, start on stage.	5
4-Oct-2023	Finish stage/prod removal of corrupt datasets, reindex on dev/stage, check index status, investigate issues, try reindexing/perm indexing to repair, investigate unresolved issues, coord re: ORCID review, dev to not allow editing file extensions	5
5-Oct-2023	Continue investigating index/status issues, investigate storJ failures	3
6-Oct-2023	Find storJ fail on bucket check, remove/document, investigate storJ rate limiting, implement configurable delay, test, coord re: ORCID member API, change stage back to public API, fix orcid icon issue, change text; update drupal webforms to 6.1.6	4

Operations

• Investigated report of failed calls to /updateCitations - confirmed that they are due to the corrupt datasets in the db
• Developed SQL queries to delete corrupt datasets and all of the versions, files, metadata, etc.
• Ran queries to clean the db on all three machines. All of the bad datasets on prod where also in the dev and stage databases and were clearly early tests of some form.
• On dev/stage, I ran reindexing after cleaning the bad datasets and then explored the few remaining datasets that showed an issue. I haven't finished this but did identify use of storJ as an issue - it appears that storJ is imposing a rate cap on their s3 gateway and we go over that, causing failures. I've started looking into rate-limiting our requests (see Dataverse section).

SSO

• Reviewed ORCID review criteria - they are not well aligned with openid focused use. Changed Keycloak ORCID plugin to use the public API again and have appropriate messages
• Fixed reported issue with an odd overlay on the ORCID icon in the login page (due to a collision with the fa-orcid css class used in the Keycloak plugin and by font-awesome)

Drupal

• Updated webforms module, deployed to dev/stage

Dataverse

• Investigated reported problem on production in viewing HTML - the page includes javascript which the standard previewer strips out. There is a new proof-of-concept 'rich html' previewer that allows users to agree to run scripts which improves the preview for the prod file but doesn't make it perfect. Work would be needed to make the rich html previewer work well (it has issues when displayed on the file page) and to only show it on some files/to integrate it with the existing html previewer, etc.
• Added changes to not allow editing the file name extension and deployed to to dev/stage. Doing this avoids having to make changes to more code to switch from name extension edits affecting the ingested/tab version instead of the original file and handling existing files with name extension changes. It may make sense in the future to allow editing extensions along with the ability to recheck/edit the mimetype, etc., but the expectation is that the change to not allow extension editing will be sufficient to be able to deploy the overall set of changes to showing the original file instead of the ingested version.
• Started adding the ability to put a configurable delay before making calls to the storJ gateway. I tried a few delays in the range of 20-100 msec but didn't have success. I suspect that storJ may not immediately adjust to the new rate, so I want to test the delays again this week to see if it will work now.
• As part of this, I removed a check that the bucket exist that would run on every file access (nominally once you're configured correctly, the bucket will exist. If ti does disappear, we would still see file access failures that would alert you to a missing bucket.). That check appeared to be failing for storJ and it isn't clear if that was because Amazon is not handling the custom endpoint in that call now or if the rate-limit at storJ was causing failures. In any case, removing this check makes sense and should make things more efficient.

AnnoRep

Discussion

• Given the state of the rich HTML viewer (see above), what's the best way forward for the prod issue of previewing/not previewing HTML with JavaScripts?
• I've run a new index/status API call and found that there are a few datasets not being indexed on dev/stage. It appears that some of the indexing issues right now cause the whole dataset to not be indexed (meaning it would disappear from the main list and search results - basically a show-stopper for deployment). It may be that this is all due to the storJ issue, in which case the fix for handling rate-limiting would resolve the problem. If not, I need to find the other issues and/or improve how exceptions are caught so that overall indexing failure doesn't happen. (This is how we handle full-text indexing failures - they don't stop the dataset or file from being indexed, they just don't get full-text.) It is also possible that the affected datasets on dev/stage are ~corrupt in some way because they were created with non-production code. In that case, it would not be a prod issue and wouldn't affect deployment there.

Plans

• SSO
  • Support testing as needed
• Fix Stata-14 ingest by allowing file inspection during direct upload or adjusting the Stata ingester.
• Fix #113 if possible
• Matomo - investigate event-level tracking via tag manager, remove non-working google scripts
• AnnoRep - explore round-trip, configure auto-start and log rotation
• Ops
  • check missing globalidcreationdates and fix via /modifyRegistration or alternative
• Dataverse
  • Make PR for guestbook adding datasetversion fix
  • Popup info accessibility - IQSS likes the recommendations from the source I linked to, so this can be implemented along those lines.
• Drupal - v10 - review compatibility and start updates.
• QDAS Previewer
  • Updates per request
  • Investigate writing aux file/previewing lower-sensitivity version and/or other write options
• TBD: FRDR Security