feat(schematic): added synapse cache purging fds-1445

[andrewelamb]

• fixes FDS-1445
• Adds synapse cache purging
• Adds deault_config.yaml for globals
• deault_config.yaml can be overridden by config.yaml file for settable globals
• Synapse Cache path is setable (set to /var/tmp/synapse)
• Synapse Cache purging is setable (set to true)
• Updated readme

[andrewelamb]

@jesusaurus This is the new Schematic API we are hoping to deploy soon, that would replace the old one.

I undertsand that it was suggested by you in this PR, that the synapse cache be moved to "/var/tmp/synapse". Is that still valid?

[linglp]

Have you tried executing one of the endpoints and see if everything is working as expected? I tried executing: http://localhost:7443/api/v1/ui/#/Storage/get_manifest_csv,
and I got an error:

{
  "detail": "local variable 'byte_size' referenced before assignment",
  "status": 500,
  "title": "Internal error"
}

But on schematic side, I tried:

store = SynapseStorage(
        access_token="<my token>", synapse_cache_path="/var/tmp/synapse"
)

and it worked...
Do you have similar errors?

[andrewelamb]

It's an int, thats what the ceil function does.

[ConsoleCatzirl]

@andrewelamb yeah, I still have a mild preference for /var/tmp/synapse. It's a standard location for linux services to store a persistent cache, and it preemptively moves the cache to a location that a service user can own, potentially making a transition to running as a service user a little simpler.

[andrewelamb]

Have you tried executing one of the endpoints and see if everything is working as expected? I tried executing: http://localhost:7443/api/v1/ui/#/Storage/get_manifest_csv, and I got an error:

{
  "detail": "local variable 'byte_size' referenced before assignment",
  "status": 500,
  "title": "Internal error"
}

But on schematic side, I tried:

store = SynapseStorage(
        access_token="<my token>", synapse_cache_path="/var/tmp/synapse"
)

and it worked... Do you have similar errors?

I can't replicate this error

[linglp]

Also, I think it worths noting some other changes that were made in the PR:

• when running schematic api locally, purge_synapse_cache function will now purge the cache of users. In the current schematic code base, I think it only purge caches on AWS. When users run schematic APIs locally, we don't touch their cache. I am not sure if this is a "desired" behavior since "users" will most likely to be developers and probably aware of what's going on.
• And also worth noting that we now clear cache before calling manifest download endpoint.

[sonarqubecloud]

Quality Gate passed for 'schematic-api'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[linglp]

Thanks. I think it looks good to me.