test metadig-engine on k8s against a hashstore #453
Assignees
Comments
|
Update: The
Next Steps:
To Do List:
For reference: # How to produce a text file with just the first level of hashstore folders to rsync
mok@dev:~/testing$ sudo find /var/metacat/hashstore -mindepth 1 -maxdepth 1 > mc_hs_dir_list.txt
mok@dev:~/testing$ cat mc_hs_dir_list.txt
/var/metacat/hashstore/objects
/var/metacat/hashstore/metadata
/var/metacat/hashstore/refs
/var/metacat/hashstore/hashstore.yaml
# How to use rsync with a list of folders
mok@dev:~/testing$ cat mc_hs_dir_list.txt | parallel --eta sudo rsync -aHAX {} /mnt/tdg-repos/dev/metacat/hashstore/# First get the list of files found under `/hashstore`
mok@dev:~/testing$ sudo find /var/metacat/hashstore -type f -printf '%P\n' > mc_obj_list.txt
# How to feed a single command at a time for a file to rsync
# The /./ between `metacat` and `hashstore` instructs rsync to copie folders from hashstore (and omits the previous directories) into the desired folder
mok@dev:~/testing$ parallel --eta sudo rsync -aHAXR /var/metacat/./hashstore/{} /mnt/tdg-repos/dev/metacat :::: mc_obj_list.txt
|
|
Metacat on
|
|
Current Status: It appears the 'Assessment Reports' (Metadig) for datasets at
Next Steps:
To Do List & Follow-up Questions
|
|
Update:
To Do List & Follow-up Questions
|
|
Update:
|
|
Update: Even after fixing the connection URL (below), I am still experiencing a http 403 forbidden error. The logging message which shows the end point is accessible via both the browser, and within the The I have a feeling that this is related to how k8s allows external REST API calls to be made (or not). The specific JAVA code to make the get request appears to be fine (since it can communicate and receive a 403 error). Investigation continues. |
k8s does not restrict pods from originating web connections to external hosts in any way unless it is configured to do so. MetaDIG is not configured to restrict anything afaik. You and I should touch base on this because I think you are following a red herring and the problem originates elsewhere. Your curl command from the pod shows that the connection is not blocked. So its something else about how you deployed. Let's chat. |
|
@mbjones I think so too - I can't find anything related to that. I just pushed a commit to test whether the request is getting rejected because it's missing a Deployment code for quick reference (taken from hand-off notes): With the following changes in the respective
|
|
@mbjones The Assessment Report generated after adding the To Do List & Follow-up Questions
|
|
Update:
|
|
@doulikecookiedough regarding your question on how to directly communicate with metadig, that would be via the API. Most operations require authentication, but you can, for example, access completed run reports with a request like:
This one doesn't work, as it appears the If the API doesn't provide what you need, you can query the database itself via |
|
Thank you for the clarification/direction @mbjones. Currently it looks like there's an issue with the scheduler - after restarting the pods (making sure the chart and app versions were both updated), some |
|
Check-in:
To Do
|
|
Check in:
To Do
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Testing locally has gone well but it would be nice to test the engine against a hashstore on the dev cluster
to that end I've mounted the
tdgsubvolume on metadig-worker, and that subvolume was mounted ondev.nceaswhere there is a hashstore metacat running. Seehelm/metadig-worker/pv.yamlandhelm/metadig-worker/pvc.yamlfor details on the existing mounts.In order to actually test though the following steps are needed:
metacat/hashstoreto/mnt/tdg-repos/devvia parallel Rsyncmetacat.propertiesstore.store_pathfield to bevar/data/respos/dev/hashstoremetadig-engineto the test clusterdev.nceas(via metacatUI or any other client)The text was updated successfully, but these errors were encountered: