Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

First version of the image sync script #327

Closed
wants to merge 44 commits into from
Closed

First version of the image sync script #327

wants to merge 44 commits into from

Conversation

enolfc
Copy link
Contributor

@enolfc enolfc commented Dec 14, 2023 •
edited
Loading

Summary

This is a early attempt to get image sync working in catchall repo.
Very much WIP at the moment, but want to start getting feedback from others and get Linters complain about issues.

Right now it depends on atrope being slightly modified for our purposes (image expiration and image format conversion) and it's assuming it will be run as a docker container. As this will be probably run in a container itself.

What's missing:

  • Build the docker container
  • Some tests for the python code
  • Deploy this into a VM and set a cron job to run it periodically

After this is working, we can look into optimisations specially related to avoiding downloading/converting images that are already processed

Related issue :

@enolfc enolfc requested review from a team as code owners December 14, 2023 13:03
@enolfc
Copy link
Contributor Author

enolfc commented Dec 15, 2023 •
edited
Loading

Sample run on test VM

Configuration: config/sync.conf:

[sync]
appdb_token = XXXXXX
sites_file = /config/sites.yaml

[checkin]
client_id = XXXX
client_secret = YYYY

config/sites.yaml

---
WALTON-CLOUD:
  formats:
    - qcow2
    - raw
INFN-CLOUD-BARI:
  formats:
    - qcow2
    - raw

And running it:

root@atrope:~# docker run -it -v /root/state:/atrope-state -v /root/config:/config enolfc/atrope:20231215-2 sites-sync --config-file /config/sync.conf
DEBUG:root:Fetching site info from AppDB
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): is.appdb.egi.eu:443
DEBUG:urllib3.connectionpool:https://is.appdb.egi.eu:443 "GET /graphql?query=%0A++++++++%7B%0A++++++++++++siteCloudComputingEndpoints%7B%0A++++++++++++++items%7B%0A++++++++++++++++endpointURL%0A++++++++++++++++site+%7B%0A++++++++++++++++++name%0A++++++++++++++++%7D%0A++++++++++++++++shares%3A+shareList+%7B%0A++++++++++++++++++VO%0A++++++++++++++++++entityCreationTime%0A++++++++++++++++++projectID%0A++++++++++++++++%7D%0A++++++++++++++%7D%0A++++++++++++%7D%0A++++++++%7D%0A++++ HTTP/1.1" 200 24533
DEBUG:root:Discarding site 100IT.
DEBUG:root:Discarding site BIFI.
DEBUG:root:Discarding site CENI.
DEBUG:root:Discarding site CESGA-CLOUD.
DEBUG:root:Discarding site CESGA.
DEBUG:root:Discarding site CESNET-MCC.
DEBUG:root:Discarding site CETA-GRID.
DEBUG:root:Discarding site CLOUDIFIN.
DEBUG:root:Discarding site CSTCLOUD-EGI.
DEBUG:root:Discarding site CYFRONET-CLOUD.
DEBUG:root:Discarding site ELKH-CLOUD.
DEBUG:root:Discarding site GRNET-OPENSTACK.
DEBUG:root:Discarding site GSI-LCG2.
DEBUG:root:Discarding site IFCA-LCG2.
DEBUG:root:Discarding site IISAS-FedCloud.
DEBUG:root:Discarding site ILIFU-UCT.
DEBUG:root:Discarding site IN2P3-IRES.
INFO:root:Configuring site INFN-CLOUD-BARI
INFO:root:Configuring vo.decido-project.eu
DEBUG:root:Running atrope --config-dir /tmp/tmpk1oia419 sync
2023-12-15 11:51:35.367 7 INFO atrope.cmd.commands [-] Atrope session starts >>>>>>>>>>
2023-12-15 11:51:35.711 7 ERROR atrope.image_list.manager [-] Error loading list 'vo.decido-project.eu' from 'https://vmcaster.appdb.egi.eu/store/vo/vo.decido-project.eu/image.list', reason: Cannot get image list, reason: (404) Not Found: atrope.exception.ImageListDownloadFailed: Cannot get image list, reason: (404) Not Found
2023-12-15 11:51:35.711 7 INFO atrope.cache [-] Syncing list with ID 'vo.decido-project.eu'
2023-12-15 11:51:35.711 7 INFO atrope.cache [-] List 'vo.decido-project.eu' is enabled, checking if downloaded images are valid
2023-12-15 11:51:35.712 7 INFO atrope.cache [-] Checking for invalid files in cache dir (/atrope-state/lists/vo.decido-project.eu).
2023-12-15 11:51:35.712 7 INFO atrope.cache [-] No invalid files in cache dir (/atrope-state/lists/vo.decido-project.eu).
2023-12-15 11:51:35.998 7 INFO atrope.dispatcher.manager [-] Preparing to dispatch list 'vo.decido-project.eu'
2023-12-15 11:51:35.999 7 ERROR atrope.image_list.source [-] Image list vo.decido-project.eu has not been fetched!
2023-12-15 11:51:35.999 7 ERROR atrope.image_list.source [-] Image list vo.decido-project.eu has not been fetched!
2023-12-15 11:51:37.782 7 INFO atrope.dispatcher.glance [-] Sync terminated for image list 'vo.decido-project.eu'
2023-12-15 11:51:37.782 7 INFO atrope.cmd.commands [-] Atrope session ends <<<<<<<<<<<<
INFO:root:Configuring vo.binare-oy.eu
DEBUG:root:Running atrope --config-dir /tmp/tmpzj6qzikq sync
2023-12-15 11:51:38.909 8 INFO atrope.cmd.commands [-] Atrope session starts >>>>>>>>>>
2023-12-15 11:51:39.313 8 WARNING atrope.image [-] Image 'eea4aa5f-9af9-5fef-96cc-cf0451586c19' expired on '2021-08-31 21:00:00+00:00'
2023-12-15 11:51:39.313 8 WARNING atrope.image [-] Image 'ac2cbc9d-d5bb-564a-8141-81bf042ae645' expired on '2021-09-09 21:00:00+00:00'
2023-12-15 11:51:39.313 8 WARNING atrope.image [-] Image '05fb14ea-d515-5a98-bed9-5dfc3288d207' expired on '2021-07-12 21:00:00+00:00'
2023-12-15 11:51:39.314 8 INFO atrope.cache [-] Syncing list with ID 'vo.binare-oy.eu'
2023-12-15 11:51:39.314 8 INFO atrope.cache [-] List 'vo.binare-oy.eu' is enabled, checking if downloaded images are valid
2023-12-15 11:51:39.314 8 INFO atrope.cache [-] Checking for invalid files in cache dir (/atrope-state/lists/vo.binare-oy.eu).
2023-12-15 11:51:39.315 8 INFO atrope.cache [-] No invalid files in cache dir (/atrope-state/lists/vo.binare-oy.eu).
2023-12-15 11:51:39.597 8 INFO atrope.dispatcher.manager [-] Preparing to dispatch list 'vo.binare-oy.eu'
2023-12-15 11:51:41.380 8 INFO atrope.dispatcher.glance [-] Sync terminated for image list 'vo.binare-oy.eu'
2023-12-15 11:51:41.381 8 INFO atrope.cmd.commands [-] Atrope session ends <<<<<<<<<<<<
INFO:root:Configuring vo.emso-eric.eu
DEBUG:root:Running atrope --config-dir /tmp/tmpg5vfksdj sync
2023-12-15 11:51:42.505 10 INFO atrope.cmd.commands [-] Atrope session starts >>>>>>>>>>
2023-12-15 11:51:42.903 10 INFO atrope.cache [-] Syncing list with ID 'vo.emso-eric.eu'
2023-12-15 11:51:42.903 10 INFO atrope.cache [-] List 'vo.emso-eric.eu' is enabled, checking if downloaded images are valid
2023-12-15 11:51:42.904 10 INFO atrope.image [-] Downloading image '883777e7-dbaa-54e4-8948-0ad1e05a1165' from 'https://api.cloud.ifca.es:8080/swift/v1/egi_endorsed_vas/Ubuntu.20.04-2023.10.24.ova' into '/atrope-state/lists/vo.emso-eric.eu/images/883777e7-dbaa-54e4-8948-0ad1e05a1165'
2023-12-15 11:54:11.096 10 INFO atrope.image [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' present in '/atrope-state/lists/vo.emso-eric.eu/images/883777e7-dbaa-54e4-8948-0ad1e05a1165', verifying checksum
2023-12-15 11:54:19.155 10 INFO atrope.image [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' present in '/atrope-state/lists/vo.emso-eric.eu/images/883777e7-dbaa-54e4-8948-0ad1e05a1165', checksum OK
2023-12-15 11:54:19.156 10 INFO atrope.image [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' stored as '/atrope-state/lists/vo.emso-eric.eu/images/883777e7-dbaa-54e4-8948-0ad1e05a1165'
2023-12-15 11:54:19.160 10 INFO atrope.image [-] Downloading image '4f140483-e3c0-569b-951d-1ceddac0e477' from 'https://api.cloud.ifca.es:8080/swift/v1/egi_endorsed_vas/centos.8-2023.10.24.ova' into '/atrope-state/lists/vo.emso-eric.eu/images/4f140483-e3c0-569b-951d-1ceddac0e477'
2023-12-15 11:59:56.195 10 INFO atrope.image [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' present in '/atrope-state/lists/vo.emso-eric.eu/images/4f140483-e3c0-569b-951d-1ceddac0e477', verifying checksum
2023-12-15 12:00:07.846 10 INFO atrope.image [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' present in '/atrope-state/lists/vo.emso-eric.eu/images/4f140483-e3c0-569b-951d-1ceddac0e477', checksum OK
2023-12-15 12:00:07.847 10 INFO atrope.image [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' stored as '/atrope-state/lists/vo.emso-eric.eu/images/4f140483-e3c0-569b-951d-1ceddac0e477'
2023-12-15 12:00:07.851 10 INFO atrope.cache [-] Checking for invalid files in cache dir (/atrope-state/lists/vo.emso-eric.eu).
2023-12-15 12:00:07.853 10 INFO atrope.cache [-] No invalid files in cache dir (/atrope-state/lists/vo.emso-eric.eu).
2023-12-15 12:00:08.169 10 INFO atrope.dispatcher.manager [-] Preparing to dispatch list 'vo.emso-eric.eu'
2023-12-15 12:00:08.170 10 INFO atrope.dispatcher.glance [-] Glance dispatching '883777e7-dbaa-54e4-8948-0ad1e05a1165'
2023-12-15 12:00:53.163 10 INFO atrope.image [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' converted and stored at /atrope-state/lists/vo.emso-eric.eu/images/883777e7-dbaa-54e4-8948-0ad1e05a1165.qcow2
2023-12-15 12:00:53.166 10 INFO atrope.dispatcher.glance [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' stored in glance as 'c5e54ddc-77ec-46f4-aed8-ffd0cb9b9ed6'.
2023-12-15 12:00:53.166 10 INFO atrope.dispatcher.glance [-] Image '883777e7-dbaa-54e4-8948-0ad1e05a1165' owned by dest project 18ef543d904b441cbb9c5f2302dbd154.
2023-12-15 12:00:53.168 10 INFO atrope.dispatcher.glance [-] Glance dispatching '4f140483-e3c0-569b-951d-1ceddac0e477'
2023-12-15 12:01:48.709 10 INFO atrope.image [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' converted and stored at /atrope-state/lists/vo.emso-eric.eu/images/4f140483-e3c0-569b-951d-1ceddac0e477.qcow2
2023-12-15 12:01:48.713 10 INFO atrope.dispatcher.glance [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' stored in glance as '35821f5e-fe7c-4d93-80eb-d2a413cf88dd'.
2023-12-15 12:01:48.714 10 INFO atrope.dispatcher.glance [-] Image '4f140483-e3c0-569b-951d-1ceddac0e477' owned by dest project 18ef543d904b441cbb9c5f2302dbd154.
2023-12-15 12:01:48.979 10 INFO atrope.dispatcher.glance [-] Sync terminated for image list 'vo.emso-eric.eu'
2023-12-15 12:01:48.980 10 INFO atrope.cmd.commands [-] Atrope session ends <<<<<<<<<<<<
[...]

gwarf
gwarf approved these changes Jan 8, 2024
View reviewed changes
Copy link
Member

@gwarf gwarf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

enolfc and others added 14 commits February 27, 2024 14:09
@enolfc
Merge branch 'main' into atrope
305d6c8
@enolfc
Merge branch 'main' into atrope
f47b4a2
@astalosj @enolfc
Update IISAS-FedCloud-cloud.yaml (EGI-Federation#333)
17bc3ba 
Cleaning of removed local projects.
@dependabot @enolfc
Bump github/super-linter from 5 to 6 (EGI-Federation#335)
5109b21 
* Bump github/super-linter from 5 to 6

Bumps [github/super-linter](https://github.com/github/super-linter) from 5 to 6.
- [Release notes](https://github.com/github/super-linter/releases)
- [Changelog](https://github.com/github/super-linter/blob/main/CHANGELOG.md)
- [Commits](github/super-linter@v5...v6)

---
updated-dependencies:
- dependency-name: github/super-linter
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>

* Add explicit reference to checkout

* Shell format

* markdown fixes

* Disable CHECKOV

* Trying to get this sorted out

* Disable linters

* Use the right name of the variable

* Site no longer available

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Enol Fernandez <[email protected]>
@mariojmdavid @enolfc
update site NCG-INGRID-PT (EGI-Federation#337)
220ac5c
@enolfc
Glue21 JSON (EGI-Federation#340)
5cdc2f9 
* Use https

* Upgrade cloud-info-provider

* Install rclone

* Add json output and upload to S3

* Make linter happy

* Remove VOs not in ops portal

* Remove VO not in ops portal

* Simplify management of tokens

We are no longer using the refresh style

* Fixes tested in the playground

* Fix linter / testing issues

* Fix tests

* Add some docs

* ADR

* Linter is my friend
@enolfc
Add the type of provider (EGI-Federation#342)
2a976bb 
* Add the type of provider

Otherwise rclone will fail

* Fix the user id

* Remove old VOs
@enolfc
Remove sites: 100IT and UPV-GRyCAP (EGI-Federation#343)
cd286db 
* Remove sites: 100IT and UPV-GRyCAP

* Remove gone VO
@dependabot @enolfc
Bump docker/build-push-action from 5 to 6 (EGI-Federation#345)
a4a8f52 
Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 5 to 6.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](docker/build-push-action@v5...v6)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@sebastian-luna-valero @enolfc
Remove sites: UNIV-LILLE (EGI-Federation#346)
94004bd
@enolfc
Use fedcloud secrets (EGI-Federation#347)
4bb7d92 
* Do not send long lived secrets to VM

Instead use fedcloud secret command with a locker that can only be used
2 times (one for putting the secret, another for getting it) and for 1
hour max.

* Move the ansible role to this repository

Instead of having this externally managed as it is a pain to update and
to keep properly aligned

* Move to the embedded role
@enolfc
Cleaning
a57eba8
@enolfc
Add image sync config
e9b9ce6
@enolfc
Some testing
081ef5b
@enolfc
Copy link
Contributor Author

enolfc commented Jul 10, 2024

Closing as this has become too messy and opening again with a clean history

@enolfc enolfc closed this Jul 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gwarf gwarf gwarf approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@enolfc @gwarf @astalosj @mariojmdavid @sebastian-luna-valero