Creates GH Actions section for sample data #13095

filipefurtad0 · 2025-01-23T22:53:18Z

What? Why?

Closes GH-Actions option to create sample data on staging servers #13094

Introduces a section on GH-Actions to provide the possibility to manage creation of sample data.

Mockup:

Marked in yellow: section on branch/tag to be removed. Not sure how to do this.

What should we test?

Visit GH-Actions section.
Notice there is a new section called "Deploy Sample Data"
Select it; Click Run Workflow (right side of the screen)
It should open a menu, similar to the "Deploy to Staging"
One should be able to select a staging server (dropdown)
One should be able to and type in sample data file name
Clicking Run Workflow should run the script, and:
- make a backup/baseline of the DB
- create sample data according to the given input

Release notes

Changelog Category (reviewers may add a label for the release notes):

User facing changes
API changes (V0, V1, DFC or Webhook)
Technical changes only
Feature toggled

The title of the pull request will be included in the release notes.

Dependencies

Documentation updates

We'll need to add this to the testers handbook.

This is intended to be used by testers, to generate sample data, on demand

…foodfoundation#12947

filipefurtad0 · 2025-01-23T23:02:51Z

It's a bit difficult to know whether this will work, as the changes will only be visible after merging. At least I'm not aware of any way to test this, in advance.

Also, I'm not sure the script section will do what it should. Some questions:

do we really need to reset the DB? Or can we run the sample_data script on top of existing data? It would be best to keep existing data.
DB backup: should we run the command provided on the test instructions from 2072 New sample data for testing #3209 or use the baseline resetting option, as introduced in Add scripts to save and restore baseline data #12947?

rioug · 2025-02-03T22:24:29Z

.github/workflows/sample_data.yaml

+          mkdir -p ~/backup/
+          pg_dump -h localhost -U ofn_user openfoodnetwork | gzip > ~/backup/staging-`date +%Y%m%d%H%M%S`-before-reset.sql.gz
+          bundle exec rake db:reset ofn:sample_data
+          sudo systemctl stop puma sidekiq


I think this should be starting puma/sidekiq again :

Suggested change

sudo systemctl stop puma sidekiq

sudo systemctl start puma sidekiq

rioug · 2025-02-03T22:30:58Z

.github/workflows/sample_data.yaml

@@ -35,5 +35,5 @@ jobs:
          sudo systemctl stop puma sidekiq
          mkdir -p ~/backup/
          pg_dump -h localhost -U ofn_user openfoodnetwork | gzip > ~/backup/staging-`date +%Y%m%d%H%M%S`-before-reset.sql.gz
-          bundle exec rake db:reset ofn:sample_data
+          bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}


I am not sure I am following this, as far as I understand we only have ofn:sample_data task for sample data ?
What are the expected inputs here ?

I can imagine it might be useful to provide individual scripts for each sample data factory (eg users, enterprises, fees). But that hasn't been done yet, so I guess Filipe was just testing to see what's possible. We can't test it until it's merged so I'm happy to leave this in.

It's worth noting the sample data factories depend on each other (eg sample enterprises are managed by the sample users), so if you execute enterprises before users, the script would probably fail. But as long as the full sample data script is executed first, that should never be a problem.

do we really need to reset the DB? Or can we run the sample_data script on top of existing data? It would be best to keep existing data.

It looks like the sample_data script can run on top of existing data. If there is a matching record (eg user with same email, or enterprise with same name), the record will be overwritten with the sample data (enterprise data shown here for example: https://github.com/openfoodfoundation/openfoodnetwork/blob/master/lib/tasks/sample_data/enterprise_factory.rb#L24)

So we can do this without a reset:

Suggested change

bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}

bundle exec rake ofn:${{ secrets.sample_data_name }}

dacook

DB backup: should we run the command provided on the test instructions from 2072 New sample data for testing #3209 or use the baseline resetting option, as introduced in Add scripts to save and restore baseline data #12947?

Use the new script. Of course, to restore the db would require another GH Action, which you could make after this one succeeds :D

dacook · 2025-02-03T22:50:51Z

.github/workflows/sample_data.yaml

@@ -35,5 +35,5 @@ jobs:
          sudo systemctl stop puma sidekiq
          mkdir -p ~/backup/
          pg_dump -h localhost -U ofn_user openfoodnetwork | gzip > ~/backup/staging-`date +%Y%m%d%H%M%S`-before-reset.sql.gz
-          bundle exec rake db:reset ofn:sample_data
+          bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}


I can imagine it might be useful to provide individual scripts for each sample data factory (eg users, enterprises, fees). But that hasn't been done yet, so I guess Filipe was just testing to see what's possible. We can't test it until it's merged so I'm happy to leave this in.

It's worth noting the sample data factories depend on each other (eg sample enterprises are managed by the sample users), so if you execute enterprises before users, the script would probably fail. But as long as the full sample data script is executed first, that should never be a problem.

dacook · 2025-02-03T22:55:32Z

.github/workflows/sample_data.yaml

@@ -35,5 +35,5 @@ jobs:
          sudo systemctl stop puma sidekiq
          mkdir -p ~/backup/
          pg_dump -h localhost -U ofn_user openfoodnetwork | gzip > ~/backup/staging-`date +%Y%m%d%H%M%S`-before-reset.sql.gz
-          bundle exec rake db:reset ofn:sample_data
+          bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}


do we really need to reset the DB? Or can we run the sample_data script on top of existing data? It would be best to keep existing data.

It looks like the sample_data script can run on top of existing data. If there is a matching record (eg user with same email, or enterprise with same name), the record will be overwritten with the sample data (enterprise data shown here for example: https://github.com/openfoodfoundation/openfoodnetwork/blob/master/lib/tasks/sample_data/enterprise_factory.rb#L24)

So we can do this without a reset:

Suggested change

bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}

bundle exec rake ofn:${{ secrets.sample_data_name }}

mkllnk · 2025-02-04T04:38:23Z

.github/workflows/sample_data.yaml

+    inputs:
+      server:
+        description: "Staging Server"
+        type: choice
+        required: true
+        options:
+          - staging.openfoodnetwork.org.uk
+          - staging.openfoodnetwork.org.au
+          - staging.coopcircuits.fr
+      sample_data_name:
+        description: "Sample Data Name"
+        type: string
+        required: true


This is really nice to have some input for an action.

mkllnk · 2025-02-04T04:41:13Z

.github/workflows/sample_data.yaml

+      - name: Run sample data script
+        if: success()
+        run: |
+          sudo systemctl stop puma sidekiq
+          ~/apps/openfoodnetwork/current/script/ci/save-staging-baseline.sh


I'm sorry but this doesn't work.

This script is executed on the Github Actions machine. It's not executed on one of the staging servers. In theory, we could execute commands via ssh on the staging servers but at the moment they don't allow that. We would need to authorize an SSH key stored in Github to perform these commands. And I'm not sure we want that.

😅 eep I missed that! We do currently execute commands remotely with ansible when deploying to staging, so it wouldn't be too much of stretch to do the same here.

What we need is for this script to be saved on the server. Then it's pretty trivial to log in and run it. eg:

ssh [email protected] script/load_sample_data

If we have that, then setting up this remote trigger doesn't seem worth the effort. But if it makes it easier for non-technical testers (ie Rachel) then perhaps it is worth the effort. Maybe we need to discuss at delivery-circle.

This script is executed on the Github Actions machine

Uff, I obviously I missed that as well. Thanks for pointing that out @mkllnk.

What we need is for this script to be saved on the server. Then it's pretty trivial to log in and run it.

Ok, I'll have a second look at this @dacook . I'm not sure though I fully grasp the complexity here - I'm looking out for this Slack thread, and then proceed. Thanks for your help and reviews.

Sorry I wasn't very prescriptive. I'm not sure if Maikel has concerns about this approach.
I suggest the next step is to create a new script in ofn-install. I had a look.. it's not easy to explain.. I'll try it..

Maikel what do you think of this? openfoodfoundation/ofn-install#994

Yes, all possible. I was just cautious because it's not quick and easy. But if there's enough appetite for this feature then sure we can do it.

filipefurtad0 · 2025-02-07T00:47:12Z

Adding the blocked label till we figure out the best way forward. Discussion ongoing here.

filipefurtad0 added 3 commits January 23, 2025 16:25

Adds a workflow file to have an UI option on GH-Actions

f59a169

This is intended to be used by testers, to generate sample data, on demand

Replaces hard-coded sample data script by input option

61162c6

Replaces DB backup strategy, by baseline saving as introduced in open…

9d4afe9

…foodfoundation#12947

filipefurtad0 self-assigned this Jan 23, 2025

filipefurtad0 marked this pull request as draft January 23, 2025 23:07

filipefurtad0 marked this pull request as ready for review February 3, 2025 19:13

rioug reviewed Feb 3, 2025

View reviewed changes

dacook approved these changes Feb 3, 2025

View reviewed changes

mkllnk reviewed Feb 4, 2025

View reviewed changes

filipefurtad0 added the blocked label Feb 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creates GH Actions section for sample data #13095

Creates GH Actions section for sample data #13095

filipefurtad0 commented Jan 23, 2025

filipefurtad0 commented Jan 23, 2025

rioug Feb 3, 2025

rioug Feb 3, 2025

dacook Feb 3, 2025

dacook Feb 3, 2025

dacook left a comment

dacook Feb 3, 2025

dacook Feb 3, 2025

mkllnk Feb 4, 2025

mkllnk Feb 4, 2025

dacook Feb 4, 2025

filipefurtad0 Feb 4, 2025 •

edited

Loading

dacook Feb 5, 2025

dacook Feb 5, 2025

mkllnk Feb 5, 2025

filipefurtad0 commented Feb 7, 2025

	sudo systemctl stop puma sidekiq
	sudo systemctl start puma sidekiq

	bundle exec rake db:reset ofn:${{ secrets.sample_data_name }}
	bundle exec rake ofn:${{ secrets.sample_data_name }}

Creates GH Actions section for sample data #13095

Are you sure you want to change the base?

Creates GH Actions section for sample data #13095

Conversation

filipefurtad0 commented Jan 23, 2025

What? Why?

What should we test?

Release notes

Dependencies

Documentation updates

filipefurtad0 commented Jan 23, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dacook left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filipefurtad0 Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

filipefurtad0 commented Feb 7, 2025

filipefurtad0 Feb 4, 2025 •

edited

Loading