Use a single source for test sites' content

[HussainTaj-arbisoft]

Description/Context

Our Playwright tests use a set of test sites. These test sites are located on RC as well as inside the ocw-hugo-themes repo, as markdown content. From what we see on RC, we were meant to keep these manually synced.

I believe that changes to the test sites should always be made on RC first. We should remove the need for a manual step, which I believe is the cause of this inconsistency. I propose that we use the GitHub content, created via Studio RC, as the source for the test sites in ocw-hugo-themes.

Plan/Design

To achieve this objective, we will need to perform the following tasks.

• Sync/update the RC site with the content we currently have in the themes repo.
• Remove test sites from the themes repo and ignore them in source control.
• Update the test:e2e command to automatically pull the test sites when needed.
• Update GitHub actions to set up the appropriate secret required to pull the repos.

I don't know why we initially chose to duplicate the content. I might have missed something, so whoever works on this might need to refine the plan accordingly.

[HussainTaj-arbisoft]

I just realized that the reason why we might duplicate content is to avoid conflicts when two different engineers work on the same set of tests.

In that case, what if instead of storing the entire content we only store the commit hash?

[HussainTaj-arbisoft]

We had a short discussion on this.

The primary challenge with the proposed approach is the occasional DB sync with production. These syncs will end up destroying the studio DB and these test courses.

The current favored approach is to use the themes repo's test site content as the source, instead of the content repo's. To keep them in sync, it is proposed that whenever a change in the themes repo happens, we update the content repos via some workflow.

[gumaerc]

@HussainTaj-arbisoft I know this is an old discussion now, but I thought I'd update it with some new information. The end to end test pipeline has been released for a little while now and is running in RC and production. You are correct in that a database restore will wipe out the test courses, along with the rest of the database, but I think that's not as big of an issue as it might seem.

• Let's say you are making some changes to the test site's content. You're going to first test that locally by adding it to your local instance of ocw-studio and running the pipeline in your local Concourse against that. When a PR is ready to be merged, part of that process would be to run the export_test_sites management command, which serializes the test sites as JSON so the changes can be checked into the repo. When the PR is merged, the loaddata management command will need to be run on RC to import the new fixtures, followed by running sync_website_to_backend¹. After testing in RC, you would release the PR and then run the same process in production to import the fixtures. Therefore, if a database restore were to happen, nothing would be wiped out because the same changes to the test site were made in every environment and production is the last one in the chain. The only potential issue with this that I can think of at the moment is if a database restore were to happen while testing was in progress on RC. In which case, you would just run the management commands to push up the fixtures again and all is well.
• I think that in ocw-hugo-themes, we should make the test site repos configure via env variable. We already have code in there that can clone a course content repo and spin it up locally, why not use the same functionality to pull down the test sites? We could set up Github actions to pull them from RC as well, since it already has credentials for doing that to create the Netlify preview builds. This way, you could use the ocw-studio UI to add the new test course content in the same way content is added to any other OCW site. If I think about the process we would take to add a new type of content to a type of OCW site, it would be along the lines of:
  • Define what the properties of the content are and create an ocw-hugo-projects PR to add it to the ocw-studio.yaml config for that site type
  • Import the new starters into ocw-studio, where new test content can now be created in the test site
  • Make a PR to commit the fixtures for the new test site content in ocw-studio
  • Once the fixtures are released and deployed in every environment, create an ocw-hugo-themes PR to implement functionality that actually utilizes the content, and create Playwright tests to verify that it is rendered correctly
  • As the PR is released, the end to end test pipeline automatically runs each stage of the way against the test site content in each environment all the way from Github actions to production

Footnotes

1. A missing piece here is that the test pipeline pulls site content from the release branch, and there's currently no way to push content up to the release branch without triggering a pipeline and pushing it live. I think what we should do in this case is change the site pipeline definition to look at the OCW_WWW_TEST_SLUG and OCW_COURSE_TEST_SLUG vars and if the current site matches either of them, automatically use the test buckets instead. This would mean we can turn on the UI for the test courses without fear of the content ever being published on the live site. ↩