Dockerization, Poetry, and Github Actions

[ibrahimjaved12]

What are the relevant tickets?

Closes https://github.com/mitodl/hq/issues/3328
Closes https://github.com/mitodl/hq/issues/3327

Description (What does it do?)

This PR adds Docker, Poetry, and Github Action to run Unit tests.

Note: We're relying on docker-compose run and docker run commands instead of docker-compose up because tqdm progress bars do not work as expected with docker-compose up. They only log when they're at 100% with docker-compose up.

How can this be tested?

1. Checkout to this branch
2. Build the container:

docker compose build

3. Start the container:

docker compose run --rm app

4. To generate a JSON file containing complete API data:

docker compose run --rm app --create_json

5. To create a CSV file from the local JSON file:

docker compose run --rm app --create_csv --source=json

6. To run unit tests:

docker run --rm ocw_oer_export python -m unittest discover

[blarghmatey]

The docker, Poetry, and GH Actions pieces look good to me. I'll defer to someone else to comment on the renaming of variables.

[ibrahimjaved12]

The docker, Poetry, and GH Actions pieces look good to me. I'll defer to someone else to comment on the renaming of variables.

Thanks Tobias!

[gumaerc]

This worked great once I resolved one small issue, noted below. Otherwise, the code and output look good.

[gumaerc]

When I ran this, the output folder that was created was owned by root. The script then failed to save any files in the folder with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/src/ocw_oer_export/cli.py", line 38, in <module>
    main()
  File "/src/ocw_oer_export/cli.py", line 30, in main
    create_csv(source=args.source)
  File "/src/ocw_oer_export/create_csv.py", line 190, in create_csv
    with open(output_path, "w", newline="", encoding="utf-8") as csv_file:
PermissionError: [Errno 13] Permission denied: '/output/ocw_oer_export.csv'

This is a fairly common issue with Docker. I was able to chown the folder to be owned by my current user, then run the commands. What I'd recommend here is creating a private folder, creating an output folder within that, committing both folders to git, then adding private to .gitignore after it's committed. You'll then need to change the output paths to ./private/output/. This will ensure the folder is created as the current user when the repo is cloned, avoiding this issue. Placing the output folder in an ignored folder called private allows the possibility of adding other gitignored folders in the future if necessary.

[ibrahimjaved12]

I committed the directories, using a .gitkeep and then removing it later after adding the directory to .gitignore. See if it works now?

[ChristopherChudzicki]

You can't remove the .gitkeep, because then the directory is empty, and git won't keep it. I'd add a comment in the gitkeep file, too.

[gumaerc]

@ibrahimjaved12 I'm not seeing the private/output folder when I pull down the latest changes. It looks like the .gitkeep file was added in 7a42ab2 and removed in c2d9dac?

[gumaerc]

👍

Everything runs great from a fresh state after adding the private folder