Use pytest-env to set up API key shims for testing

[AetherUnbound]

Problem

Presently we use a copy of the env.template file when testing (via the just dotenv recipe here in the CI/CD workflow)

The env template includes placeholders for the various API keys we use:

https://github.com/WordPress/openverse-catalog/blob/895e907b6f9594b4d43db6a07eae0b3cb4a0e7d0/env.template#L27-L34

Per @krysal's comment (WordPress/openverse-catalog#795 (comment)):

Do you think we should let these variables with no value instead of the "not_set" string? 🤔 That way the scripts that require it will fail with a more clear error, before even trying to make a request, but as it's currently they take the string as the API key.

Description

We should comment these fields out from the env.template file but still keep them present as examples (just like the env.template files for the API). Particularly now that we are no longer retrieving those values at DAG parse time as part of the provider refactor, we no longer need to have these defined in the environment whenever running the project. Leaving them undefined will give us more obvious errors during local DAG testing if we happen to forget to populate a variable.

We do need placeholder values for testing though. That's where we might use the pytest-env plugin to define these with placeholders in the pytest.ini file.

Implementation

• 🙋 I would be interested in implementing this feature.

[Pmeet]

Hi @AetherUnbound , I'd like to work on this. But can you elaborate on why we need placeholder values for testing?
I am asking because before that you said you need the obvious errors that will generate when testing.

[AetherUnbound]

Sure @Pmeet, I'll assign it to you! And great call - I think not_set works great as-is. The obvious errors situation is more for the occasions where we're running the DAGs locally ourselves manually rather than during our CI/CD or unit test suite.

[ngken0995]

@AetherUnbound What are the placeholder value in pytest.ini?

[AetherUnbound]

@ngken0995 any value will do! You can just use "placeholder" or "apikey" for testing

[ngken0995]

@AetherUnbound I have commented all AIRFLOW_VAR_API_KEY from env.template and add them to pytest.ini . Also, install pytest-env and pipfile and pipfile.lock was updated automatically. What is the best way to test the changes?

[AetherUnbound]

I believe if the tests run successfully, that should be sufficient! The tests will fail otherwise if the API keys aren't defined in the env.template.