Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace notebooks with pipelines that use SQL Runner #9

Closed
inglesp opened this issue Aug 3, 2022 · 3 comments
Closed

Replace notebooks with pipelines that use SQL Runner #9

inglesp opened this issue Aug 3, 2022 · 3 comments

Comments

@inglesp
Copy link
Contributor

inglesp commented Aug 3, 2022

This is more general issue around notebooks that run on the server, that encompasses some others, including:
opensafely-core/cohort-extractor#830
opensafely-core/cohort-extractor#831

There are several notebooks that are intended to be run on the server, including the database notebooks, and the ones in this repo. Applications and status of notebooks is described in this doc.

The current process for running the database notebooks is described in the team manual, and in the raw-data-plausibility-checks repo README for other notebooks.

We need to

  1. Define a common process for running notebooks on the server and how/where to publish outputs
  2. Find a way to automate the running of certain notebooks - typically ones that should rerun after a database build (database notebooks, vaccine names notebook, maybe others?). This could possibly be triggered by a database build, if we can identify that - perhaps by job-runner going into maintenance mode? Or if the notebooks are not too heavy-duty, a weekly (or whatever) cron job.

Will has started a discussion in #issue-db-notebooks-release and also in this thread.

@inglesp inglesp changed the title Centralise and automate notebooks on server(s) Replace notebooks with pipelines that use SQL Runner Aug 3, 2022
@inglesp
Copy link
Contributor Author

inglesp commented Aug 3, 2022

Was a draft issue by @rebkwok

@inglesp
Copy link
Contributor Author

inglesp commented Sep 26, 2022

After discussion with @wjchulme, we should start with the TPP schema notebook.

@iaindillingham
Copy link
Member

I've broken this issue into several smaller issues. First, some conversion issues:

Then, some process issues:

Then, a governance issue:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

2 participants