Skip to content

feat: create github repo on course creation #615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: create github repo on course creation #615

wants to merge 8 commits into from

Conversation

marslanabdulrauf
Copy link
Contributor

@marslanabdulrauf marslanabdulrauf commented Sep 18, 2025 •
edited
Loading

What are the relevant tickets?

https://github.com/mitodl/hq/issues/6962

Description (What does it do?)

This PR:

  1. adds receivers to create Github repo(if enabled) on course creation signal
  2. creates Github repo(if enabled) on course re-run creation

Screenshot

image

How can this be tested?

  1. Install and configure ol_openedx_git_auto_export
  2. Run Migration in CMS: tutor dev exec cms ./manange.py cms migrate

Testing management command

  1. You must have a course with giturl set in its advance settings -- It shouldn't have gitrepo entry in admin
  2. Run Management command tutor dev exec cms ./manage.py cms migrate_giturl
  3. Verify that the giturl is migrate to CourseGitRepo model
  4. Make sure that giturl is migrated for re-run courses as well

Auto repo creation testing

  1. Install and configure ol_openedx_git_auto_export
  2. Make sure to:
    • Enable ENABLE_AUTO_GITHUB_REPO_CREATION
    • Set GITHUB_ORG_API_URL
    • Set GITHUB_ACCESS_TOKEN -- This token should have repo read/write and pull/push access.
  3. Create new course and verify from your organization that a new repository is created.
  4. Check admin and see if the entry is added for this newly created course
  5. Create re-run of any course and repeat steps 3 and 4 to verify
  6. Make any change and publish to make sure you are getting latest course changes/commits

@marslanabdulrauf marslanabdulrauf marked this pull request as ready for review September 19, 2025 12:05
pdpinch
pdpinch reviewed Sep 19, 2025
View reviewed changes
src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/signals.py Outdated
"Course published with auto-export enabled. Starting export... (course id: %s)", # noqa: E501
course_key,
)
# HACK: To create auto git repo for Re-runs as it does not emit COURSE_CREATED signal # noqa: E501 FIX004
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a bug that reruns don't emit COURSE_CREATED signal?

Is there a different signal emitted when a rerun is created? @asadali145

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a different signal emitted when a rerun is created? @asadali145

I am not sure about a specific signal, but we listen for the post_save signal on changes in CourseRerunState model. It actually tracks the status of reruns, like in_progress, failed, and completed. When a rerun state is completed, then we update our course sync mappings.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I discussed this with @marslanabdulrauf and @arslanashraf7; reruns do not trigger the COURSE_CREATED signal, which looks like a bug to me. The reason is that course reruns follow a completely different code path, and that never sends a COURSE_CREATED signal.

We can listen for CourseRerunState post save here like we do for the course sync plugin.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @asadali145 I will update it to catch course re-run creation through post_save signal 👍

@pdpinch
Copy link
Member

pdpinch commented Sep 19, 2025 •
edited
Loading

When we create a repo, I want to make sure that it is set to private in Github. Course exports are considered sensitive since they can contain exam solutions. This is critical, so should be in this PR.

If possible, I'd also like to set the repo description at creation time, with a string like f"{display_name}, exported from {studio_url}" so repo users can easily see which repo goes with which course and can link to it in Studio (this could be a follow-up PR).

@cachob anything else we learned recently about creating repos?

@marslanabdulrauf
Copy link
Contributor Author

The repository is set to Private.

I can update the repository description, current set to very minimal description, can be seen from attached SS as well:

image

pdpinch
pdpinch reviewed Sep 19, 2025
View reviewed changes
src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/signals.py Outdated
}
payload = {
"name": course_id_slugified,
"description": f"Git repository for {course_id!s}",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do something like f"{display_name}, exported from {studio_url}" so repo users can easily see which repo goes with which course and can link to it in Studio?

pdpinch
pdpinch reviewed Sep 19, 2025
View reviewed changes
src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/signals.py Outdated

def listen_for_course_created(**kwargs):
course_id = kwargs.get("course").course_key
course_id_slugified = slugify(str(course_id))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you give an example of the output of slugify for a typical course key?

I'm concerned that we might be dropping all the +s, which act as delimiters and make it easier to read and parse.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Github converts all +, / or : to - and there could be other characters, I haven't tested it for all of them. So I slugify it. Slugify returns course-v1xprogh101101_1 for course-v1:xpro+gh101+101_1

image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested it with all special characters that I could type from my laptop and other than ., - and _ it converts all of them into -
Let me add a formatter to convert course_id into Github supported format, it will increase readability

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also drop the course-v1- from the start of the repo name? It will be the same for all repos. (unless maybe someday there is a course-v2).

@marslanabdulrauf
Copy link
Contributor Author

@pdpinch I updated the repository name and added a screenshot of repo page in PR description

@asadali145 asadali145 self-assigned this Sep 22, 2025
asadali145
asadali145 requested changes Sep 22, 2025
View reviewed changes
Copy link
Contributor

@asadali145 asadali145 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a high-level code review. I will proceed with the testing in the next review.

src/ol_openedx_git_auto_export/README.rst
"FEATURES": {
"ENABLE_EXPORT_GIT": true,
"ENABLE_AUTO_GITHUB_REPO_CREATION": true # Optional, to auto create github repo for new courses
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we create a waffle flag? A feature flag will require a code change in ol-infra to enable/disable it, followed by redeployment.

src/ol_openedx_git_auto_export/README.rst Outdated
}
# Set when ENABLE_AUTO_GITHUB_REPO_CREATION is true
GITHUB_ORG_API_URL = "https://api.github.com/orgs/<GITHUB_ORG_NAME>" # For GitHub Enterprise, change the URL accordingly
GITHUB_ACCESS_TOKEN = "<GITHUB_PERSONAL_ACCESS_TOKEN>" # Token must have repo creation permissions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we list down the steps and exact permissions required?

src/ol_openedx_git_auto_export/README.rst Outdated
----------------------
- Open studio then course and go to advanced settings.
- Choose field GIT URL and add you OLX git repo. For example ``[email protected]:<GITHUB_USERNAME>/edx4edxlite.git``.
- Open studio `admin <http://studio.local.openedx.io:8001/admin/ol_openedx_git_auto_export/coursegitrepo/>`_
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Open studio `admin <http://studio.local.openedx.io:8001/admin/ol_openedx_git_auto_export/coursegitrepo/>`_
- Open studio admin at `/ol_openedx_git_auto_export/coursegitrepo/`

src/ol_openedx_git_auto_export/README.rst Outdated
- Open studio then course and go to advanced settings.
- Choose field GIT URL and add you OLX git repo. For example ``[email protected]:<GITHUB_USERNAME>/edx4edxlite.git``.
- Open studio `admin <http://studio.local.openedx.io:8001/admin/ol_openedx_git_auto_export/coursegitrepo/>`_
- Add your course_id and in GIT URL add you OLX git repo. For example ``[email protected]:<GITHUB_USERNAME>/edx4edxlite.git``.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Add your course_id and in GIT URL add you OLX git repo. For example ``[email protected]:<GITHUB_USERNAME>/edx4edxlite.git``.
- Add your course_id and in the GIT URL, add your OLX git repo. For example ``[email protected]:<GITHUB_USERNAME>/edx4edxlite.git``.

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/admin.py Outdated
list_per_page = 20


admin.site.register(CourseGitRepo, CourseGitRepoAdmin)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can use the @admin.register decorator here.

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/utils.py Outdated
Comment on lines 84 to 89
"""Create a GitHub repository for the given course key.
Args:
course_key (CourseKey): The course key for which to create the repository.
Returns:
str or None: The SSH URL of the created repository, or None if creation failed.
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"""Create a GitHub repository for the given course key.
Args:
course_key (CourseKey): The course key for which to create the repository.
Returns:
str or None: The SSH URL of the created repository, or None if creation failed.
"""
"""
Create a GitHub repository for the given course key.
Args:
course_key (CourseKey): The course key for which to create the repository.
Returns:
str or None: The SSH URL of the created repository, or None if creation failed.
"""

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/utils.py Outdated
return repo_name.replace("course-v1-", "")


def create_github_repo(course_key):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should move the repo creation to a Celery task.

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/utils.py Outdated
)
return None

# SignalHandler.course_published is called before COURSE_CREATED signal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a valid comment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I faced issues but its not valid now as we have moved repo creation to post_save signal

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/utils.py Outdated
"description": repo_desc,
"private": True,
"has_issues": False,
"has_project": False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct key is has_projects

src/ol_openedx_git_auto_export/ol_openedx_git_auto_export/utils.py Outdated
"auto_init": True,
}
response = requests.post(url, headers=headers, json=payload, timeout=30)
if response.status_code != 201: # noqa: PLR2004
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can use restframework status.201

@asadali145
Copy link
Contributor

We discussed this in the morning, and I am posting here on behalf of @marslanabdulrauf as he is out sick and could not post this comment.

@pdpinch Do we want to auto-create GitHub repos for all the new courses, including UAI and non-UAI courses?

If the goal is to create repositories only for UAI courses, would it make sense to use UAI_ as the standard prefix for repo creation logic, or did you have a different approach in mind?

CC: @marslanabdulrauf @arslanashraf7

@pdpinch
Copy link
Member

pdpinch commented Sep 24, 2025

Do we want to auto-create GitHub repos for all the new courses, including UAI and non-UAI courses?

All courses should get a repo, UAI and non-UAI.

In particular, we need to have version control of all UAI_SOURCE courseruns.

@marslanabdulrauf
Copy link
Contributor Author

Thanks for clarifying @pdpinch

I have also updated migration command to:

  1. Migrate giturl from advance settings to our custom managed table CourseGithubRepository
  2. Create new repo for courses with duplicate giturl
  3. Create new repo if the course does not have any giturl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdpinch pdpinch pdpinch left review comments

@asadali145 asadali145 asadali145 requested changes

Assignees

@asadali145 asadali145

Labels
Needs Review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@marslanabdulrauf @pdpinch @asadali145