Develop an automated report on Study Group Activity

[auremoser]

Description

Right now, stats about Study Groups globally are scattered between multiple repositories (the Study Group Project Website, the Study Group Lessons Repo, the Orientation Gitbook, just to name a few).

Process

Right now, the process is somewhat manual:
For collecting events or activity data on Study Groups:

1. Aurelia runs the scripts in the sg-events to collect data on forks.

For outreach to new Study Groups:

1. Aurelia gets a ping from Zapier each time a new fork is made to the Study Group Project.
2. Aurelia reaches out to the Study Group forker via issues on their forked repository with a welcome message, pointing to the Study Group Orientation Guide.
3. Aurelia manually logs their repository URL and contact information in a spreadsheet, and updates the map on our website.

For reporting back about Study Group Events:

1. Person hosts an event for their study group and adds it to their Study Group Site
2. Person (optionally) enters in information about their event via the form linked on the Study Group Events Page

For sourcing events material:

1. Person hosts event but needs ideas for how to structure the workshop or tutorial they would like to host
2. Person visits the Study Group Lessons repo for scripts, outlines, and ideas on what types of sessions they can host
3. Person (optionally) contributes back their own lessons via the issues in the lessons repo

Expected result

A unified set of scripts that pulls contributor data from these repos and compiles it into a dataset or short report would be lovely, and would greatly help us showcase what our Study Groups are doing, and support them locally.

Current result

We pull and store some data from github in the sg-events repository, and have a geojson dataset that allows us to populate a map on our website of where active study groups are located.

Possible Fix

However possible, it would be great to streamline the process for collecting data about new forked Study Group Projects and their activity, members and events for display on a visualization dashboard here.

---

References

• Study Group Project Website
• Study Group Lessons Repo
• Study Group Orientation Gitbook

[lvyachao]

I have a question here: Why you build such a "patch-like" scripts like sg-events instead of add some feature in core project itself. For example, how about added a special function into main repository, letting every repository can send a POST(or JSON or whatever) to a same mirco server, and that mirco server keep monitoring every POST and doing data combination and virualiztion. Then you don't even need to "monitor" new ones and keep crawling everything....I just wonder is there any privacy concerns?

[auremoser]

Yes @lvyachao , that is why we submitted a GSOC project, to tidy up that workflow!

It gets complicated in github when you have forks and want to monitor them regularly. I get alerts via Zapier when people fork the Study Group repo, and then followup with them individually, but regular alerts when each forks add events would be untenable, since there are around 300 forks now.

The separate sg-events repo was an attempt to collect quick analytics on our Study Group forks occasionally, and I try not to complicate the base repo (/studyGroup) because it can be confusing for new Study Group leads to see scripts they aren't relevant to the maintenance of their actual jekyll study group.

Hopefully that makes sense!

[lvyachao]

Is that possible to sent notices to all the forks letting them to patch a new feature(Or just send them a single JS file and let them to include it into main index). After all, I think that's a dead-loop if we add one "patch" again and again dealing with the manually jobs. If we add the features directly like I mention on every patch, the separate sg-events repo will be no longer needed. Of course if you think we have no choice but still need to use sg-events doing those jobs, I can rewrite my proposals now. (That's even easier for me but I think there are still potential problems exist in the future)

[auremoser]

Yes, you can send an email to all of the Study Group leads, though this is not automated through Github and comes from an email list that they subscribe to after they fork the repo. We suggest that they subscribe in the Study Group Orientation and in the README, but they need to "opt-in" per Mozilla's policy about letting participants decide that they want to be a part of the community. In the past though, we've only requested those federated pull changes a few times, and it's a bit messy, as not everyone does it, those forks become outdated, and later, if Github changes the structure of pages, it might be harder to retrofit older repositories.

You are welcome to propose any fix that you think could be feasibly accomplished in 3 months of work, for GSOC. I'm not opposed to well-thought-out new ideas. I just made the separate sg-events repo as a temporary solution, to keep the mentorship process cleaner until all changes can be streamlined to update all of the base repo forks.

[lvyachao]

Well I think I understand the wired about complex issues now. I'd like to say if we want to shade a better light for this project, update our current main repo is necessary for the future, which is I want to include in my proposals as main target. For those older repositories, I already read carefully sg-events code and I think I can "make it more automatically" as my second target. A new feature for new repo(who agree with sharing the data) and sg-events for old repo is not conflict actually :)... We can do both on the same time until sg-events finally retire....

[harshavamsi]

@auremoser What I can think of is, let the sg-events script continue and add new functionality to it though the Github API. Store all the data in a JSON file and any new forks already not present in the JSON file will be retrieved each time. This file can then be loaded statically into a visualization library to update the charts. The sg-events script can be a cron event in the server.