Design and implement activity storing system

[arkid15r]

Is your feature request related to a problem? Please describe.
We get a bunch of different events from GitHub API: issues, PRs, releases, repositories, users.

Describe the solution you'd like
There is a need to standardize this data within a separate entity. Each record should be linked to a project/repo and user (normally we have this data). After indexing it we can use it for project and user timelines on corresponding Nest pages.

Additional context
https://docs.github.com/en/rest/activity/events?apiVersion=2022-11-28#list-public-events

[yashpandey06]

So we ll store this activity in db?

[arkid15r]

So we ll store this activity in db?

Yes, and index it using our index engine.

[yashpandey06]

Okay !

[yashpandey06]

Can I take up this issue ?

[arkid15r]

Yes, just note it's a purely backend tasks that will require dealing with models and sync process updates.

[yashpandey06]

Yeah ,I understand ... this will be my approach broken down in steps .

Note : Earlier i was thinking for going with webhooks for real time updates ...but we already have celery here right ? so we can go with with schedule polling ,just want to make sure which one of the two would be good ?

1. Event Data Model

• Event: GitHub event data linked to repositories and users.
• Repository: Repository metadata.
• User: GitHub user information.

2. Data Fetching

• Fetch events using GitHub API.

3. Data Storage

• Store data in DB.
• Link events to repositories and users.

4. Indexing with Algolia

• Push event data to Algolia indices.

5. Sync Mechanism

• Monitor Any changes
• Sync DB and Algolia.

[arkid15r]

No, we don't have celery and in general I'd prefer subscribing to webhook events instead of polling the API for data in order to stay within the API rate limit. Eventually we're going to have fetch (for historical data) + webhook subscription system for activity. At least that's my vision for now as I haven't looked into implementation details yet.

So I guess you understand the complexity and willing to tackle it. Assigning this to you @yashpandey06