[RFC]: build a developer dashboard for tracking repository build status #39

the-r3aper7 · 2024-03-15T16:30:10Z

I am a 2nd year B.Tech. Student in the department of Computer Science and Technology at
Birla Institute of Technology, Mesra. I started my programming journey from 2019 in High School
i.e 12th Standard.
I am a self learnt programmer who likes to learn new things by creating new projects. Here are
some programming languages and frameworks which I have learn:

Go
Python
JavaScript
TypeScript
Reactjs
Svelte
HTML/CSS
Jest
Stimulus

In College I have completed many courses like Problem Solving with Programming in C,
Object Oriented Programming, Data Structure and Algorithms and many more.
Currently attending courses of Database Management Systems, Design Analysis of Algorithms, Operating System.
My Interests lie in Web Development and creating new things.

Timezone

Asia/Kolkata UTC +0530

Contact details

Email: [email protected]
GitHub: https://github.com/the-r3aper7
Gitter: https://matrix.to/#/@the-r3aper7:gitter.im

Platform

Linux (Fedora)

Editor

For development I use Fedora as Operating System and my editor is VSCode because it is simple to use, learning curve is less and comes with lots build int features like emmet features and extensions. However, I have learned basics of vim for working on servers.

Programming experience

I have significant knowledge of programming, when I first started to learn programming in high school where I started my first language with Python I was fascinated by it. Afterwards I started to explore various fields but Web Development caught my interest. I have created some projects using Django Framework. Then, I started to learn JavaScript so that I can learn React for the front-end. Then I started contributing to Open Source to learn new things and it was great for exploration. I have learned a lot from Open Source and met a lot of amazing people.

JavaScript experience

I've been having a great time learning JavaScript and working with it on front-end projects in open source environments. The best part about JavaScript is that you can literally build anything with it, from mobile apps with frameworks like React Native to web apps, games, and much more. Recently, I even came across a project where they built a whole operating system in the browser called OS.js

Problems with JavaScript is that it has loose typing which can lead to problems if you don't use tools like TypeScript to add type safety. While JSDoc can help with documentation, it doesn't enforce types.However, there is a RFC going on where we can Static Types in JavaScript in Future. Another challenge is callback hell where nested callbacks can make code difficult to read and hard to maintain.

Here are some of my works with JavaScript:

Checkout my personal site which I have created while I was learning Svelte link to Repo.
Checkout Next.js file server which I have created while I was learning Next.js.
Revamped the media page of Qwik checkout the PR.
Small Changes to Sidebar to Qwik checkout the PR.
Another Small Improvement in Qwik's Media Page checkout the PR.
Here is a website which I created for a client which was needed in 24hrs.

Node.js experience

I haven't directly worked with Node.js yet, but I did leverage the API features of Next.js v12 to create a file server interface for a local network. This project allowed for basic upload, download, and streaming functionalities. Check it out on GitHub here: Nextjs File Server

I'm definitely interested in getting more hands-on experience with Node.js in the future.

C/Fortran experience

While I haven't had the chance to create a full project in C yet, I did take a course in Programming for Problem Solving in C during college. This course involved implementing basic algorithms in C, which provided a strong foundation in the language's core concepts.

Interest in stdlib

While I am still developing my knowledge of statistics and machine learning algorithms, I have been impressed by the stdlib-js/stdlib library. It offers a collection of smaller, focused packages instead of a single, massive library that could bloat your bundle size. What particularly excites me is the ability to perform NumPy-like operations directly in the browser. The plotting API seems like a fantastic example of this functionality. I've also found the stdlib community to be very welcoming and approachable.

Version control

Yes, Currently I am using Git as primary VCS.

Contributions to stdlib

Here, are the statuses of some PR's which i have authored.

Status Merged

All PRs which are merged

Status Opened

All Prs which are opened

docs(added readme examples for random/iter): added examples

Issues Opened

All issues which are currently opened

none

Goals

Abstract

Stdlib is a Open Source Project written in JavaScript, Stdlib provides 3000+ high performance packages which can be used for numerical and scientific computation, statistics. Since, it is written in JavaScript we can used it any where some mobile via react-native, Node.js and Browsers. My proposal is to create a developer dashboard for tracking build status of those 3000+ repository. We may want to achieve this using Reactjs and Fastify for backend which will serve static files and API responses.

Description for Project

For my completing my project I am going to divide it into 2 parts:

Roadmap to Status Dashboard.
Roadmap to Historical overviews and per package metrics.

Here are some libraries and tools which will be useful in this project.

Vitejs is a great build tool as it provides various features like code-splitting, optimized builds and faster workflows.
Reactjs is a popular framework for building websites. Since we are in the initial stages we may not want to bloat the application with various technologies. Let's go simple and clear.
postgres we can use this library querying data from database directly in JSON format which we can be easily used for formatting and sending it to frontend for rendering.
Fastify as the name suggests is fast and less overhead for creating REST API's. Fastify Ecosystem provides lots of plugins which will make DX easier and simpler. Since, we will be serving our react app from the same port as Fastify this will avoid the process of CORS.
Tape for testing the REST APIs and server.

Roadmap to Status Dashboard

Here are some designs which I am thinking of for the Status Dashboard.

Index Page

Index Page Explanation

Index Page when filter button is clicked

Explanation

Index Page when data is fetching

Explanation

Index Page when error occur in data fetching

Explanation

Links for the above designs:
This may be the project structure for our frontend part of the website.

Components: from the above design we may use these components initially.
- Table.jsx
- Header.jsx
- Chip.jsx
- Icon.jsx
- Card.jsx
- Modal.jsx
Context: Since, we are making it leaner and simple we may want to use Reactjs Context API to avoid prop drilling. We may store recently fetched data, search query and as project requirements while developing.

Constants: we will store all constants like API urls that may be removed as per suggestion.
Communication with APIs

Since, we are seeking the project to be as simple as possible so we will use the Fetch API which is used by browsers natively for communication. We can create a Component for rendering table data.

Checkout the example component which fetched the data from an API.

NOTE! This is a mockup component for the proposal that might change in the real project.

Now, we easily render table data with changing urls and store the api response to a context as it would be easy to modify the data when a user clicks on Sort Icon as this would be deeply nested in the component tree.
Implementation of REST API
- / (GET): for serving the index.html file.
- /assets (GET): for serving static files like css, js, svgs etc.
- /search (GET): this route serves JSON data for a given query. Here are some search query parameters which it will take.
  - q (required) (type: string) for querying by the package-name.
    Here, are some DX improving shortcuts to search.
    
    Basic Syntax will [header cell/filter type]=[...]
    - if type is string then use ^.
    - if type is number then use -(for range), >=, >, <=, <.
    1. Initially when you type into search bar it will search for package which contains the package name in SQL which will look like this select packagename from packages packagname like '%q%' (NOTE JUST A MOCK QUERY).
    2. Suppose you want to search for package.
      
      Starts with given query you can type name=^name-of-package.
      
      Ends with given query you can type name=name-of-package^.
      
      if you type name=name-of-package this will be treated as default.
      
      Examples: name=^iter
    3. Suppose you want to filter your searches based on stars which is a number.
      
      Range of star you can type stars=1-10
      
      Less than a specific number you can type stars=<10
      
      Greater than or Equal a specific number you can type stars=<=10
    4. You can also combine multiple shortcut which is delimetered by ' ' (space) will look like this.
      
      stars which ranges from 1 to 10, having math in their package name you can type something like name=^math stars=1-10.
  - page (required) (type: integer) which will be useful for pagination.
  - size (required) (type: integer) size of table rows which will render on client side.
  - filters (required) (type: string delimetered by ",") we can use these serve only the data required by the client.
  Response Skeleton
```
{
  data: [
    {
      id: string, // useful for as key while mapping
      name: string,
      stars: number,
      status: string,
      license: string,
      lastModified: string,
      ... // add more to response as filters are applied
    }
    ...
  ],
  nextPage: string, // url with filters, size and q to next page.
  length: number, // current length of data.
}
```
- /packages (GET) this route will the table data.
  - page (required) (type: integer) which will be useful for pagination.
  - size (required) (type: integer) size of table rows which is currently selected.
  - filters (required) (type: string delimetered by ",") to serve only the data required by the client.
  Response Skeleton
```
{
  data: [
    {
      id: string, // useful for as key while mapping
      name: string,
      stars: number,
      status: string,
      license: string,
      lastModified: string,
      ... // add more to response as filters are applied
    }
    ...
  ],
  nextPage: string, // url with filters, size to next page.
  length: number, // current length of data.
}
```

Roadmap to Historical overviews and other drill down metrics.

Here are some designs which I am thinking of for the metrics page.

Overview Page when Downloads is selected.

Overview Page Explained.

Here are some Ideas for Other Metric like Build Failures.

Only failures

Only success

Fusion of Both

Links for the above designs:
- Prototype of the website
Extending Frontend
- Currently for demo purposes I have used Lightweight Charts library by Trading View.
NOTE! this may change as recommended by mentors in Community Bonding Period.
- Adding Polling to Status Dashboard with dropdown menu for selecting the time interval.

Extending API

/metrics/:package-id (GET) which will take these search params.

type (required) (type: string delimetered by ",") this mean we can have downloads, code coverage, build failures and many more .

Response Skeleton

{
  pkgName: string,
  githubURl: string,
  documentation: string,
  data: {
    downloads: [
      { date: string, value: number },
      ...
    ],
    codeCoverage: [
      { date: string, value: number },
      ...
    ],
    buildFailures: [
      { date: string, value: number },
      ...
    ]
    // add more data as defined in type
  },
}

Project Documentation

To ensure the codebase remains well-documented, I plan to document it with each pull request (PR) I submit. This way, the documentation burden stays manageable, especially as deadlines approach. Specifically, I will focus on documenting the changes I introduce in each PR, keeping the overall documentation up-to-date without requiring a separate, time-consuming documentation task.

Why this project?

I am particularly interested in this project because it allows me to use my skills to create a tool that will positively affect the community. Best part of this, I can apply the frontend development skills I have learned through contributions to various open-source projects, along with my experience creating and working with APIs. This project presents a fantastic opportunity to use my knowledge and experience to work on something.

With the help of the developer dashboard, we can quickly view which packages have problems in their build status. This allows us to address issues for specific packages easily. Without the dashboard, you would have to go to every single package repository to check their build statuses. By using the dashboard, we can navigate and work more efficiently.

Qualifications

I've made significant contributions to several well-respected frameworks. Here are some highlights:

Qwik's website a frontend framework which introduced Resumability to frontend world.
- I helped enhance the user experience of Qwik's website by redesigning the Media Page
- I'm also nearing completion on the development of the package manager tabs which will be integrated into production soon.
Wagtail
- Implementing a production-ready url generator copy button using stimulusjs
- Migration of components from vanillajs to stimulusjs here are some migrations enableDirtyFormCheck and tabs.js

In addition to my open-source experience, I'm currently enrolled in a Database Management System (DBMS) course at college. This course is equipping me with the skills to execute queries efficiently and apply optimization techniques, which I'm eager to leverage in this project.

Prior art

Yes, there is similar project like this from npm where they are also tracking the build status, total downloads and many more things. From what I have understand from their Github Repo

They have created a folder named workspaces which contains 2 projects

www from here they are serving simple HTML, CSS, JS and using jquery for DOM manipulation.
data from here they are serving their content from REST service or GraphQL as needed.

All these things are automated by using GitHub Actions.

Commitment

I plan on devoting my entire summer to this. I won't have any other kind of job, so I will be able to work a standard 40 hour week for the coding period. Also, my classes will end well before the coding period begins and will not start up again until well after the pencils down date, so I will be totally free for the whole period.

Schedule

Weeks 1: Community Bonding & Planning

Activities:
- Finalize frontend design with mentors (including user interface (UI) mockups or wireframes).
- Discuss and document API routes and website theme.
- Set up the development environment (including database setup if needed).
Deliverables:
- Approved UI mockups/wireframes.
- Documented API routes and website theme.
- Development environment configured (including database setup, if applicable).

Weeks 2-4: Frontend Development & Core Functionality (Roadmap to Status Dashboard)

Activities:
- Implement core frontend components with clear and concise code.
- Thoroughly document components with comments and per-PR documentation.
- Integrate front-end with APIs (if possible at this stage).
Deliverables:
- Functional frontend components.
- Well-documented code with per-PR documentation.
- (Optional) Initial frontend-API integration.

Weeks 5-7: Midterm and Per Package metrics (Roadmap to Historical overviews and per package metrics)

Activities:
- Conduct a mid-term review with mentors to assess progress and address any concerns.
- Start working on the Package Overview page.
- Continue documenting code throughout development.
Deliverables:
- Fully functional website with core functionality.
- Functional Package Overview page.
- Ongoing code documentation.

Week 8: Performance Optimization

Activities:
- Analyze and optimize database queries using the Postgres library.
- Consider performance optimization techniques for the frontend (e.g., caching, lazy loading).
Deliverables:
- Optimized database queries.
- Improved frontend performance (if applicable).

Weeks 9-11: Testing & Refinement

Activities:
- Write tests for APIs using Tape.
- Fix any bugs identified during testing.
- Continue with code documentation.
Deliverables:
- Comprehensive unit tests for APIs.
- Bug-free website.
- Well-documented codebase.

Week 12: Finalization & Review

Activities:
- Conduct a final review with mentors to ensure project completion aligns with initial goals.
- Document any remaining parts of the codebase.
- Prepare for presentation or deployment (if applicable).
Deliverables:
- Project reviewed and approved by mentors.
- Fully documented codebase.
- Project ready for presentation or deployment.

Notes:

The community bonding period is a 3 week period built into GSoC to help you get to know the project community and participate in project discussion. This is an opportunity for you to setup your local development environment, learn how the project's source control works, refine your project plan, read any necessary documentation, and otherwise prepare to execute on your project project proposal.
Usually, even week 1 deliverables include some code.
By week 6, you need enough done at this point for your mentor to evaluate your progress and pass you. Usually, you want to be a bit more than halfway done.
By week 11, you may want to "code freeze" and focus on completing any tests and/or documentation.
During the final week, you'll be submitting your project.

Checklist

I have read and understood the Code of Conduct.
I have read and understood the application materials found in this repository.
I have read and understood the patch requirement which is necessary for my application to be considered for acceptance.
The issue name begins with [RFC]: and succinctly describes your proposal.
I understand that, in order to apply to be a GSoC contributor, I must submit my final application to https://summerofcode.withgoogle.com/ before the submission deadline.

The text was updated successfully, but these errors were encountered:

Pranavchiku · 2024-03-18T12:55:07Z

Hi @the-r3aper7 suggestions that you may incorporate in your proposal

Please link websites created by you in Javascript Experience section
In version control please state which version control you have worked with
How are planning to fetch data and render on dashboard you create?

How about linking a prototype wireframe to this? This will make things more clear. This is good. Thanks for your interest, looking forward to your contribution.

the-r3aper7 · 2024-03-18T14:02:56Z

Hi @the-r3aper7 suggestions that you may incorporate in your proposal

Please link websites created by you in Javascript Experience section

In version control please state which version control you have worked with

How are planning to fetch data and render on dashboard you create?

How about linking a prototype wireframe to this? This will make things more clear. This is good. Thanks for your interest, looking forward to your contribution.

Thanks @Pranavchiku will implement those suggestions.

kgryte · 2024-03-19T08:31:42Z

A few comments:

How will the table header cell UI element for ordering work? Do you plan to have it work over all table elements or just those displayed?
How will search work? What if I want to search for, say, all repos whose most recent builds are failing? Or which repos have >10 stars? And what if I want to "filter" for, say, all repos having names containing the substring "iter"?
You've chosen to use pagination rather than infinite virtual scrolling. Can you describe why?
Can you describe a bit more, in your own words, the need for this developer dashboard? What problem is trying to solve? Why is creating this dashboard important to the community?
Currently, the website is implemented with React using class components. You've chosen, in your code snippets, to use hooks. Was this intentional? If so, why?

the-r3aper7 · 2024-03-19T10:50:40Z

So What I was thinking was to store a object in context provider like this {key: 'name(default)', orderBy: 'ascending'}.
whenever user clicks on header cell this change the key in context and we will use memoize to sort the data accordingly or create a api route if performance degrades, and now we can render the data which we have fetched initially which has been sorted. That's why I have chosen pagination over infinite scroll. Problem with infinite scroll will be reordering the elements when the fetched data becomes huge and this will degrade the performance of the website.
No, I have not think in such a way It will be really nice implementation for search let me think of query design for this.
explained above
With the help of developer dashboard we quickly view which package is having problem in building such that we can quickly address the problem for that specific package easily and without the dashboard you will have to go to every single package repository to check there build statuses and all by using this we have a easy time navigating and working more efficiently.
It was not purely intentional as I have seen in react.dev says "Class components are still supported by React, but we don’t recommend using them in new code.". Although I have just shared how data fetching will work. If we are opting in for Class Based Components it's fine. I will update the code if you want but with functional based components we will can leverage more code reuse-ability, read-ability and can have more concise and maintainable code and React Team also RECOMMENDED it. I means it purely independent of what we choose, however we will achieve same the results in website.

kgryte · 2024-03-20T06:13:29Z

A few follow-up comments:

Developers are not likely to need to view this on mobile devices, so I don't think that needs to be emphasized or even pursued. We should assume that the 99% use case will be stdlib devs viewing the dashboard from their desktops.
You may want to peruse the code at https://github.com/stdlib-js/www/tree/master/lib/server, which implements a fastify server for serving our API documentation. It may be beneficial for understanding how we'd prefer the server code be organized.

kgryte · 2024-03-20T06:15:47Z

Also, do you have mock-ups for displaying drill down metrics on a per-package basis?

steff456 · 2024-04-01T00:30:20Z

Hi @the-r3aper7, thanks for opening this draft proposal. I just have one additional question:

Status graphs views are going to be created only for the selected package?

the-r3aper7 · 2024-04-01T02:53:47Z

Hi @the-r3aper7, thanks for opening this draft proposal. I just have one additional question:

Status graphs views are going to be created only for the selected package?

Yes as of now I thought graphs views are for the selected package.

the-r3aper7 added 2024 2024 GSoC proposal. rfc Project proposal. labels Mar 15, 2024

kgryte closed this as completed Apr 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC]: build a developer dashboard for tracking repository build status #39

[RFC]: build a developer dashboard for tracking repository build status #39

the-r3aper7 commented Mar 15, 2024 •

edited

Loading

Here are some designs which I am thinking of for the Status Dashboard.

Index Page

Index Page Explanation

Index Page when filter button is clicked

Explanation

Index Page when data is fetching

Explanation

Index Page when error occur in data fetching

Explanation

This may be the project structure for our frontend part of the website.

Communication with APIs

Implementation of REST API

Basic Syntax will `[header cell/filter type]=[...]`

Response Skeleton

Response Skeleton

Here are some designs which I am thinking of for the metrics page.

Overview Page when Downloads is selected.

Overview Page Explained.

Here are some Ideas for Other Metric like `Build Failures`.

Only failures

Only success

Fusion of Both

Extending Frontend

Extending API

Response Skeleton

Pranavchiku commented Mar 18, 2024

the-r3aper7 commented Mar 18, 2024

kgryte commented Mar 19, 2024

the-r3aper7 commented Mar 19, 2024 •

edited

Loading

kgryte commented Mar 20, 2024

kgryte commented Mar 20, 2024

steff456 commented Apr 1, 2024 •

edited

Loading

the-r3aper7 commented Apr 1, 2024

[RFC]: build a developer dashboard for tracking repository build status #39

[RFC]: build a developer dashboard for tracking repository build status #39

Comments

the-r3aper7 commented Mar 15, 2024 • edited Loading

TABLE OF CONTENTS

Introduction

Full name

University status

University name

University program

Expected graduation

Short biography

Timezone

Contact details

Platform

Editor

Programming experience

JavaScript experience

Node.js experience

C/Fortran experience

Interest in stdlib

Version control

Contributions to stdlib

Goals

Abstract

Description for Project

Roadmap to Status Dashboard

Here are some designs which I am thinking of for the Status Dashboard.

Index Page

Index Page Explanation

Index Page when filter button is clicked

Explanation

Index Page when data is fetching

Explanation

Index Page when error occur in data fetching

Explanation

This may be the project structure for our frontend part of the website.

Communication with APIs

Implementation of REST API

Basic Syntax will [header cell/filter type]=[...]

Response Skeleton

Response Skeleton

Roadmap to Historical overviews and other drill down metrics.

Here are some designs which I am thinking of for the metrics page.

Overview Page when Downloads is selected.

Overview Page Explained.

Here are some Ideas for Other Metric like Build Failures.

Only failures

Only success

Fusion of Both

Extending Frontend

Extending API

Response Skeleton

Project Documentation

Why this project?

Qualifications

Prior art

Commitment

Schedule

Checklist

Pranavchiku commented Mar 18, 2024

the-r3aper7 commented Mar 18, 2024

kgryte commented Mar 19, 2024

the-r3aper7 commented Mar 19, 2024 • edited Loading

kgryte commented Mar 20, 2024

kgryte commented Mar 20, 2024

steff456 commented Apr 1, 2024 • edited Loading

the-r3aper7 commented Apr 1, 2024

the-r3aper7 commented Mar 15, 2024 •

edited

Loading

Basic Syntax will `[header cell/filter type]=[...]`

Here are some Ideas for Other Metric like `Build Failures`.

the-r3aper7 commented Mar 19, 2024 •

edited

Loading

steff456 commented Apr 1, 2024 •

edited

Loading