Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/remove netlify #163

Merged
merged 1 commit into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# Local Netlify folder
.netlify
node_modules
.env
coverage
Expand Down
41 changes: 0 additions & 41 deletions .vscode/launch.json

This file was deleted.

17 changes: 12 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,22 +73,29 @@ COMMITTER_NAME | no | User name of commit creator | Jay Doe
COMMITTER_EMAIL | no | User email of commit creator | [email protected]
AUTHOR_NAME | no | User name of commit author | Jay Doe
AUTHOR_EMAIL | no | User email of commit author | [email protected]
CONFLICT_RETRIES | no | Number of retries, minimum of 2 recommended | 2
CONFLICT_RETRIES | no | Number of retries, minimum of 3 recommended | 3

2. Create a new file in the root directory named `.env` and paste the list of environment variables in, exactly as it appears in the env.template file.

### Usage with manual triggering

1. In the command line, start the Netlify function with:
1. In the command line, deploy local changes to an AWS test environment with:

```sh
npm run dev:netlify
npm run dev
```

This way you can test the netlify function and manually send events to the
`http://localhost:9999/.netlify/functions/metrics`, e.g. using the provided
This way you can test the lambda function and manually send events to the returned URL from the command above, e.g. using the provided
Postman collection.

Once the test environment is not needed anymore the following command needs to be called to delete the test environment:

```sh
npm run dev:destroy
```

To use the test environment the secrets also need to be configured for it, please follow the guide in the [deployment documentation](./docs/DEPLOYMENT.md#secrets).

### Usage with GitHub triggers

> [!WARNING]
Expand Down
1 change: 0 additions & 1 deletion biome.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
"node_modules",
"package-lock.json",
"coverage",
".netlify",
".husky",
"postman",
"cdk.out",
Expand Down
4 changes: 2 additions & 2 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@

## Overall Structure

telemetry-functions is a Node.js application that is designed to be **serverless**. It is written in TypeScript, which allows for type checking and other features that can help catch errors early in the development process. The function is accessed through HTTP endpoints, which are exposed to the internet through a hosting provider (currently Netlify).
telemetry-functions is a Node.js application that is designed to be **serverless**. It is written in TypeScript, which allows for type checking and other features that can help catch errors early in the development process. The function is accessed through HTTP endpoints, which are exposed to the internet through a hosting provider (currently AWS).

The function itself consists of several modules that work together to collect metrics and store them in a database (currently Github has been used as DB).

Expand All @@ -45,7 +45,7 @@ This module is designed to be flexible and can be customized to identify a vario

The "signed event" (_SignedTriggerEvent_) is then sent to the third module, the processing module, "**collectMetrics**", which is responsible for collecting the metrics, normalizing and transforming the data into a standardized format (_EnhancedTriggerEvent_).

Once the event has been processed (_EnhancedTriggerEvent_), the fourth module stores it in a database (currently pushes it to a GitHub repo as JSON file.). _telemetry-functions_ is hosted on Netlify, but this is a temporary solution. In the future, the function will be hosted on a more robust and scalable platform that can handle large amounts of data.
Once the event has been processed (_EnhancedTriggerEvent_), the fourth module stores it in a database (currently pushes it to a GitHub repo as JSON file.). _telemetry-functions_ is hosted on AWS.

### Code base

Expand Down
11 changes: 2 additions & 9 deletions docs/CONTRIBUTE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ Thank you for taking the time to contribute to **telemetry-functions**! We since
- [Metrics Signatures](#metrics-signatures)
- [Core Functions](#core-functions)
- [Store Data](#store-data)
- [OAuth App](#oauth-app)
- [Issues](#issues)
- [Create a new issue](#create-a-new-issue)
- [Solving an Issue](#solving-an-issue)
Expand Down Expand Up @@ -43,7 +42,6 @@ Our project has the following file structure:

.
└── .github // GitLab CI/CD pipeline
└── .netlify // Dist folder created by Netlify CLI
└── aws
└── lambda // Amazon Lambda script
└── lib // Amazon CDK script
Expand All @@ -55,7 +53,6 @@ Our project has the following file structure:
└── shared // Global configuration and log messages
└── trigger-signatures // Trigger identification modules
└── postman // Postman collections
└── netlify // Netlify Serverless functions
└── .env // Env variables
└── CHANGELOG.md // This file is automatically created by the release stage of the main pipeline. Please don't touch it.

Expand Down Expand Up @@ -111,7 +108,7 @@ flowchart TB

### Data Events

telemetry-functions operates on the basis of event objects. An event object (_TriggerEvent_) can be created either automatically by Github, manually, or by a Cronjob. Event objects represent a specific event that occurred, such as a pull request. Regardless of how it is created, event objects are sent to the endpoint URL provided by Netlify.
telemetry-functions operates on the basis of event objects. An event object (_TriggerEvent_) can be created either automatically by Github, manually, or by a Cronjob. Event objects represent a specific event that occurred, such as a pull request. Regardless of how it is created, event objects are sent to the endpoint URL provided by AWS.

### Event Signatures

Expand Down Expand Up @@ -152,14 +149,10 @@ The storeData function is the final step in the telemetry-functions process. sto
- `AUTHOR_NAME`
- `AUTHOR_EMAIL`

These variables are stored in Netlify for production, but when running the app locally, a `.env` file should be added to the root folder. The `.env` file should use the format established in the `/.env.template` file already in the repo.
These variables are stored in GitHub repository variables for production, but when running the app locally, a `.env` file should be added to the root folder. The `.env` file should use the format established in the `/.env.template` file already in the repo.

The variable `CONFLICT_RETRIES` is used to define the number of attempts to store the data in the github repository. If it's not set it defaults to 0, that means there is no retry if the data can't be stored with the first attempt. For testing purposes the variable could also be set to -1, so that the data isn't stored at all.

### OAuth App

In order to access Github data, telemetry-functions uses a Githup OAuth App. This app gets added to a Github repo or organization from which data will be collected, and passes an authorization token to telemetry-functions. The token is then used during REST API requests to Github in for authentication.

## Issues

### Create a new issue
Expand Down
8 changes: 4 additions & 4 deletions docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,10 @@ This chapter will explain everything to know about deployments for telemetry-fun
## Tooling
Telemetry-functions uses [Github Actions](https://github.com/features/actions) and [Release Please](https://github.com/googleapis/release-please) to automate the release process. The team chose these tools in order to eliminate as many human touch points as possible in the process. This is important because telemetry-functions will be worked on by many different people, so automating as much as possible will reduce complexity and help to eliminate errors.

Telemetry-functions uses Netlify as part of its functional architecture. Whenever there is a push on the `develop` branch, Netlify is automatically updated with the latest code and the serverless function is re-deployed.
Telemetry-functions uses AWS as part of its functional architecture. Whenever there is a push on the `main` branch, AWS is automatically updated with the latest code and the serverless function is re-deployed.

## How to deploy
Telemetry-functions uses the [`Release & Publish`]('./../.github/workflows/release.yml) Github action to initiate the deployment process. `Release & Publish` can be triggered manually on Github, but it also runs automatically whenever there is a push to the `main` branch.
Telemetry-functions uses the [`Release`]('./../.github/workflows/release.yml) Github action to initiate the deployment process. `Release` can be triggered manually on Github, but it also runs automatically whenever there is a push to the `main` branch.

The workflow includes the following steps:

Expand Down Expand Up @@ -66,7 +66,7 @@ Multiple instances of Telemetry-functions can be installed side-by-side. Set env
Deployment via CDK requires two steps: Synthesizing and deployment.

### Synthesize
Run `npx aws-cdk synth` to synthesize a AWS CloudFormation template. This script uses the following environment variables, see [README](../README.md#setup) [Please note that the secrets are excluded and not stored as environment variables] for details:
Run `npx aws-cdk synth` to synthesize a AWS CloudFormation template. This script uses the following environment variables, see [README](../README.md#setup) for details [Please note that the secrets are excluded and not stored as environment variables]:

Variable | Required | Example
---------------- | -------- | ----------------------
Expand All @@ -78,7 +78,7 @@ COMMITTER_NAME | no | John Doe
COMMITTER_EMAIL | no | [email protected]
AUTHOR_NAME | no | John Doe
AUTHOR_EMAIL | no | [email protected]
CONFLICT_RETRIES | no | 2
CONFLICT_RETRIES | no | 5

### Deploy
Run `npx aws-cdk deploy --all` to deploy the software to AWS cloud. Add parameter `--require-approval=never` to skip the approval confirmation.
Expand Down
13 changes: 8 additions & 5 deletions docs/PROJECT_BACKGROUND.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# Project Background
## Content

- [Background](#background)
- [Strategy](#strategy)
- [Tech Changes](#tech-changes)
- [Project Background](#project-background)
- [Content](#content)
- [Background](#background)
- [Strategy](#strategy)
- [Tech Changes](#tech-changes)

## Background

Expand All @@ -13,8 +15,9 @@ DEVEN Telemetry will eventually provide services to projects utilizing on Github

## Strategy

The project uses [Netlify](https://www.netlify.com/) as a provider of serverless functions. Netlify hosts the telemetry-functions service and provides an endpoint to receive a POST request when certain events happen. The free tier includes 125,000 requests per month and 100 hours of run time.
The project uses [AWS](https://aws.amazon.com/) as a provider of serverless functions. AWS hosts the telemetry-functions service and provides an endpoint to receive a POST request when certain events happen.

## Tech Changes

Getting project- and repository-level data from Github is possible through two different types of integrations, [*Github Apps*](https://docs.github.com/en/apps/creating-github-apps/creating-github-apps/about-apps#about-github-apps) and [*Github OAuth Apps*](https://docs.github.com/en/apps/creating-github-apps/creating-github-apps/about-apps#about-oauth-apps) (hereinafter referred to as *OAuth Apps*). Github Apps act on their own behalf, while OAuth Apps act on behalf of a user. This means that Github Apps are *installed* on organizations and personal accounts, while OAuth Apps are *invited*. With project requirements stating that nothing should be installed in clients' organizations or repositories, the team chose to create an OAuth App to gather data.
We moved from hosting our telemetry function on [Netlify](https://www.netlify.com/) to [AWS](https://aws.amazon.com/).
For Netlify we only had the free tier and that was not enough for the long term.
45 changes: 0 additions & 45 deletions netlify/functions/metrics.ts

This file was deleted.

Loading