Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mergebot #1023

Merged
merged 16 commits into from
Jul 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ packages/publisher/output
**/testsource/**
**/fixtures/**
packages/eslint-plugin/test/fixtures
packages/mergebot/src/queries/schema
6 changes: 5 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,16 @@ jobs:
run: pnpm build
- name: test
run: pnpm test
- name: Bundle mergebot
working-directory: packages/mergebot
run: pnpm run bundle
- name: Check mergebot bundle
run: BOT_AUTH_TOKEN=secret node packages/mergebot/dist/functions/index.js
- name: knip
run: pnpm knip
check-parse-results:
needs: build_and_test
uses: ./.github/workflows/check-parse-results.yml

changesets:
name: changesets
runs-on: ubuntu-latest
Expand Down
33 changes: 33 additions & 0 deletions .github/workflows/daily.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
name: Daily Open PR Sync

on:
workflow_dispatch: ~
schedule:
- cron: '37 */6 * * *'

jobs:
build:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2
with:
node-version: '20.x'
- uses: pnpm/action-setup@fe02b34f77f8bc703788d5817da081398fad5dd2 # v4.0.0
- run: pnpm install
- run: pnpm run build

# Go through all open PRs and run the bot over them
- run: node packages/mergebot/dist/run.js
env:
TYPESCRIPT_BOT_TOKEN: ${{ secrets.TYPESCRIPT_BOT_TOKEN }}

keepalive-job:
name: Keepalive Workflow
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: gautamkrishnar/keepalive-workflow@05456e7809058d586d96392e99217726ccc10076 # v2.0.5
83 changes: 83 additions & 0 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
name: Deploy mergebot to Azure

on:
push:
branches:
- main
workflow_dispatch:

env:
NODE_VERSION: '18.x' # set this to the node version to use (supports 8.x, 10.x, 12.x)
RESOURCE_GROUP_NAME: dtmergebot3
FUNCTION_APP_NAME: dtmergebot3
STORAGE_ACCOUNT_NAME: dtmergebot3b1b7
STORAGE_CONTAINER_NAME: deployment
FUNCTION_ZIP_NAME: function.zip

defaults:
run:
shell: bash

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6

- uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2
with:
node-version: ${{ env.NODE_VERSION }}

- uses: pnpm/action-setup@fe02b34f77f8bc703788d5817da081398fad5dd2 # v4.0.0

- name: Install and test
run: |
pnpm install
pnpm run --if-present build
pnpm run --if-present test
- name: Bundle
working-directory: packages/mergebot
run: pnpm run bundle

- name: Check bundle
run: BOT_AUTH_TOKEN=secret node packages/mergebot/dist/functions/index.js

- name: Create zip
working-directory: packages/mergebot
run: zip -r ${{ env.GITHUB_WORKSPACE }}/${{ env.FUNCTION_ZIP_NAME }} dist host.json package.json

- name: Upload artifact for deployment job
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
with:
name: node-app
path: ${{ env.FUNCTION_ZIP_NAME }}

deploy:
runs-on: ubuntu-latest
needs: build
environment:
name: 'Production'
permissions:
contents: read
id-token: write
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
id-token: write
contents: read
id-token: write


steps:
- name: Download artifact from build job
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
name: node-app

- uses: azure/login@6c251865b4e6290e7b78be643ea2d005bc51f69a # v2.1.1
with:
client-id: ${{ secrets.AZURE_CLIENT_ID }}
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

- name: Upload blob
run: az storage blob upload -f ${{ env.FUNCTION_ZIP_NAME }} --account-name ${{ env.STORAGE_ACCOUNT_NAME }} -c ${{ env.STORAGE_CONTAINER_NAME }} -n ${{ env.FUNCTION_ZIP_NAME }} --overwrite true --auth-mode login

# - name: Set package path
# run: az functionapp config appsettings set -g ${{ env.RESOURCE_GROUP_NAME }} -n ${{ env.FUNCTION_APP_NAME }} --settings WEBSITE_RUN_FROM_PACKAGE="https://${{ env.STORAGE_ACCOUNT_NAME }}.blob.core.windows.net/${{ env.STORAGE_CONTAINER_NAME }}/${{ env.FUNCTION_ZIP_NAME }}"

- name: Restart app
run: az functionapp restart -g ${{ env.RESOURCE_GROUP_NAME }} -n ${{ env.FUNCTION_APP_NAME }}
7 changes: 6 additions & 1 deletion .knip.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,13 @@
"entry": ["src/index.ts", "src/types/*.ts"],
"project": "**/*.ts",
"ignoreDependencies": ["@qiwi/npm-types"]
},
"packages/mergebot": {
"entry": ["src/functions/index.ts", "src/run.ts", "src/commands/create-fixture.ts", "src/commands/update-all-fixtures.ts", "src/commands/update-test-data.ts", "src/scripts/updateJSONFixtures.ts"],
"project": "**/*.ts",
"ignore": ["src/queries/schema/graphql-global-types.ts"]
}
},
"ignore": ["**/fixtures/**", "**/testsource/**", "**/dtslint/test/*/**", "**/*.d.ts"],
"ignore": ["**/dist/**", "**/fixtures/**", "**/testsource/**", "**/dtslint/test/*/**", "**/*.d.ts"],
"ignoreExportsUsedInFile": true
}
2 changes: 1 addition & 1 deletion .npmrc
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
prefer-workspace-packages=true
engine-strict=true
engine-strict=false
disallow-workspace-cycles=true
2 changes: 1 addition & 1 deletion jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ module.exports = {
preset: "ts-jest",
testEnvironment: "node",
modulePathIgnorePatterns: ["packages\\/publisher\\/output", "testsource", "fixtures"],
testMatch: ["<rootDir>/packages/*/test/**/*.test.ts", "<rootDir>/packages/dts-critic/index.test.ts"],
testMatch: ["<rootDir>/packages/*/test/**/*.test.ts", "<rootDir>/packages/dts-critic/index.test.ts", "<rootDir>/packages/mergebot/src/_tests/*.test.ts"],
transform: {
"^.+\\.tsx?$": [
"ts-jest",
Expand Down
2 changes: 2 additions & 0 deletions packages/mergebot/.eslintignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
/dist/
/src/queries/schema/
142 changes: 142 additions & 0 deletions packages/mergebot/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
This is the bot that controls the workflow of Definitely Typed PRs.

## Meta

* __State:__ Production
* __Dashboard:__ [Azure](https://ms.portal.azure.com/#@72f988bf-86f1-41af-91ab-2d7cd011db47/resource/subscriptions/57bfeeed-c34a-4ffd-a06b-ccff27ac91b8/resourceGroups/dtmergebot/providers/Microsoft.Web/sites/DTMergeBot) — [Logs](https://ms.portal.azure.com/#blade/WebsitesExtension/FunctionsIFrameBlade/id/%2Fsubscriptions%2F57bfeeed-c34a-4ffd-a06b-ccff27ac91b8%2FresourceGroups%2Fdtmergebot%2Fproviders%2FMicrosoft.Web%2Fsites%2FDTMergeBot) — [GH Actions](https://github.com/microsoft/DefinitelyTyped-tools/actions) — [GH Webhook](https://github.com/DefinitelyTyped/DefinitelyTyped/settings/hooks/193097250)

TODO: Update these links for dtmergebot2

It is both a series of command line scripts which you can use to test different states, and an Azure Function App which handles incoming webhooks from the DefinitelyTyped repo.

This repo is deployed to Azure on every push to master.
To ensure we can handle timeouts on older PRs, there is a [GitHub Action](https://github.com/microsoft/DefinitelyTyped-tools/actions) that runs the bot every 6 hours against [all open PRs](./src/run.ts), and has a bunch of useful flags for running manually too.

# Setup

```sh
# Clone it
git clone https://github.com/microsoft/DefinitelyTyped-tools.git
cd DefinitelyTyped-tools

# Deps
pnpm install

# Validate it works
pnpm test
```

# How the app works

There are three main stages once the app has a PR number:

* Query the GitHub GraphQL API for PR metadata ([`pr-info`](src/pr-info.ts))
* Create a PR actions metadata object ([`compute-pr-actions`](src/compute-pr-actions.ts))
* Do work based on the resulting actions ([`execute-pr-actions`](src/execute-pr-actions.ts))

# How the bot works

There is an Azure function in `PR-Trigger` that receives webhooks; this function's job is to find the PR number then it runs the above steps.

# Running Locally

You _probably_ don't need to do this. Use test to validate any change inside the src dir against integration tests.

However, you need to have a GitHub API access key in either: `DT_BOT_AUTH_TOKEN`, `BOT_AUTH_TOKEN` or `AUTH_TOKEN`.
Ask Ryan for the bot's auth token (TypeScript team members: Look in the team OneNote).
Don't run the bot under your own auth token as this will generate a bunch of spam from duplicate comments.

```sh
# Windows
set BOT_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxx

# *nix
export BOT_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
```

Then to run locally you'll need to install the [Azure Functions cli](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=macos%2Ccsharp%2Cbash).

# Development

```sh
# Build
pnpm run build

# Run the CLI to see what would happen to an existing PR
pnpm run single-info -- [PR_NUM]
# or
pnpm run single-info-debug -- [PR_NUM]
```

### If you update any queries

Run this to update the generate types:

```sh
# Code-gen the schema
pnpm run graphql-schema
```

### If you change project columns or labels

Run this to update the cached values:

```sh
# Regenerate src/_tests/cachedQueries.json
pnpm run update-test-data
```

# Tests

```sh
# Run tests, TypeScript is transpiled at runtime
pnpm test
```

Most of the tests run against a fixtured PR, these are high level integration tests that store the PR info and then re-run the latter two phases of the app.

To create fixtures of a current PR:

```sh
# To create a fixture for PR 43161
pnpm run create-fixture -- 43161
```

Then you can work against these fixtures offline with:

```sh
# Watch mode for all tests
pnpm test -- --watch
# Just run fixtures for one PR
pnpm test -- --testNamePattern 44299
```

Run a test with the debugger:

```sh
node --inspect --inspect-brk ./node_modules/.bin/jest -i --runInBand --testNamePattern 44299
```

Then use "Attach to Process ID" to connect to that test runner

If your changes require re-creating all fixtures:

```sh
pnpm run update-all-fixtures
```

Be careful with this, because PRs may now be in a different state e.g. it's now merged and it used to be a specific
weird state.

## Running with real webhooks

You need a tool like [ngrok](https://ngrok.com) to expose a URL from the [webhooks section](https://github.com/DefinitelyTyped/DefinitelyTyped/settings/hooks/new) on DT.

Start two terminal sessions with:

- `yarn watch` (for TypeScript changes)
- `yarn start` (for the app)

Then start a third with your localhost router like ngrok:

- `ngrok http 7071`
17 changes: 17 additions & 0 deletions packages/mergebot/apollo.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
module.exports = {
client: {
includes: ["src/queries/**"],
service: {
name: "github",
url: "https://api.github.com/graphql",
headers: {
authorization: `Bearer ${
process.env["DT_BOT_AUTH_TOKEN"] ||
process.env["BOT_AUTH_TOKEN"] ||
process.env["AUTH_TOKEN"]
}`,
accept: "application/vnd.github.starfox-preview+json, application/vnd.github.bane-preview+json",
},
}
}
};
Loading
Loading