Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adds nodejs-typescript-github sample #73

Merged
merged 1 commit into from
Jan 23, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions samples/nodejs-typescript-github/.funcignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
*.js.map
*.ts
.git*
.vscode
__azurite_db*__.json
__blobstorage__
__queuestorage__
local.settings.json
test
tsconfig.json
99 changes: 99 additions & 0 deletions samples/nodejs-typescript-github/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
lerna-debug.log*

# Diagnostic reports (https://nodejs.org/api/report.html)
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (https://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# TypeScript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env
.env.test

# parcel-bundler cache (https://parceljs.org/)
.cache

# next.js build output
.next

# nuxt.js build output
.nuxt

# vuepress build output
.vuepress/dist

# Serverless directories
.serverless/

# FuseBox cache
.fusebox/

# DynamoDB Local files
.dynamodb/

# TypeScript output
dist
out

# Azure Functions artifacts
bin
obj
appsettings.json
local.settings.json

# Azurite artifacts
__blobstorage__
__queuestorage__
__azurite_db*__.json
5 changes: 5 additions & 0 deletions samples/nodejs-typescript-github/.vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"recommendations": [
"ms-azuretools.vscode-azurefunctions"
]
}
13 changes: 13 additions & 0 deletions samples/nodejs-typescript-github/.vscode/launch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"version": "0.2.0",
"configurations": [
{
"name": "Attach to Node Functions",
"type": "node",
"request": "attach",
"restart": true,
"port": 9229,
"preLaunchTask": "func: host start"
}
]
}
9 changes: 9 additions & 0 deletions samples/nodejs-typescript-github/.vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"azureFunctions.deploySubpath": ".",
"azureFunctions.postDeployTask": "npm install (functions)",
"azureFunctions.projectLanguage": "TypeScript",
"azureFunctions.projectRuntime": "~4",
"debug.internalConsoleOptions": "neverOpen",
"azureFunctions.projectLanguageModel": 4,
"azureFunctions.preDeployTask": "npm prune (functions)"
}
50 changes: 50 additions & 0 deletions samples/nodejs-typescript-github/.vscode/tasks.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{
"version": "2.0.0",
"tasks": [
{
"type": "func",
"label": "func: host start",
"command": "host start",
"problemMatcher": "$func-node-watch",
"isBackground": true,
"dependsOn": "npm watch (functions)"
},
{
"type": "shell",
"label": "npm build (functions)",
"command": "npm run build",
"dependsOn": "npm clean (functions)",
"problemMatcher": "$tsc"
},
{
"type": "shell",
"label": "npm watch (functions)",
"command": "npm run watch",
"dependsOn": "npm clean (functions)",
"problemMatcher": "$tsc-watch",
"group": {
"kind": "build",
"isDefault": true
},
"isBackground": true
},
{
"type": "shell",
"label": "npm install (functions)",
"command": "npm install"
},
{
"type": "shell",
"label": "npm prune (functions)",
"command": "npm prune --production",
"dependsOn": "npm build (functions)",
"problemMatcher": []
},
{
"type": "shell",
"label": "npm clean (functions)",
"command": "npm run clean",
"dependsOn": "npm install (functions)"
}
]
}
194 changes: 194 additions & 0 deletions samples/nodejs-typescript-github/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
# Ingest GitHub issues, pull requests and commits using TypeScript and Azure Functions

## Summary

This sample demonstrates how you can ingest GitHub issues, pull requests, and commits into Microsoft Graph using a Graph connector. The sample is written in TypeScript and runs on Azure Functions. Data is ingested by using a GitHub webhook to send events to the Azure Function HTTP trigger.

![Solution architecture](./assets/solution-overview.png)

Once the Graph connector is enabled in Microsoft Search, you can use Microsoft Search and Copilot to search for GitHub issues, pull requests, and commits.

![Use case](./assets/use-case.png)

## Contributors

* [Lee Ford](https://www.github.com/leeford)

## Version history

Version|Date|Comments
-------|----|--------
1.0|January 14, 2025|Initial release

## Prerequisites

* Microsoft tenant with a Microsoft 365 subscription (this will allow up to 50 million items to be ingested through Graph connectors). You can also use a [Microsoft 365 developer subscription](https://developer.microsoft.com/microsoft-365/dev-program)
* [Node@20](https://nodejs.org) or later

```bash
# determine node version
node --version
# v20.18.0
```

* [Azure Functions Core Tools@4](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local)or later

```bash
# determine core tools version
func --version
# 4.0.6610
```

* A tunnel service like [Dev Tunnels](https://learn.microsoft.com/azure/developer/dev-tunnels/get-started#install) or [ngrok](https://ngrok.com/) to expose your local Azure Function to the internet

## Minimal path to awesome

### Initial setup

* Create a Dev Tunnel on port 7071

```bash
# create a tunnel on port 7071
dev-tunnels start --port 7071

# Hosting port: 7071
# Connect via browser: https://12345678-7071.uks1.devtunnels.ms
# Inspect network activity: https://12345678-7071-inspect.uks1.devtunnels.ms
```

> Take a copy of the URL that the tunnel is running on. You will need this to create a GitHub webhook

* [Create a new GitHub webhook](https://docs.github.com/en/webhooks/using-webhooks/creating-webhooks), ensuring that:
* The **Payload URL** is the webhook is set to send events for issues, pull requests, and commits to the Dev Tunnel's URL with the path `/api/ingest` (e.g. `https://12345678-7071.uks1.devtunnels.ms/api/ingest`)
* The **Content type** is set to `application/json`
* The following **Events** are selected:
* Issues
* Pull requests
* Pushes

![GitHub webhook configuration](./assets/github-webhook.png)

### Create an Entra ID App registration

* Create an Entra ID App registration to grant the code access to the Microsoft Graph API. For ease of use, you can use [Azure Cloud Shell](https://shell.azure.com) or [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/) to create the app registration.

```bash
# create the app registration
az login
az ad app create --display-name "GitHub Graph connector"
```

> Take note of the **appId** value from the output as this will be required shortly.

* Grant the app registration the required permissions to create an external connection (Graph connector) and items.

```bash
# give app the permissions to create an external connection and items
az ad app permission add --id <APP ID> --api 00000003-0000-0000-c000-000000000000 --api-permissions "f431331c-49a6-499f-be1c-62af19c34a9d=Role" "8116ae0f-55c2-452d-9944-d18420f5b2c8=Role"

# admin consent the permissions
az ad app permission admin-consent --id <APP ID>
```

* Create a secret for the app registration

```bash
# create a secret for the app registration
az ad app credential reset --id <APP ID>
```

> Take note of the **password** and **tennant** values from the output as this will be required shortly.

### Running the sample

* Clone this repository (or [download this solution as a .ZIP file](https://pnp.github.io/download-partial/?url=https://github.com/pnp/graph-connectors-samples/tree/main/samples/nodejs-typescript-github) then unzip it)
* In the command line, navigate to the `nodejs-typescript-github` folder and install the dependencies

```bash
# navigate to the folder
cd nodejs-typescript-github

# install the dependencies
npm install
```

* Create and populate a local.settings.json file in the root folder with the following (with your own values):

```json
{
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "node",
"AzureWebJobsStorage": "",
"AzureWebJobsFeatureFlags": "EnableWorkerIndexing",
"MicrosoftAppTenantId": "<tenantId>",
"MicrosoftAppId": "<appId>",
"MicrosoftAppPassword": "<appPassword>",
"CONNECTION_ID": "github"
}
}
```

> * Replace `<tenantId>` with the `tenant` value from the app registration created earlier
> * Replace `<appId>` with the `appId` value from the app registration created earlier
> * Replace `<appPassword>` with the `password` value from the app registration created earlier

* Start the Azure Function

```bash
# start the Azure Function
func start
```

### Creating the Graph connector

* From the previous step, you may notice on there are three functions running, you need to browse to the **createConnection** endpoint to create a Graph connector on `http://localhost:7071/api/createConnection`

> This can take upwards of 15 minutes for the connection to be created

### Enabling the Graph connector in search results

Once created, you Enable the Graph connector in Search & intelligence data sources

* Go to the [Search & intelligence data sources](https://admin.microsoft.com/?source=applauncher#/MicrosoftSearch/connectors) page
* Find the **GitHub** connection in the table, and in the **Required actions** column, select **Include Connector Results** and confirm the prompt
* It should now have no required actions and be **Ready**

![Search & Intelligence Data Sources](./assets/search-sources.png)

### Testing the Graph connector

* Make some changes to a GitHub repository that you have the webhook configured for (e.g. create an issue, pull request, or commit)
* Wait a few minutes for the changes to be ingested
* Go to [Microsoft 365](https://www.microsoft365.com/) and search for the changes you made e.g. open issues

![Search results](./assets/search-results.png)

## Features

Overall, this sample demonstrates the following:

* Create an external connection, schema and result layout all using Graph API
* Ingest data from GitHub using a webhook

Looking a bit deeper, this sample shows how you can use the Webhook feature to handle events from GitHub and ingest them into Microsoft Graph instead of crawling a data source periodically. This differs from crawling GitHub in the following ways:

* **Real-time updates**: The Graph connector is updated in real-time as events occur in GitHub
* **Customizable**: You can choose which events to ingest and how to handle them
* **Scoped**: You can scope the connector to only ingest data from specific repositories rather than all repositories

## Help

We do not support samples, but this community is always willing to help, and we want to improve these samples. We use GitHub to track issues, which makes it easy for community members to volunteer their time and help resolve issues.

You can try looking at [issues related to this sample](https://github.com/pnp/graph-connectors-samples/issues?q=label%3A%22sample%3A%20nodejs-typescript-github%22) to see if anybody else is having the same issues.

If you encounter any issues using this sample, [create a new issue](https://github.com/pnp/graph-connectors-samples/issues/new).

Finally, if you have an idea for improvement, [make a suggestion](https://github.com/pnp/graph-connectors-samples/issues/new).

## Disclaimer

**THIS CODE IS PROVIDED *AS IS* WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING ANY IMPLIED WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR NON-INFRINGEMENT.**

![](https://m365-visitor-stats.azurewebsites.net/SamplesGallery/nodejs-typescript-github)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 5 additions & 0 deletions samples/nodejs-typescript-github/extensions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"recommendations": [
"ms-azuretools.vscode-azurefunctions"
]
}
Loading
Loading