Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill synonym slugs (DO NOT MERGE) #2785

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions app/backfill-synonym-slugs.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import { DynamoDBClient, paginateScan } from '@aws-sdk/client-dynamodb'
import { GetParameterCommand, SSMClient } from '@aws-sdk/client-ssm'
import { BatchWriteCommand } from '@aws-sdk/lib-dynamodb'
import { unmarshall } from '@aws-sdk/util-dynamodb'
import { slug } from 'github-slugger'
import chunk from 'lodash/chunk'

async function getTableNameFromSSM(dynamoTableName) {
const ssmClient = new SSMClient({ region: 'us-east-1' })

Check warning on line 9 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L8-L9

Added lines #L8 - L9 were not covered by tests

try {
const command = new GetParameterCommand({ Name: dynamoTableName })
const response = await ssmClient.send(command)

Check warning on line 13 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L11-L13

Added lines #L11 - L13 were not covered by tests

if (!response.Parameter?.Value) {
throw new Error('dynamoTableName not found in SSM')

Check warning on line 16 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L16

Added line #L16 was not covered by tests
}

return response.Parameter.Value

Check warning on line 19 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L19

Added line #L19 was not covered by tests
} catch (error) {
console.error('Error fetching table name from SSM:', error)
throw error

Check warning on line 22 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L21-L22

Added lines #L21 - L22 were not covered by tests
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, you could safely remove this entire try/except block. You'll get a good stack trace either way.

}

export async function backfillSlugsOnSynonyms() {
const startTime = new Date()
console.log('Starting SYNONYM SLUG backfill... ', startTime)
Comment on lines +16 to +17
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const dynamoTableName = '/RemixGcnProduction/tables/synonyms'
const TableName = await getTableNameFromSSM(dynamoTableName)
const client = new DynamoDBClient({ region: 'us-east-1' })
const pages = paginateScan({ client }, { TableName })

Check warning on line 32 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L26-L32

Added lines #L26 - L32 were not covered by tests

for await (const page of pages) {

Check warning on line 34 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L34

Added line #L34 was not covered by tests
const chunked = chunk(page.Items || [], 25)
const synonymsToUpdate = []
for (const chunk of chunked) {
for (const record of chunk) {
const synonym = unmarshall(record)

Check warning on line 39 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L36-L39

Added lines #L36 - L39 were not covered by tests
if (!synonym.slug) {
synonymsToUpdate.push(synonym)

Check warning on line 41 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L41

Added line #L41 was not covered by tests
}
}
if (synonymsToUpdate.length > 0) {
const command = new BatchWriteCommand({

Check warning on line 45 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L45

Added line #L45 was not covered by tests
RequestItems: {
[TableName]: synonymsToUpdate.map((synonym) => ({

Check warning on line 47 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L47

Added line #L47 was not covered by tests
PutRequest: {
Item: {
synonymId: synonym.synonymId,
eventId: synonym.eventId,
slug: slug(synonym.eventId),
},
},
})),
},
})
await client.send(command)
console.log(`updated ${synonymsToUpdate.length} records`)

Check warning on line 59 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L58-L59

Added lines #L58 - L59 were not covered by tests
}
}
}
const endTime = new Date()
console.log('... End SYNONYM SLUG backfill... ', endTime)

Check warning on line 64 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L63-L64

Added lines #L63 - L64 were not covered by tests
}

backfillSlugsOnSynonyms()

Check warning on line 67 in app/backfill-synonym-slugs.js

View check run for this annotation

Codecov / codecov/patch

app/backfill-synonym-slugs.js#L67

Added line #L67 was not covered by tests