Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DEV-2073] Add sync users script using tsnode #1289

Merged
merged 59 commits into from
Jan 10, 2025
Merged
Show file tree
Hide file tree
Changes from 55 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
4984a2a
sync user wip
petros-double-test1 Dec 5, 2024
33802ef
sync user
petros-double-test1 Dec 5, 2024
c432ad4
changeset
petros-double-test1 Dec 5, 2024
fd01f78
Merge branch 'main' into dev-2017
MarcoPonchia Dec 6, 2024
5865dc1
Update .changeset/cuddly-cats-retire.md
tommaso1 Dec 6, 2024
b1b8180
Update packages/active-campaign-client/src/index.ts
tommaso1 Dec 6, 2024
6ec2fc2
Update packages/active-campaign-client/src/index.ts
tommaso1 Dec 6, 2024
b102553
Update packages/active-campaign-client/src/helpers/resyncUser.ts
tommaso1 Dec 6, 2024
3a7569d
pr comments
petros-double-test1 Dec 6, 2024
941f3a7
bulk add contact
petros-double-test1 Dec 10, 2024
cdbf4cc
bulk add contact
petros-double-test1 Dec 10, 2024
210f144
resync user
petros-double-test1 Dec 10, 2024
53f3ea1
changeset
petros-double-test1 Dec 10, 2024
49f2320
Update packages/active-campaign-client/src/clients/activeCampaignClie…
tommaso1 Dec 16, 2024
29e7407
Update packages/active-campaign-client/src/handlers/resyncUserHandler.ts
tommaso1 Dec 16, 2024
2e93e4c
pr changes
petros-double-test1 Dec 16, 2024
997e3ad
Merge branch 'dev-1614' of github.com:pagopa/developer-portal into de…
petros-double-test1 Dec 16, 2024
e1949e6
Update packages/active-campaign-client/src/helpers/fetchSubscribedWeb…
tommaso1 Dec 17, 2024
a082a7b
Update packages/active-campaign-client/.env.example
tommaso1 Dec 17, 2024
696fab5
Update packages/active-campaign-client/src/handlers/resyncUserHandler.ts
tommaso1 Dec 17, 2024
2874681
pr changes
petros-double-test1 Dec 18, 2024
3808be8
Merge remote-tracking branch 'origin/main' into dev-1614
petros-double-test1 Dec 18, 2024
f3dd9a4
Merge branch 'main' into dev-1614
MarcoPonchia Dec 19, 2024
d86cd54
add makeContactPayload
MarcoPonchia Dec 19, 2024
f3846c8
Add types
MarcoPonchia Dec 19, 2024
466b1a9
refactor AC client
MarcoPonchia Dec 19, 2024
7ea308c
add getListByName return type
MarcoPonchia Dec 19, 2024
ab13428
refactor getListIdByName and add error manage to sqsQueueHandler
MarcoPonchia Dec 19, 2024
d917094
Add addOrUpdateContact helper
MarcoPonchia Dec 19, 2024
06b99b1
add getNewWebinarsAndUnsubsriptionLists
MarcoPonchia Dec 19, 2024
e58012d
add addArrayOfListToContact and removeArrayOfListFromContact helpers
MarcoPonchia Dec 19, 2024
7c73575
Update resync user handlers
MarcoPonchia Dec 19, 2024
3d705b2
Fix manageError
MarcoPonchia Dec 19, 2024
7837078
Merge remote-tracking branch 'origin/dev-1614' into refactor-resync-e…
MarcoPonchia Dec 19, 2024
3d70311
Fix after merge
MarcoPonchia Dec 19, 2024
a3b8052
add changeset
MarcoPonchia Dec 19, 2024
599506e
Merge remote-tracking branch 'origin/main' into refactor-resync-error…
MarcoPonchia Dec 19, 2024
2416aba
Fix list ac client call
MarcoPonchia Dec 19, 2024
8af9e78
Merge branch 'main' into refactor-resync-error-handler
MarcoPonchia Dec 19, 2024
2477350
Refactor env var for test
MarcoPonchia Dec 19, 2024
a3d4bed
fix make payload
MarcoPonchia Dec 19, 2024
294c2b9
fix resync function
MarcoPonchia Dec 19, 2024
55eabb9
add sync user script
MarcoPonchia Dec 19, 2024
cd19ea4
add changeset
MarcoPonchia Dec 19, 2024
7d8eeef
Merge remote-tracking branch 'origin/main' into add-sync-users-script
MarcoPonchia Dec 20, 2024
62a6f6b
add env
MarcoPonchia Dec 20, 2024
cf461b9
import users
petros-double-test1 Dec 20, 2024
c4834f5
Fix env
MarcoPonchia Dec 20, 2024
84dde4a
update script and configuration
MarcoPonchia Dec 20, 2024
f8b597f
Add changeset
MarcoPonchia Dec 20, 2024
2f1c8a5
update after review
MarcoPonchia Dec 20, 2024
f3bc094
Merge branch 'main' into add-sync-users-script-tsnode
marcobottaro Dec 20, 2024
df7ff69
Merge branch 'main' into add-sync-users-script-tsnode
marcobottaro Jan 7, 2025
1849d34
Merge branch 'main' into add-sync-users-script-tsnode
marcobottaro Jan 9, 2025
7a04af4
Manage error during webinar fetching
MarcoPonchia Jan 9, 2025
d8966ef
remove try catch
MarcoPonchia Jan 9, 2025
7e9c988
Merge remote-tracking branch 'origin/main' into add-sync-users-script…
MarcoPonchia Jan 9, 2025
9d201b9
Manage AC POST contactList error
MarcoPonchia Jan 9, 2025
2159187
Remove unused type
MarcoPonchia Jan 10, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/sour-zebras-bathe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"active-campaign-client": patch
---

Add sync users script
3 changes: 2 additions & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 8 additions & 1 deletion packages/active-campaign-client/.env.example
Original file line number Diff line number Diff line change
@@ -1,8 +1,15 @@
SENDER_URL=localhost:3000
SENDER_URL=http://localhost:3000/
AWS_REGION="region"
COGNITO_USER_POOL_ID="your_region"
DYNAMO_WEBINARS_TABLE_NAME="name"
AC_BASE_URL_PARAM='/ac/base_url'
AC_API_KEY_PARAM='/ac/api_key'
AC_RESYNC_TIMEOUT_IN_MS=1000

# Only needed run import script locally
AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY
USERS_TO_IMPORT_LIMIT=5 #optional only for local testing

TEST_AC_BASE_URL=your_account_url
TEST_AC_API_KEY=your_api_key
Expand Down
5 changes: 3 additions & 2 deletions packages/active-campaign-client/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@
"scripts": {
"lint": "eslint src",
"test": "jest",
"build": "tsc"
"build": "tsc",
"sync-all-users": "ts-node src/scripts/sync_all_users.ts"
},
"devDependencies": {
"@eslint/eslintrc": "^2.1.4",
"@eslint/js": "^8.57.1",
"dotenv": "16.4.5",
"dotenv": "^16.4.5",
"eslint": "^8.57.1",
"eslint-config-custom": "^0.1.0"
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -187,10 +187,12 @@ export class ActiveCampaignClient {
last_name: payload.contact.lastName,
phone: payload.contact.phone,
customer_acct_name: payload.contact.lastName,
fields: payload.contact.fieldValues.map((field) => ({
id: Number(field.field),
value: field.value,
})),
fields: payload.contact.fieldValues
.filter((field) => field.value)
.map((field) => ({
id: Number(field.field),
value: field.value,
})),
subscribe: payload.listIds.map((listId) => ({ listid: listId })),
})),
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,11 @@ export async function getNewWebinarsAndUnsubsriptionLists(
cognitoUsername
);

if (userWebinarsSubscriptions.statusCode !== 200) {
// eslint-disable-next-line functional/no-throw-statements
throw new Error('Error fetching user webinars subscriptions');
}

const webinarSlugs: readonly string[] = JSON.parse(
userWebinarsSubscriptions.body
)
Expand Down
150 changes: 150 additions & 0 deletions packages/active-campaign-client/src/scripts/sync_all_users.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
// To run this script you need to have proper environment variables set up.
// Must be logged in to AWS CLI and have the proper permissions to access the Cognito User Pool and DynamoDB tables.
// Then run 'npm run sync-all-users -w active-campaign-client'

/* eslint-disable */
import { ActiveCampaignClient } from "../clients/activeCampaignClient";
import { CognitoIdentityServiceProvider } from 'aws-sdk';
import { DynamoDBClient, QueryCommand } from '@aws-sdk/client-dynamodb';
import dotenv from 'dotenv';

dotenv.config();

const activeCampaignClient = new ActiveCampaignClient(
process.env.AC_BASE_URL!,
process.env.AC_API_KEY!
);

const userPoolId = process.env.COGNITO_USER_POOL_ID!;

// Initialize Cognito client
const cognito = new CognitoIdentityServiceProvider({
region: process.env.AWS_REGION!,
credentials: {
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!
}
});

export async function getSubscribedWebinars(
username: string
): Promise<any[]> {
try {
const dynamoClient = new DynamoDBClient({ region: process.env.AWS_REGION });
const command = new QueryCommand({
TableName: process.env.DYNAMO_WEBINARS_TABLE_NAME,
KeyConditionExpression: 'username = :username',
ExpressionAttributeValues: {
':username': { S: username },
},
});

const response = await dynamoClient.send(command);
return response.Items as any[];
} catch (error) {
console.error('Error querying items by username:', error);
return [];
}
}

// Load all users from Cognito
async function listAllUsers() {
try {
const allUsers = [];
let paginationToken = undefined;

do {
const params: any = {
UserPoolId: userPoolId,
PaginationToken: paginationToken
};

const response: any = await cognito.listUsers(params).promise();
allUsers.push(...response.Users);
paginationToken = response.PaginationToken;
} while (paginationToken);

return allUsers;
} catch (error) {
console.error('Error fetching users:', error);
return [];
}
}

async function main() {
const users = await listAllUsers();

console.log(process.env.DYNAMO_WEBINARS_TABLE_NAME);

const usersAndWebinars = [];

for (const user of users) {
const subscribedWebinars = await getSubscribedWebinars(user.Username);
usersAndWebinars.push({
username: user,
subscribedWebinars: subscribedWebinars.map((webinar: any) => webinar?.webinarId?.S).filter(Boolean)
});
}

const subscribedWebinarNames = usersAndWebinars.flatMap((userAndWebinars) => userAndWebinars.subscribedWebinars);

const allLists: any = await activeCampaignClient.getLists();

const webinarIdByName = allLists.lists.reduce((acc: any, list: any) => {
acc[list.name] = Number(list.id);
return acc;
}, {});

console.log(webinarIdByName);

const emails = new Set();
// remove duplicates on emails
const uniqueUsersAndWebinars = usersAndWebinars.filter((userAndWebinars) => {
const email = userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'email')?.Value;
if (emails.has(email)) {
return false;
}
emails.add(email);
return true;
});

const usersToImportLimit = !!process.env.USERS_TO_IMPORT_LIMIT && parseInt(process.env.USERS_TO_IMPORT_LIMIT);
const limitedUniqueUsersAndWebinars = usersToImportLimit ? uniqueUsersAndWebinars
.filter((userAndWebinars) => userAndWebinars.subscribedWebinars.length > 0)
.slice(0, usersToImportLimit)
: uniqueUsersAndWebinars;

const acPayload = limitedUniqueUsersAndWebinars.map((userAndWebinars, index) => ({
contact: {
email: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'email')?.Value,
firstName: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'given_name')?.Value,
lastName: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'family_name')?.Value,
phone: `cognito:${userAndWebinars.username.Username}`,
fieldValues: [
{
field: '2',
value: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'custom:company_type')?.Value,
},
{
field: '1',
value: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'custom:job_role')?.Value,
},
{
field: '3',
value: userAndWebinars.username.Attributes.find((attr: any) => attr.Name === 'custom:mailinglist_accepted')?.Value === 'true' ? 'TRUE' : 'FALSE',
},
],
},
listIds: userAndWebinars.subscribedWebinars.map((webinar: any) => webinarIdByName[webinar]).filter(Boolean),
}));

const response = await activeCampaignClient.bulkAddContactToList(acPayload);

console.log(response);
}

// Execute the main function
main().catch(error => {
console.error('Error executing script:', error);
process.exit(1);
});
Loading