Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(event-source): ignore cache directories #621

Merged
merged 1 commit into from
Oct 29, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions config/stacks/shared.ts
Original file line number Diff line number Diff line change
@@ -94,9 +94,12 @@ const getEventSourceConstructProps = (stage: AppStage): EventSourceProps => {
rules: [
{
bucket: oncoanalyserBucket[stage],
eventTypes: ['Object Created', 'Object Deleted'],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are those the only event types the FileManager is currently monitoring?
If yes, that's OK, just be aware that this is a while-listing and you'll miss any other events (which is OK for the portal use case, but you'll have to judge if it's also OK for the FileManager).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, those are the only ones considered right now. It derives all the other information from HeadObject and GetObjectTagging. In the future, I can add support for "Object Storage Class Changed" and "Object Tags Added" to avoid using an API call.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sound good. Go for it, Marko. Pls merge when you ready.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding to this issue: #322 (comment)

},
{
bucket: icav2PipelineCacheBucket[stage],
eventTypes: ['Object Created', 'Object Deleted'],
key: [{ 'anything-but': { wildcard: 'byob-icav2/*/cache/*' } }],
},
],
};
Original file line number Diff line number Diff line change
@@ -13,16 +13,18 @@ export type EventSourceRule = {
* Bucket to receive events from. If not specified, captures events from all buckets.
*/
bucket?: string;

/**
* The types of events to capture for the bucket. If not specified, captures all events.
* This should be from the list S3 EventBridge events:
* https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
*/
eventTypes?: string[];

/**
* A prefix of the objects that are matched when receiving events from the buckets.
* Rules matching specified keys in buckets.
*/
prefix?: string;
key?: { [key: string]: any }[];
};

/**
@@ -33,13 +35,15 @@ export type EventSourceProps = {
* The name of the queue to construct.
*/
queueName: string;

/**
* The maximum number of times a message can be unsuccessfully received before
* pushing it to the DLQ.
*/
maxReceiveCount: number;

/**
* A set of EventBridge rules to define..
* A set of EventBridge rules to define.
*/
rules: EventSourceRule[];
};
@@ -80,13 +84,9 @@ export class EventSourceConstruct extends Construct {
name: [prop.bucket],
},
}),
...(prop.prefix && {
...(prop.key && {
object: {
key: [
{
prefix: prop.prefix,
},
],
key: prop.key,
},
}),
},
7 changes: 5 additions & 2 deletions test/stateful/shared/eventSourceConstruct.test.ts
Original file line number Diff line number Diff line change
@@ -75,14 +75,14 @@ test('Test EventSourceConstruct created props with event types', () => {
});
});

test('Test EventSourceConstruct created props with prefix', () => {
test('Test EventSourceConstruct created props with key rule', () => {
new EventSourceConstruct(stack, 'TestEventSourceConstruct', {
queueName: 'queue',
maxReceiveCount: 100,
rules: [
{
bucket: 'bucket',
prefix: 'prefix',
key: [{ 'anything-but': { wildcard: 'wildcard/*' } }, { prefix: 'prefix' }],
},
],
});
@@ -94,6 +94,9 @@ test('Test EventSourceConstruct created props with prefix', () => {
detail: {
object: {
key: [
{
'anything-but': { wildcard: 'wildcard/*' },
},
{
prefix: 'prefix',
},