Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: filemanager deployment code and Makefile #98

Merged
merged 15 commits into from
Feb 19, 2024

Conversation

mmalenic
Copy link
Member

@mmalenic mmalenic commented Feb 8, 2024

Closes #85.

No rush on this, I can merge after #92.

Changes

  • Add Makefile consistent with other services that defines build, test, lint, etc.
  • Remove stateful constructs from the filemanager and use the shared database cluster.
  • Move filemanager from the stateful to stateless directory.
  • Make cdk_resource_invoke.ts generic over the type of Lambda function so that it can be used in other contexts and not just Rust functions.
    • I feel like this could be useful for other services, e.g. creating multiple databases within one database cluster (related to Initiate CDK Pipeline #92).

@mmalenic mmalenic force-pushed the refactor/filemanager branch from 6a7034c to 463b10b Compare February 8, 2024 02:59
@mmalenic mmalenic added filemanager an issue relating to the filemanager refactor labels Feb 8, 2024
@victorskl
Copy link
Member

I will wait for rebase, Marko. Then, will review after one go...

…ager

# Conflicts:
#	lib/workload/stateful/database/component.ts
#	lib/workload/stateless/filemanager/README.md
#	lib/workload/stateless/filemanager/database/queries/ingester/aws/update_deleted.sql
#	lib/workload/stateless/filemanager/database/queries/ingester/aws/update_reordered_for_created.sql
#	lib/workload/stateless/filemanager/database/queries/ingester/aws/update_reordered_for_deleted.sql
#	lib/workload/stateless/filemanager/filemanager/src/events/aws/collector_builder.rs
#	test/stateful/databaseConstruct.test.ts
@mmalenic mmalenic force-pushed the refactor/filemanager branch from 9e675b2 to 360b2b3 Compare February 18, 2024 22:40
Copy link
Member Author

@mmalenic mmalenic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be ready for a review now, /cc @victorskl.

lib/workload/orcabus-stateless-stack.ts Show resolved Hide resolved
lib/workload/stateful/database/component.ts Outdated Show resolved Hide resolved
@@ -0,0 +1,117 @@
{
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, these query-*.json files are required for filemanager to compile without an active postgres database running, which means that any cdk synth doesn't need to know about the filemanager's docker compose configuration.

Copy link
Member

@victorskl victorskl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good most of it. Except the db SG refactor may need revert as explain its original intent. Ping me pls if we'd like catchup a bit more focus/details, etc.

lib/workload/orcabus-stateless-stack.ts Show resolved Hide resolved
lib/workload/stateful/database/component.ts Outdated Show resolved Hide resolved
lib/workload/stateful/database/component.ts Outdated Show resolved Hide resolved
lib/workload/orcabus-stateless-stack.ts Outdated Show resolved Hide resolved
lib/workload/orcabus-stateless-stack.ts Show resolved Hide resolved
config/constants.ts Outdated Show resolved Hide resolved
config/constants.ts Outdated Show resolved Hide resolved
lib/workload/stateless/filemanager/docker-compose.yml Outdated Show resolved Hide resolved
],
},
});
});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! thanks for testing. :)

test/stateful/databaseConstruct.test.ts Outdated Show resolved Hide resolved
@mmalenic
Copy link
Member Author

Thanks for the review @victorskl. I'll revert that SG change and also address the other recommendations.

@victorskl
Copy link
Member

Sound good, no worries.

@mmalenic mmalenic requested a review from victorskl February 19, 2024 04:53
Copy link
Member

@victorskl victorskl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mmalenic mmalenic merged commit 2572b42 into main Feb 19, 2024
2 checks passed
@mmalenic mmalenic deleted the refactor/filemanager branch February 19, 2024 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filemanager an issue relating to the filemanager refactor
Projects
No open projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

filemanager: refactor code to match project structure
3 participants