Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEAT: Memory leak, IPFS fixes, and more Arch improvements #11

Merged
merged 31 commits into from
Oct 2, 2024
Merged

FEAT: Memory leak, IPFS fixes, and more Arch improvements #11

merged 31 commits into from
Oct 2, 2024

Conversation

xadahiya
Copy link
Member

@xadahiya xadahiya commented Sep 27, 2024 •
edited
Loading

  • Memory Leak fixes
  • Better asyncio task tracking, management, and cleanup
  • Grpc stream handling improvements
  • IPFS throughput improvements
  • Logging improvements
  • No more processor core hub
  • Pm2 related improvements
  • Cleaner aggregation config
  • No more invalid json files

@xadahiya
chore: tracking tasks and cleaning them up, minor bug fixes
1ab8699
@xadahiya
Merge remote-tracking branch 'refs/remotes/origin/dockerify' into doc…
86c0ab8 
…kerify
@xadahiya
chore: timing out stream submissions after 10 seconds
eb768b4
@xadahiya
chore: reraising IPFS exceptions
6e5a1c5
@xadahiya
chore: using mounted docker volumes and master-latest image tag
a57d7ad
@xadahiya
chore: treating timeouts for stream submission as success
8a4402a
@xadahiya
chore: no timeouts for simulation mode, increased timeout to 60 secon…
bab7255 
…ds for normal mode
@xadahiya
chore: removed simulation response timeout
d38b8a6
@xadahiya
chore: assuming success in case of timeout, final cleanup
138c205
@xadahiya
chore: stream handling final changes
83881ab
@xadahiya
chore: added stream message timeout again
75c3a18
@xadahiya
chore: Improve IPFS config, running in fast mode now with increased s…
96644ff 
…ystem limits. Other minor cleanup
@xadahiya
chore: better management of asyncio tasks and cleanup
30497bf
@xadahiya
chore: remove extra sysctls
1ba296d
@xadahiya
chore: cleanup bootstrap.sh
0d6cbcf
@xadahiya
fix: IPFS init
ea008ae
@xadahiya
chore: update bootstrap script, minor init fix for ipfs
5bc4c31
@xadahiya xadahiya changed the title Perf Improvements FEAT: Memory leak and IPFS fixes Sep 30, 2024
@xadahiya xadahiya marked this pull request as ready for review September 30, 2024 10:13
@xadahiya xadahiya marked this pull request as draft September 30, 2024 13:34
@xadahiya xadahiya changed the title FEAT: Memory leak and IPFS fixes FEAT: Memory leak, IPFS fixes, and more Arch improvements Sep 30, 2024
@xadahiya xadahiya marked this pull request as ready for review September 30, 2024 14:09
Seth-Schmidt
Seth-Schmidt requested changes Oct 1, 2024
View reviewed changes
Copy link
Collaborator

@Seth-Schmidt Seth-Schmidt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost everything looks good. One minor change suggested.

snapshotter/processor_distributor.py Show resolved Hide resolved
anomit
anomit approved these changes Oct 2, 2024
View reviewed changes
Copy link
Member

@anomit anomit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xadahiya xadahiya merged commit 28079f3 into main Oct 2, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@anomit anomit anomit approved these changes

@Seth-Schmidt Seth-Schmidt Awaiting requested review from Seth-Schmidt

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xadahiya @anomit @Seth-Schmidt