Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: turbo Initial Blockchain Download #37

Closed
wants to merge 1 commit into from
Closed

WIP: turbo Initial Blockchain Download #37

wants to merge 1 commit into from

Conversation

Sjors
Copy link
Owner

@Sjors Sjors commented May 5, 2018 •
edited
Loading

Builds on top of #36.

Uses a m5.24xlarge c5.2xlarge EC2 instance with a 230 GB RAM drive to speed up
Initial Blockchain Download. The result is then pruned and copied
to the normal instance.

Todo:

  • don't bother with the RAM disk, set dbcache to 10 GB
  • 8 CPUs is likely enough
  • compare t2.2xlarge, m5.2xlarge and compute optimized c5.2xlarge
  • poll bitcoin-cli until IBD is done
  • append prune=2000 to bitcoin.conf, restart
  • wait for prune to complete, shutdown bitcoind
  • copy chainstate and blocks/ to normal instance
  • self destruct, or instructions for user to do so
  • remove git checkout 2018/05/fast-ibd before merging

Optional / later:

  • don't use Docker? (no need to since this process isn't portable)

@Sjors Sjors mentioned this pull request May 5, 2018
Closed
18 tasks
@Sjors Sjors force-pushed the 2018/05/fast-ibd branch 3 times, most recently from cff849e to 05d3a51 Compare May 7, 2018 09:58
@Sjors Sjors mentioned this pull request May 14, 2018
Open
@Sjors Sjors force-pushed the 2018/05/fast-ibd branch 2 times, most recently from 8142424 to 0e68170 Compare May 23, 2018 12:11
@Sjors
EC2: Turbo mode IBD
561f8e2 
Use a c5.2xlarge EC2 instance to speed up Initial Blockchain Download.
The result is then pruned and copied to the normal instance.

dbcache=40000 should be more than enough (currently needs about 7 GB)
maxuploadtarget=10000 although not much upload happens during IBD and
                      the instance only lives for a few hour, this
                      caps upload fees to ~$1 per day just in case.
@Sjors Sjors force-pushed the 2018/05/fast-ibd branch from 0e68170 to 561f8e2 Compare May 23, 2018 12:35
@Sjors
Copy link
Owner Author

Sjors commented May 23, 2018

Using two instances creates more problems than it solves at the moment.

  1. Afaik there's no way to self destruct the second instance or to delete it remotely from the first one
  2. It's not easy to copy files between the two instances (would probably require an ad hoc web server)

@Sjors Sjors closed this May 23, 2018
@Sjors Sjors mentioned this pull request May 23, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Sjors