Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(miner): externalise SectorOnChainInfo from AMT #338

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rvagg
Copy link
Member

@rvagg rvagg commented Jan 22, 2025

Ref: filecoin-project/builtin-actors#1604

Draft for now, for benchmarking purposes

@codecov-commenter
Copy link

codecov-commenter commented Jan 22, 2025

Codecov Report

Attention: Patch coverage is 0% with 123 lines in your changes missing coverage. Please review.

Project coverage is 3.39%. Comparing base (5f30872) to head (7bc0344).

Files with missing lines Patch % Lines
builtin/v16/migration/miner.go 0.00% 74 Missing ⚠️
builtin/v16/miner/miner_types.go 0.00% 26 Missing ⚠️
builtin/v16/migration/top.go 0.00% 18 Missing ⚠️
builtin/v15/miner/miner_types.go 0.00% 5 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #338      +/-   ##
==========================================
- Coverage    3.39%    3.39%   -0.01%     
==========================================
  Files         662      663       +1     
  Lines      177834   177949     +115     
==========================================
  Hits         6041     6041              
- Misses     169944   170059     +115     
  Partials     1849     1849              
Files with missing lines Coverage Δ
builtin/v16/miner/miner_state.go 0.00% <ø> (ø)
builtin/v15/miner/miner_types.go 0.00% <0.00%> (ø)
builtin/v16/migration/top.go 0.00% <0.00%> (ø)
builtin/v16/miner/miner_types.go 0.00% <0.00%> (ø)
builtin/v16/migration/miner.go 0.00% <0.00%> (ø)

@rvagg rvagg force-pushed the rvagg/sector-gas/external-soi branch from 3d4abd9 to 7bc0344 Compare January 22, 2025 05:41
@rvagg
Copy link
Member Author

rvagg commented Jan 22, 2025

Migration takes too long and I run out of memory for this

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1731013 lotus     20   0  414.0g 169.9g   5.4g S 380.7  90.4  91:32.50 lotus-shed

just before Killed, I don't have enough RAM to support this.

It got to:

2025-01-22T16:46:48.801+1100    INFO    fil-consensus   filcns/upgrades.go:3012 Performing migration: 1184187 of 1185201 jobs processed (749/s) [26m21s elapsed]

Which is 245000 miners processed (I had another log line I didn't push in this commit). So about 1/3rd of the way through in 26 minutes, on a fairly fast machine.

@rvagg
Copy link
Member Author

rvagg commented Jan 22, 2025

I think I'd like to explore an approach of updates to the AMT being externalised but existing sectors prior to the network upgrade left as they are. So we'd just have to do some byte sniffing - is it a CID, or is it a SectorOnChainInfo? I believe we've avoided such half-way schemas before, opting to do a full migration into a new schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants