This repository has been archived by the owner on Jun 6, 2023. It is now read-only.
Increase cbor-gen max bytearray length to support maximum partition sectors bitfield #676
Comments
anorth
added
the
P1
11 tasks
Stebalien
added a commit
that referenced
this issue
Jul 16, 2020
…ues (#648) For motivation, see #599. This giant PR restructures the miner actor's state, representing partitions as first-class objects. Sectors, faults, recoveries, expiration and terminations are all tracked per-partition. A few totals of power and pledge are maintained in the partition so that power and penalty accounting for faults etc need not load the `SectorOnChainInfo`s for all the sectors (which can be a lot). The heavy per-sector information is only loaded in miner-initiated messages, never from cron. Significant: - Partitions are now numbered per-deadline, rather than miner-wide - When a message references a sector, it usually needs to include information about which deadline and partition the sector is allocated to. The miner must do this search off-chain to avoid on-chain search or indices. - New sectors are allocated to deadlines and partitions immediately, rather than waiting for cron. - A cron callback now happens at the end of every deadline, rather than every proving period (#552) - Penalties are now payed for the proving period in arrears, rather than in advance (in general, penalties are reduced) - The fault and expiration queues are combined into one (per partition) - Terminated sectors are not removed from partitions until an explicit Defrag method; they are masked over for Window PoSt like faults Follow-up: - Tests - Implement CompactPartitions method #673 - For early termination, pay SP(t, power) immediately, pay the rest on termination processing (#674) - ncrease the CBOR-gen maximum bitfield length to accomodate 2350 sector numbers (#676) - Fix termination of entire miner (#675) Closes #391 Closes #357 Closes #391 Closes #411 Closes #418 Closes #483 Closes #519 Closes #535 Closes #552 Closes #593 Co-authored-by: Steven Allen <[email protected]> Co-authored-by: acruikshank <[email protected]> Co-authored-by: Alex Cruikshank <[email protected]>
|
Looking at the code |
|
As of filecoin-project/go-bitfield#41, we now use cbor-gen's byte array limit (2MiB). But this is too much (it can force us to allocate 128MiB of memory to expand the runs). That's obviously not OK. We need at least 20KiB to encode a partition's sectors. Let's bump to 32 to be safe. 32KiB can encode at most 256Ki runs = 4MiB of memory expanded. We can reduce this to 2MiB with filecoin-project/go-bitfield#24 if necessary. New PR: filecoin-project/go-bitfield#42 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
With current parameters, a partition may have up to 2350 sectors. This issue tracks changing the maximum in the cbor-gen project so that we can store a full partition of maximally-spaced, maximum-magnitude sector numbers safely.
The text was updated successfully, but these errors were encountered: