Skip to content

Commit

Permalink
Add Amazon S3's parallel SHA2-256 (#343)
Browse files Browse the repository at this point in the history
When data is uploaded to S3 in can be chunked. Each chunk is hashed, those hashes
are then concatenated and hashes again into a single hash.

Quilt is using 8MiB chunks, but increases the size in case the input is large, as the
maximum size of chunks is 10k.

For all details about the hash algorithm see:
https://github.com/quiltdata/quilt/blob/fb36269def0f8b811e0c1293fcf3e853732a7b71/docs/sha2-256-chunked.md
  • Loading branch information
drernie authored Feb 23, 2024
1 parent d06fc61 commit 11bd20b
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions table.csv
Original file line number Diff line number Diff line change
Expand Up @@ -510,6 +510,7 @@ poseidon-bls12_381-a2-fc1-sc, multihash, 0xb402, draft, Pose
rdfc-1, ipld, 0xb403, draft, The result of canonicalizing an input according to RDFC-1.0 and then expressing its hash value as a multihash value.
ssz, serialization, 0xb501, draft, SimpleSerialize (SSZ) serialization
ssz-sha2-256-bmt, multihash, 0xb502, draft, SSZ Merkle tree root using SHA2-256 as the hashing function and SSZ serialization for the block binary
sha2-256-chunked, multihash, 0xb510, draft, Hash of concatenated SHA2-256 digests of 8*2^n MiB source chunks; n = ceil(log2(source_size/(10^4 * 8MiB)))
json-jcs, ipld, 0xb601, draft, The result of canonicalizing an input according to JCS - JSON Canonicalisation Scheme (RFC 8785)
iscc, softhash, 0xcc01, draft, ISCC (International Standard Content Code) - similarity preserving hash
zeroxcert-imprint-256, zeroxcert, 0xce11, draft, 0xcert Asset Imprint (root hash)
Expand Down

0 comments on commit 11bd20b

Please sign in to comment.