feat: Interpret data descriptors when reading zip file from (read, nonseek) stream

[0xCCF4]

When reading a zipfile from within a zipfile, the inside zipfile stream is not seekable. Given the case that we are dealing with large files, it might be unfeasible to buffer the whole nested file to RAM. It may be better to stream the zip file instead. Relying on the local file header and data descriptors to determine the entries of the zip file. Completely ignoring the central directory at the file end.

Current project state:

• There exists a method for streaming a non-seekable zip file stream.
• This method does not handle data descriptors
• Zip files with a data descriptor can not be processed by this function

My contribution:

• Changed this method to allow parsing a zip file with data descriptors
• Added tests for this feature

Current state:

• Prepare the library for reading without requiring a Take object
• Change the CRC checker to allow setting the CRC to check in the end
• Security considerations regarding this zip reading approach
• Add a custom reader the looks ahead for a data descriptor and then terminates the reading of a zip file content

Security Considerations:
Reading a zip archive that way without knowing the file size in advance may result in parsing inconsistency. Since the file content of a single file may be attacker controlled, we must assume that an attacker may craft a file that contains a sequence looking like a data descriptor. After that an attacker might put the header of a new zip file entry.
Parsing the file in streaming fashion will return two files. Parsing with the central directory information will results in just one file read. This should be regarded as security risk. Still, allowing to read zip files with data descriptors in a streamed fashion is useful for some applications. I therefore introduced the types UntrustedValue and MaybeUnstrusted. See https://github.com/0xCCF4/zip2/blob/f6b5da958028c5cb90d6de7b5b56a378de52fc95/src/result.rs#L110
If a library user would like to use this streaming functionality this security risk must be explicitly accepted.

Related issues:
#162

[0xCCF4]

The changes kind of exploded 🙈 😅 - but now it finally works. Looking forward to a review.
Best regards

[Pr0methean]

Decompressing readers only exist when the feature flag for their compression method is enabled. I may have more comments later.

[Pr0methean]

Seems mostly functionally correct, but I have some performance concerns.

[Pr0methean]

Reading one byte at a time like this won't be efficient. Instead, read a block into the look-ahead buffer and use memchr::memmem::find to check for a data descriptor signature. (Be sure to leave size_of::<Magic>() bytes in the buffer, in case the descriptor signature straddles two blocks!)

[Pr0methean]

interpret already checks whether the magic matches.

Suggested change

	if spec::Magic::from_first_le_bytes(&self.look_ahead_buffer)
	== spec::Magic::DATA_DESCRIPTOR_SIGNATURE
	{
	// potentially found a data descriptor
	// check if size matches
	let data_descriptor = match ZipDataDescriptor::interpret(&self.look_ahead_buffer) {
	Ok(data_descriptor) => data_descriptor,
	Err(_) => return None,
	};
	let data_descriptor_size = data_descriptor.compressed_size;
	if data_descriptor_size == self.number_read_total_actual as u32 {
	return Some(data_descriptor);
	}
	}
	let Ok(data_descriptor) = ZipDataDescriptor::interpret(&self.look_ahead_buffer) else {
	return None;
	};
	let data_descriptor_size = data_descriptor.compressed_size;
	if data_descriptor_size == self.number_read_total_actual as u32 {
	Some(data_descriptor)
	} else {
	None
	}

[Pr0methean]

Check the CRC32 here as well.

[0xCCF4]

This probably needs some architectural changes. The function above operates on the compressed zip file. The CRC is calculated from the uncompressed data. With the current architecture checking CRC in this function is not possible.

Currently I have no good idea to solve this. Still, the CRC is checked, so if an "data descriptor" occurs in the stream - Which is only the case if the file size in the data descriptor is correct - the CRC is likely wrong. In this case only the first half/or part of the file is returned to the user, but then an error is thrown when the CRC check fails; reaching the stream EOF.

[0xCCF4]

Review todos:

• Simple review suggestions
• Convert Vec to VecDequeue
• Optimize read_a_byte to read more than a byte at once 😄
• Check CRC once a potential data descriptor is found