Skip to content

Commit

Permalink
Version 0.1.1: better progress indication and ignoring of system and …
Browse files Browse the repository at this point in the history 
…hidden files
  • Loading branch information
@AlexTMjugador
AlexTMjugador committed Jun 14, 2020
1 parent 8c383a3 commit e079d9e
Show file tree
Hide file tree
Showing 5 changed files with 124 additions and 124 deletions.
3 changes: 1 addition & 2 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "packsquash"
version = "0.1.0"
version = "0.1.1"
authors = ["AlexTMjugador"]
edition = "2018"

Expand All @@ -13,7 +13,6 @@ codegen-units = 1
lazy_static = "1.4.0"
clap = "2.33.1"
simple-error = "0.2.1"
pbr = "1.0.2"
num_cpus = "1.13.0"
threadpool = "1.8.1"
tempfile = "3.1.0"
Expand Down
43 changes: 24 additions & 19 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,29 +17,33 @@ You can get a pre-built release for 64-bit Windows, Linux and macOS systems from

## Usage
```
PackSquash 0.1.0
PackSquash 0.1.1
AlexTMjugador
Lossily compresses and prepares Minecraft resource packs for distribution
USAGE:
packsquash [FLAGS] [OPTIONS] <resource pack directory> [result ZIP file]
FLAGS:
-c, --compress-compressed If specified, resource files that are already compressed by design (like PNG and OGG
files) will be losslessly compressed again in the result ZIP file, unless that yields
no significant space savings. This is disabled by default, as it is expected that
compressing already compressed file formats yields marginal space savings that do not
outweigh the time cost
-o, --zip-obfuscation If provided, the generated ZIP file will not be able to be read by some common
programs, like WinRAR, 7-Zip or unzip, but will function normally within Minecraft. It
will also not duplicate some metadata in the ZIP file, reducing its size a bit more.
You shouldn't rely on this feature to keep the resources safe from unauthorized
ripping, and there is a low possibility that a future version of Minecraft rejects
these ZIP files. Therefore, it is disabled by default
-h, --help Prints help information
-n, --skip-pack-icon If specified, the pack icon in pack.png will be skipped and not included in the
resulting file
-V, --version Prints version information
-c, --compress-compressed
If specified, resource files that are already compressed by design (like PNG and OGG files) will be
losslessly compressed again in the result ZIP file, unless that yields no significant space savings. This is
disabled by default, as it is expected that compressing already compressed file formats yields marginal
space savings that do not outweigh the time cost
-a, --do-not-ignore-system-and-hiden-files
If specified, filenames which start with a dot or are likely generated by a software system (Windows,
VCS...) won't be skipped without a message
-o, --zip-obfuscation
If provided, the generated ZIP file will not be able to be read by some common programs, like WinRAR, 7-Zip
or unzip, but will function normally within Minecraft. It will also not duplicate some metadata in the ZIP
file, reducing its size a bit more. You shouldn't rely on this feature to keep the resources safe from
unauthorized ripping, and there is a low possibility that a future version of Minecraft rejects these ZIP
files. Therefore, it is disabled by default
-h, --help Prints help information
-n, --skip-pack-icon
If specified, the pack icon in pack.png will be skipped and not included in the resulting file
-V, --version Prints version information
OPTIONS:
-t, --threads <THREADS> The number of resource pack files to process in parallel, in different threads. By
Expand All @@ -54,6 +58,7 @@ ARGS:
## Potentials for improvement
These are some tweaks to the application which could further improve the compression it achieves, but are not scheduled for a release. Feel free to submit a PR with some (or all) of these changes!

* Parse JSON files to determine unused assets (models, textures, sounds...), and skip them from the result ZIP file
* Understand the TTF and OGG formats more, to remove metadata and further compress them
* Implement an "append mode", which only adds to a result ZIP file resource pack files which are newer than it (so the entire ZIP file isn't generated again if a single file changes)
* Determine unused assets (models, textures, sounds...) by analyzing JSON files, and skip them from the result ZIP file.
* Understand the OGG format more, to remove metadata from it.
* Implement TTF minification.
* Implement an "append mode", which only adds to a result ZIP file resource pack files which are newer than it (so the entire ZIP file isn't generated again if a single file changes).
97 changes: 59 additions & 38 deletions src/main.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,27 @@ mod micro_zip;
mod resource_pack_file;

use std::error::Error;
use std::io::Write;
use std::ffi::OsStr;
use std::iter::FromIterator;
use std::path::PathBuf;
use std::str::FromStr;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
use std::{cmp, env, fs, process};

use threadpool::ThreadPool;

use clap::{App, ArgMatches};
use pbr::MultiBar;
use simple_error::SimpleError;

use micro_zip::MicroZip;
use micro_zip::ZipFileType;

use lazy_static::lazy_static;

lazy_static! {
static ref EMPTY_OS_STR: &'static OsStr = OsStr::new("");
}

fn main() {
let parameters = App::new("PackSquash")
.version(env!("CARGO_PKG_VERSION"))
Expand All @@ -29,6 +33,7 @@ fn main() {
[number of threads] -t --threads=[THREADS] 'The number of resource pack files to process in parallel, in different threads. By default, a value appropriate to the CPU the program is running on is used, but you might want to decrease it to reduce memory usage'
[ZIP obfuscation] -o --zip-obfuscation 'If provided, the generated ZIP file will not be able to be read by some common programs, like WinRAR, 7-Zip or unzip, but will function normally within Minecraft. It will also not duplicate some metadata in the ZIP file, reducing its size a bit more. You shouldn't rely on this feature to keep the resources safe from unauthorized ripping, and there is a low possibility that a future version of Minecraft rejects these ZIP files. Therefore, it is disabled by default'
[Compress already compressed files] -c --compress-compressed 'If specified, resource files that are already compressed by design (like PNG and OGG files) will be losslessly compressed again in the result ZIP file, unless that yields no significant space savings. This is disabled by default, as it is expected that compressing already compressed file formats yields marginal space savings that do not outweigh the time cost'
[Do not ignore system and hidden files] -a --do-not-ignore-system-and-hiden-files 'If specified, filenames which start with a dot or are likely generated by a software system (Windows, VCS...) won't be skipped without a message'
<resource pack directory> 'The directory where the resource pack to process is'
[result ZIP file] 'The path to the resulting ZIP file, ready to be distributed'
").get_matches_from(env::args());
Expand All @@ -43,14 +48,14 @@ fn main() {
}

fn run(parameters: ArgMatches) -> Result<(), Box<dyn Error>> {
let mut progress = MultiBar::new();

let skip_pack_icon = parameters.is_present("skip pack icon");
let use_zip_obfuscation = parameters.is_present("ZIP obfuscation");
let compress_already_compressed = parameters.is_present("Compress already compressed files");
let ignore_system_hidden_files =
!parameters.is_present("Do not ignore system and hidden files");
let threads = cmp::max(
match parameters.value_of("number of threads") {
Some(threads_string) => FromStr::from_str(threads_string),
Some(threads_string) => threads_string.parse(),
None => Ok(num_cpus::get() * 2)
}?,
1
Expand All @@ -71,14 +76,14 @@ fn run(parameters: ArgMatches) -> Result<(), Box<dyn Error>> {
canonical_root_path,
&file_count,
&file_process_thread_pool,
&mut progress,
skip_pack_icon,
compress_already_compressed,
&micro_zip
&micro_zip,
ignore_system_hidden_files
)?;

// Listen for progress bar changes and wait until all is done
progress.listen();
// Wait until all the work is done
file_process_thread_pool.join();

// Append the central directory
println!("> Finishing up resource pack ZIP file...");
Expand All @@ -96,15 +101,15 @@ fn run(parameters: ArgMatches) -> Result<(), Box<dyn Error>> {
/// Recursively processes all the resource pack files in the given path,
/// storing the resulting processed resource pack file data in a vector.
#[allow(clippy::too_many_arguments)] // Not really much point in grouping the arguments
fn process_directory<T: Write>(
fn process_directory(
canonical_root_path: &PathBuf,
current_path: &PathBuf,
file_count: &Arc<AtomicUsize>,
thread_pool: &ThreadPool,
progress: &mut MultiBar<T>,
skip_pack_icon: bool,
compress_already_compressed: bool,
micro_zip: &Arc<MicroZip>
micro_zip: &Arc<MicroZip>,
ignore_system_hidden_files: bool
) -> Result<(), Box<dyn Error>> {
for entry in fs::read_dir(current_path)? {
let entry = entry?;
Expand All @@ -114,39 +119,49 @@ fn process_directory<T: Write>(
let is_directory = file_type.is_dir();
let is_file = file_type.is_file();

// Check whether this is a system or hidden file, and if so skip it without printing anything
if ignore_system_hidden_files {
let file_name = match path.file_name().unwrap_or(&EMPTY_OS_STR).to_str() {
Some(file_name) => file_name,
None => {
return Err(Box::new(SimpleError::new(
"A file name contains invalid UTF-8 codepoints"
)))
}
};

let dot_file = file_name.chars().next().unwrap_or('x') == '.';

if dot_file || is_system_file(file_name, is_directory, is_file) {
continue;
}
}

if is_directory || is_file {
if is_directory {
process_directory(
canonical_root_path,
&path,
file_count,
&thread_pool,
progress,
skip_pack_icon,
compress_already_compressed,
micro_zip
micro_zip,
ignore_system_hidden_files
)?;
} else {
let mut relative_path = relativize_path_for_zip_file(&canonical_root_path, &path)?;
let relative_path_str = match relative_path.to_str() {
Some(path) => String::from(path),
None => {
return Err(Box::new(SimpleError::new(
"The path contains invalid UTF-8 codepoints"
"A path contains invalid UTF-8 codepoints"
)))
}
};

let path_in_root = path.parent().unwrap() == canonical_root_path;

let mut file_progress = progress.create_bar(0);
file_progress.message(format!("> {}... ", relative_path_str).as_str());
file_progress.show_tick = true;
file_progress.show_speed = false;
file_progress.show_percent = false;
file_progress.show_counter = false;
file_progress.tick();

let file_count = file_count.clone();
let micro_zip = micro_zip.clone();

Expand All @@ -156,7 +171,7 @@ fn process_directory<T: Write>(
skip_pack_icon,
path_in_root
) {
let result = resource_pack_file.process(&mut file_progress);
let result = resource_pack_file.process();

if result.is_ok() {
let (processed_bytes, message) = result.ok().unwrap();
Expand All @@ -169,33 +184,29 @@ fn process_directory<T: Write>(
&relative_path,
ZipFileType::RegularFile,
&processed_bytes,
resource_pack_file.is_compressed() && !compress_already_compressed,
&mut file_progress
resource_pack_file.is_compressed() && !compress_already_compressed
);

if add_result.is_ok() {
file_progress.finish_println(&format!(
"> {}: {}",
relative_path_str, message
));
println!("> {}: {}", relative_path_str, message);

file_count.fetch_add(1, Ordering::Relaxed);
} else {
file_progress.finish_println(&format!(
"> {}: Error occurred while adding to the ZIP file: {}",
println!(
"> {}: An error occurred while adding to the ZIP file: {}",
relative_path_str,
add_result.err().unwrap()
));
);
}
} else {
file_progress.finish_println(&format!(
"> {}: Error occurred while processing: {}",
println!(
"> {}: An error occurred while processing: {}",
relative_path_str,
result.err().unwrap()
));
);
}
} else {
file_progress.finish_println(&format!("> {}: Skipped", relative_path_str));
println!("> {}: Skipped", relative_path_str);
}
});
}
Expand Down Expand Up @@ -227,3 +238,13 @@ fn relativize_path_for_zip_file(

Ok(relativized_path)
}

/// Checks whether the specified path likely represents a file generated for use
/// with some specific software that is not Minecraft, like VCS or operating system components.
fn is_system_file(file_name: &str, _is_directory: bool, is_file: bool) -> bool {
(
file_name == "desktop.ini" ||
file_name == "thumbs.db" ||
file_name == "README.md"
) && is_file
}
27 changes: 11 additions & 16 deletions src/micro_zip.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ use simple_error::SimpleError;

use crc32fast::Hasher;

use pbr::ProgressBar;

/// Compression methods supported by Minecraft in a ZIP file.
enum ZipCompressionMethod {
DEFLATE,
Expand Down Expand Up @@ -106,7 +104,7 @@ impl ZipFileType {
fn to_ms_dos_attributes(&self) -> u8 {
match self {
ZipFileType::RegularFile => 0x01 // FILE_ATTRIBUTE_READONLY
//ZipFileType::Folder => 0x10 // FILE_ATTRIBUTE_DIRECTORY (16)
//ZipFileType::Folder => 0x10 // FILE_ATTRIBUTE_DIRECTORY (16)
}
}
}
Expand Down Expand Up @@ -153,7 +151,7 @@ struct CentralDirectoryFileHeader {
impl Default for CentralDirectoryFileHeader {
fn default() -> CentralDirectoryFileHeader {
CentralDirectoryFileHeader {
signature: 0x02014b50, // Magic number from the spec
signature: 0x02014B50, // Magic number from the spec
version_made_by: 20, // MS-DOS OS, 2.0 ZIP spec compliance
version_needed_to_extract: 20, // MS-DOS OS. Zip spec version may be higher than actually needed
general_purpose_bit_flag: 0, // Irrelevant metadata, always 0
Expand Down Expand Up @@ -225,7 +223,7 @@ struct LocalFileHeader<'a> {
impl<'a> Default for LocalFileHeader<'a> {
fn default() -> LocalFileHeader<'a> {
LocalFileHeader {
signature: 0x04034b50, // Magic number from the spec
signature: 0x04034B50, // Magic number from the spec
version_needed_to_extract: 20,
general_purpose_bit_flag: 0,
last_mod_time: 0,
Expand Down Expand Up @@ -277,7 +275,7 @@ struct CentralDirectoryEndRecord {
impl Default for CentralDirectoryEndRecord {
fn default() -> CentralDirectoryEndRecord {
CentralDirectoryEndRecord {
signature: 0x06054b50, // Magic number from the spec
signature: 0x06054B50, // Magic number from the spec
disk_number: 0, // No multi-disk support
central_directory_start_disk: 0, // No multi-disk support
comment_length: 0, // No comments
Expand Down Expand Up @@ -328,13 +326,12 @@ impl MicroZip {

/// Adds a file to the to-be-finished ZIP file represented by this struct,
/// returning an error if something went wrong during the operation.
pub fn add_file<W: Write>(
pub fn add_file(
&self,
relativized_path: &PathBuf,
file_type: ZipFileType,
data: &[u8],
skip_compression: bool,
progress: &mut ProgressBar<W>
skip_compression: bool
) -> Result<(), Box<dyn Error>> {
if self.writing_central_directory.load(Ordering::SeqCst) {
return Err(Box::new(SimpleError::new(
Expand Down Expand Up @@ -383,9 +380,11 @@ impl MicroZip {
(compressed_data, Some(crc)) => {
(ZipCompressionMethod::STORE, compressed_data, crc)
}
_ => return Err(Box::new(
SimpleError::new("The contract of compress was violated"))
)
_ => {
return Err(Box::new(SimpleError::new(
"The contract of compress was violated"
)))
}
}
} else {
ZipCompressionMethod::best_compress(data)
Expand All @@ -400,8 +399,6 @@ impl MicroZip {
)));
}

progress.tick();

// Generate the local file header
let actual_path_str = path_str_arc.as_ref().as_bytes();
let local_file_header = LocalFileHeader {
Expand Down Expand Up @@ -454,8 +451,6 @@ impl MicroZip {
}
};

progress.tick();

partial_central_directory_entries.push(PartialCentralDirectoryFileHeader {
compression_method: zip_compression_method_field,
crc_32: crc,
Expand Down
Loading

0 comments on commit e079d9e

Please sign in to comment.