v0.3.1

These are the downloads and release notes for the PackSquash v0.3.1 release.

For information about how to install and use PackSquash, please see the installation guide and the options files documentation.

v0.3.1 is a patch update for v0.3.0, focused on fixing most known defects, introducing backward-compatible tweaks and improvements, and preparing its codebase for more extensive upgrades that will come in the future. Most users are encouraged to update to v0.3.1 at their convenience, which is meant to be a drop-in replacement for v0.3.0.

Nonetheless, those using PackSquash on computers with so underpowered x64 CPUs, in unsupported scenarios (i.e., for Bedrock Edition packs), passing uncommon combinations of non-default values for some options, optimizing packs that contain audio files encoded with uncommon parameters, or using PackSquash on Linux distros that ship older software may find incompatibilities. Please check the changelog below for more detailed information about these incompatibilities.

📌 Changelog from v0.3.0

Additions

• As requested in #44, basic support for Minecraft 1.17 include shaders was added. Please note that PackSquash is, for now, more restrictive than Minecraft when parsing these files, imposing that they must be a syntactically valid translation unit by themselves. In other words, include shaders that would only be valid within an external declaration of another shader, such as those containing only variable assignment expressions, are not accepted. We are looking forward to lifting this restriction in the future. Thanks to Ancientkingg#0420 over Discord for making me notice this quirk.
• Language files used in Minecraft 1.12.2 and older versions are now fully supported. Thanks to a Discord user whose nickname I can't find right now for bringing this to my attention.
• Added support for the resource pack assets used by the Custom Resources feature of the Minecraft Transit Railway mod, major version 3. Thank you @Kenny-Hui for getting in touch over Discord to suggest this improvement and for your keenness to answer technical questions about the mod. The tracking issue for this feature is #42.
• Custom assets can now be defined in options files by setting the file-specific option force_include to true. This option is honored by PackSquash for files that would otherwise be unrecognized and skipped from the generated ZIP file, and it can be used to include files that are not read by the game in the generated pack, such as credits or license information. Please get in touch if you feel the urge to use this feature to compensate for PackSquash lacking proper support for asset files used by the game, though! Thank you @MinecraftAdmin for suggesting this improvement on issue #40.

Compression improvements

• PackSquash now only processes files whose path in the pack matches a hardcoded list of path patterns. Previously, any file that had a recognized extension, such as .json, was optimized and included in the generated ZIP file, even if there was no way for Minecraft to read it. This improvement allows PackSquash to recognize different asset types much more precisely, enabling more specific and advanced optimizations. It also allowed the PackSquash codebase to be greatly refactored and simplified. However, this is a breaking change for users relying on PackSquash to optimize files that are not used by Minecraft: Java Edition, or only used by unsupported mods that do not follow the same asset location scheme as vanilla Minecraft.
• Merged @sya-ri's PR #45, which ensures that Blockbench cube groups data is removed from JSON model files when debloating is enabled. Thank you very much for the PR!
• The JSON model debloat code now removes keys conventionally used by some authors to hold comment text, but completely ignored by the game.
• The Ogg Vorbis encoder used is now told to disregard ease of seeking of the files it generates, as Minecraft does not need to rewind or fast forward audio. Consequently, longer sounds are now more likely to be transcoded to files with less muxing overhead, slightly reducing their size with no changes to audio quality.
• When keeping the color_quantization_target option value to the new default of auto, now PackSquash won't use the color-quantized representation of a texture if it does not yield space savings. The previous behavior of always quantizing to 256 colors can be restored by explicitly setting color_quantization_target to the previous default value. In addition, a quick, never size-increasing first optimization pass is now done on PNG files, and unless expressly requesting color quantization to be done, its results will be used if PackSquash can't figure out how to make the input PNG smaller. Thanks Mergu#0001 for sharing a previously problematic texture for analysis.
• The dithering effect introduced while performing color quantization on textures is now slightly less intense and can be configured via the color_quantization_dithering_level option. This is meant to balance visual quality with size savings better.
• Several tweaks were made to the Zopfli iteration count calculation code for PNG files to make it behave more as originally envisioned. PackSquash should now spend a bit more time optimizing smaller PNG images as it did on v0.2.1 while still hurrying up for bigger images as introduced on v0.3.0. The default value of the image_data_compression_iterations option was increased to 5 to reflect this change and allow reducing it even more.
• Command function files in data packs are now minified by PackSquash, removing comments and empty lines. Thank you @sya-ri for submitting this improvement on PR #68!
• The sampling_frequency option value can now only lower the sampling frequency of audio files, not increase it. The rationale for the new behavior is that upsampling sounds of an initially lower sampling frequency increases file sizes due to the need to store data about the additional samples, but no audio information is added. However, if the lowest sampling frequency chosen for an audio file is not supported by the encoder used by PackSquash, it will error out. This could also happen before when using certain values for the sampling_frequency option.
• Introduced the internal asset type mask abstraction, used throughout the source code to know what asset types are appropriate for the targeted Minecraft versions. Thanks to this abstraction, PackSquash can skip files only used in older Minecraft versions when the pack targets newer versions according to its pack.mcmeta file, for example. The automatic_asset_types_mask_detection option was added to control how this asset type mask feature operates. Setting the validate_pack_metadata_file option to false will now only work if automatic_asset_types_mask_detection is set to false, too, as this feature needs to validate the pack metadata file too.

Performance improvements

• The thread synchronization primitives used internally were replaced with more efficient implementations, yielding modest total pack processing time savings. Thanks to @clrxbl and jilchu#4474 for your spot-on comments about PackSquash's performance which lead me to make this and other performance-related changes.
• JSON debloating is now limited to files that contain a known-debloatable asset type, and any debloating done is specific to the asset type. In addition to improving performance by doing less unnecessary work, this behavior is more accurate than the previous one of applying every debloating technique to every JSON file. End-users can know whether a JSON file was debloated or not thanks to a different optimization strategy message, which will now have the "and debloated" suffix when debloating.
• Output ZIP file operations now use a single lock to synchronize access to their mutable state instead of several locks acquired in succession, slightly reducing the synchronization cost of these operations.
• The official Linux and Windows PackSquash executables for x64 CPUs now target the second microarchitecture level of x64, allowing PackSquash to use more efficient instructions found on most, but not all, x64 processors. Any full-featured desktop, laptop, workstation, or server CPU manufactured on or after 2009-2010 should be compatible.
• Other minor micro-optimizations.

Pack file validation improvements

• PackSquash now sanity-checks that JSON files belonging to a known asset type (Minecraft model, etc.) are JSON objects. Minecraft and some JSON standards expect JSON data to be contained within a JSON object.
• Added an always_allow_json_comments JSON-specific file option to control whether PackSquash allows comments in the matched files or not. The default value is true, which allows and removes comments from JSON files. When this option is set to false, comments will only be accepted in files whose usual extension ends with an extra c letter, such as .jsonc, .mcmetac or .jpmc.

Fixes

• Fixed issue #43 about shaders containing functions with parameters or a single statement being broken by PackSquash when minifying them. Thanks to @tomlister and @Aiamded for reporting this.
• Shaders that contain a preprocessor directive following an external declaration, such as a variable declaration, are no longer corrupted by PackSquash minifications. Thank you Esron#1571 for reporting this issue over Discord.
• Fixed sounds in resource packs not playing when the zip_spec_conformance_level and percentage_of_zip_structures_tuned_for_obfuscation_discretion options are set to disregard and 0, respectively. Thank you @Aiamded for reporting this issue.
• Resolved issues #37 and #39 by adding the restrictive_banner_layer_texture_format_check and bad_entity_eye_layer_texture_transparency_blending quirks, respectively, which PackSquash automatically applies when necessary by default. Thanks to ! " Marco Pollo#1898 and @Aiamded, respectively, for reporting these problems.
• When PackSquash ends up normalizing two files to the same path, which can happen when changing, for example, the extension of specially named .jsonc files to .json, an error will now happen again, as it did in v0.2.1. v0.3.0 introduced a regression that made it possible to leverage this normalization to store two files with the same name on the generated ZIPs, negatively impacting their sizes for no good reason.

User experience improvements

• PackSquash progress messages are now shown with color and emojis on supported platforms, making them prettier and facilitating the visual location of message types. This work has been done by @sya-ri on PR #46. Thank you!
• A title is now shown on the interactive console or terminal attached to PackSquash, showing the application name and the progress of the optimization process.
• When an attempt is made to reuse a previously generated ZIP file, but that is not possible due to system identifier changes or other reasons, PackSquash will now automatically ignore that previous ZIP file after showing a warning instead of exiting with an error. This change is a concession to the fact that external programs or scripts are likely to handle this condition by just retrying without the previous ZIP file: doing otherwise is not practical in most scenarios. Also, non-savvy end users might be confused by the former behavior. Those who are savvy enough can still interrupt the optimization after reading that warning and apply a fix to the underlying cause, should they choose to do so.
• PackSquash now promptly shows a unique, user-friendly error message if the value of the output_file_path option points to an existing directory, and thus it cannot be a normal file. Experience has shown that this is a common mistake, and Windows reported a generic "access denied" error message after the whole pack was optimized, confusing users.
• The help texts shown when reading options from the standard input and after a pack processing error happens were tweaked, with the intent of making them easier to understand and better highlight helpful solutions or alternatives.
• The error message that shows when trying to reuse an empty or too small previously generated ZIP file was changed to a less generic, more user-friendly one.
• The error message shown when validating a pack.mcmeta file that does not contain a valid pack_format key now correctly points to that key instead of pack_format_version.
• Some common error messages shown for PNG files were changed to be more actionable and user-friendly.
• PackSquash now shows both the count of files it found on the pack directory and the count of pack files that were actually stored on the generated ZIP file. This may be handy for troubleshooting purposes and quickly knowing whether files were skipped.

Distribution changes

• macOS executables now work with the upstream GStreamer libraries instead of Homebrew ones. The migration was necessary because the latest Homebrew packages have broken plugin resolution, and installation instructions were updated accordingly. As a side bonus, now the pitch-shifting feature also works on macOS.
• The PackSquash Linux CI build workflows now generate a PackSquash AppImage. AppImages are universal Linux packages that integrate relatively well with most versions of most major distros. It even is possible to run the generated AppImages on musl Linux environments, such as Alpine, but it's necessary to extract them beforehand on a glibc environment in this case. AppImages are now the recommended way to run PackSquash when more environment-specific packages are not provided or can't be used.
• The PackSquash Linux CI build workflows are now executed in Debian Bullseye containers, which markedly improved build reproducibility and predictability, availability of development packages, and compatibility with a wide range of distros (stable Debian releases tend to ship older, but forward-compatible glibc versions than most other distros). This change should also make it easier to replicate the PackSquash build environment for private builds. However, executables built in Debian Bullseye are not compatible with Ubuntu 18.04 or any distro that uses a glibc version older than 2.31. The recommended migration path for cases where upgrading is not feasible is to use the PackSquash AppImage.
• The distribution of PackSquash executables for Linux is now deprecated. AppImages are a better alternative to them in all regards, as they are self-contained and more compatible across distros. Future releases may stop distributing PackSquash executables for Linux. For now, they are kept to help their users ease and plan for the transition to AppImages or distribution-specific packages.

Codebase changes

• Unit tests for the official Linux AArch64/ARM64 builds are now run via QEMU AArch64 user-mode emulation, increasing the confidence that these builds work as expected for end-users and paving the way towards recognizing them as fully-supported ports.
• An automated Criterion.rs microbenchmark suite was added to the project. The benchmark suite runs automatically on each commit thanks to a new GitHub Actions job, adding a comment and updating a performance evolution graph, enabling continuous performance monitoring and improvement. Future changes may leverage these benchmarks to establish baselines, assessing their performance impact with hard data. @victorlf4 contributed a Python script to sample packs in smaller ones and shared some mathematical optimization ideas. jilchu#4474 and @Aiamded kindly authorized the inclusion of their packs in the benchmarking dataset.
• The Rust source tree was refactored into a Cargo workspace with independent library and CLI packages. This change promotes separation of concerns, as the CLI package uses the library package as a dependency, like external code would, and frees the library package from any UI-related code. It is expected for this change to help achieve a better API design and make it easier to add new UI frontends for PackSquash that may be developed independently from the PackSquash core optimization routines.
• Contributing guidelines were added to the repository. Thank you @sya-ri for suggesting these.
• A code of conduct was added to the repository, documenting the behavior we've already been expecting of the PackSquash contributors community.
• A security policy was added to the repository.
• Updated the codebase to the Rust 2021 edition, bringing along a new dependency resolver that generates potentially smaller executables in some cases.
• The build workflow responsible for generating unstable PackSquash builds is now run at least once a week, helping ensure that PackSquash still builds with newer Rust toolchain versions and that programs consuming the latest unstable build can always download one.
• A rust-toolchain rustup override file was checked-in to the repository, making the nightly Rust toolchain requirement much more transparent for fellow developers and automated tools.
• The static analysis GitHub Actions build workflow tasks are now run in a concurrent, separate job. A step that runs the cargo-deny Cargo plugin was added.
• Some dependencies were patched to avoid several versions of the same transitive dependencies being compiled to the executable. This was discovered thanks to cargo-deny checks. These patches streamlined executable sizes.
• The audio-transcoding Cargo feature was added to choose at compile time whether PackSquash implements audio transcoding functionality using GStreamer. This feature is enabled by default, and all official builds have this feature enabled. Disabling the feature stops the PackSquash executable from linking to GStreamer libraries, which may be helpful for some advanced use cases, but disables parsing of audio file optimization options and the optimizations themselves, turning audio files into assets that are merely copied as-is.
• A logging framework is now used to show status and progress messages, allowing for further expandability. The PACKSQUASH_LOG environment variable can be used to customize what logging messages are shown, whose contents are parsed by the env_logger library, but this is considered for now an implementation detail that may change in the future. @sya-ri did this change as a part of PR #46.

Miscellaneous

• Trailing bytes at the end of PNG files are now considered errors, as they violate the PNG format standard, do not make sense to store, and pose compatibility problems with select programs.
• The ZIP file path handling code was refactored to check the file path length. This saves repeated length checks over the code, making it more efficient and maintainable.
• Several dependency updates, bringing the latest upstream fixes and improvements.
• Updated GitHub Actions build workflow runners to their latest versions.
• Other minor refactors to PackSquash, its documentation, and the GitHub Actions build workflow.
• JetBrains now provides Open Source Development licenses of their development tools for core PackSquash contributors.

Options files changes

• The following options are new (i.e. they were introduced in v0.3.1 and do not have an equivalent in v0.3.0):
  • automatic_asset_types_mask_detection
  • color_quantization_dithering_level
  • always_allow_json_comments
  • minify_legacy_language
  • strip_legacy_language_bom
  • minify_command_function
  • force_include
• The following values were added to the color_quantization_target option:
  • auto (new default value)
• The following values were added to the work_around_minecraft_quirks option:
  • restrictive_banner_layer_texture_format_check
  • bad_entity_eye_layer_texture_transparency_blending
• The following values were added to the allow_mods option:
  • Minecraft Transit Railway 3
• The default value of image_data_compression_iterations was changed to 5 from 3.

As usual, we're very grateful for the community support we've received during the development of this release. Some highlights include:

• @sya-ri, who continued submitting significant code contributions.
• Those few heroes that donated some money to help keep the project going ❤️
• And last but not least, those that were mentioned through the changelog.

Check out some of those who helped out below!