Skip to content
This repository has been archived by the owner on Sep 5, 2020. It is now read-only.

Add LZ4 support #18

Open
1 of 8 tasks
leandor opened this issue Nov 5, 2016 · 1 comment
Open
1 of 8 tasks

Add LZ4 support #18

leandor opened this issue Nov 5, 2016 · 1 comment
Assignees
@leandor
Labels
enhancement Request to add or enhance a feature.
Milestone
0.8.0

Comments

@leandor
Copy link
Contributor

leandor commented Nov 5, 2016 •
edited
Loading

(see original discussion here too -> wrye-bash/wrye-bash#338)

The official lz4 is here: https://github.com/lz4/lz4

TODO

  • Incorporate dependence on LZ4 distribution
  • (1) Wrap simple functions from the library into python functions
  • Add python tests for simple functions in (1)
  • (2) Next step is to wrap archive support functions
  • Add python tests for (2)
  • (3) Implement high level interface supporting python idioms and primitives based on the simple wrappers on (1) & (2)
  • Add tests for functionality on (3)
  • Documentation
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Add LZ4 dependencies to CBash build system
b231310
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Added link dependency to cint module
7bc3f8b
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Added GSL (https://git.io/vX8Ln)
c5da283 
Added dependency to GSL for coding safer low level C++ when
interacting with the raw lz4 API.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Update test_initializer to be static
e7cafcf 
That way each translation unit that defines a module can have the same
private declaration and the linker won't complain.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Started wrapping the LZ4 API
7caacde 
Started definition of internal redirection functions under
namespace lz4::api that have a safer interface in general by using
constructs taken from the CPP guidelines.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Added init.cpp for the lz4 sub-module
c2331f1 
Added definition of pybind sub-module for containing the lz4
encapsulation.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Fixed test runner target
4c7f960 
Test runner through CMake was broken because it still looked for the
old test1.py files.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Updated test_imports
6cc87c9 
Changed test_imports.py to try to import submodule lz4
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Moved implementation to the header
a861da4 
Moved the implementation from the `api.cpp` to the header `api.h` to
allow for using constexpr for re-declaring constants, and to allow for
inlining when useful.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Published the versionNumber function
71ea5cc 
Published the `versionNumber` function as `lz4.version()` and added a
test case for checking if it's working.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Published the version function as a tuple
787a3fd 
Published a `version` function that returns the version as a tuple
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Moved inline functions to a separate header
9a6eeab 
Moved a bunch of inline function to a separate `api_impl.h` header to
avoid polluting too much the main header.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Added function to return the version string
6583eab 
Added a VERSION_STRING() function that gives the version string in
the standard format.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Rename init.cpp -> module.cpp
38f2581 
As it's the file that will be defining the main api.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Added python implementation of compress
51d9780 
Published compress_default function as a simple python
implementation, and added a test to see if it works.
leandor added a commit that referenced this issue Nov 5, 2016
@leandor
(#18) Specified safer types as source
a0f676f 
Source buffers should be const and point to const content.
@leandor leandor added enhancement Request to add or enhance a feature. goal Long term goal. May be code-related or a meta goal. papi Issue is primarily related to the Python API. labels Nov 5, 2016
@Utumno
Copy link
Member

Utumno commented Nov 5, 2016 •
edited
Loading

Thanks @leandor !

I really just need to decompress (save headers for SSE and bsas/b2as not sure what :P)

I guess this now depends on CBash 7 integration ?

EDIT: I first saw this and then the other (more noisy) issue - wrye-bash/wrye-bash#338

@leandor leandor mentioned this issue Nov 5, 2016
Closed
@leandor leandor self-assigned this Nov 5, 2016
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Specified safer types as source
a4eb9d8 
Source buffers should be const and point to const content.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Moved info functions out of namespace api
fc66208 
Since they just redirect internal constants, no need for them to be
part of the internal "hidden" API.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Moved info functions out of namespace api
2dfbb81 
Functions used are in the base namespace.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Moved compress from lambda to named func
7022b9a 
`compress` was implemented inlined as a lambda in the module
initialization.

Since it wasn't a "trivial" implementation, I think it's better for it
to be a "real" function.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Moved initialization block to main.cpp
5ce29d0 
Moved the initialization block that has the `lz4` module declarations
to a new `main.cpp` file that will only contain the delarations, since
the actual implementations remain in `module.cpp`.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Fix of incorrect constant name
e3a4b73
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Better error messages
45c2037 
Included boost::format for generating formatted strings in the
exception messages.
leandor added a commit that referenced this issue Nov 6, 2016
@leandor
(#18) Added implementation of decompress
33f867b 
Published basic decompress() function. Next in the line is to publish
the actual functions that are needed for compression/decompression
support of files.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Started wrapping the LZ4 API
1c0d854 
Started definition of internal redirection functions under
namespace lz4::api that have a safer interface in general by using
constructs taken from the CPP guidelines.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added init.cpp for the lz4 sub-module
7a55ed5 
Added definition of pybind sub-module for containing the lz4
encapsulation.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Fixed test runner target
19f6718 
Test runner through CMake was broken because it still looked for the
old test1.py files.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Updated test_imports
922ad84 
Changed test_imports.py to try to import submodule lz4
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved implementation to the header
0fd43dd 
Moved the implementation from the `api.cpp` to the header `api.h` to
allow for using constexpr for re-declaring constants, and to allow for
inlining when useful.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Published the versionNumber function
aba1b73 
Published the `versionNumber` function as `lz4.version()` and added a
test case for checking if it's working.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Published the version function as a tuple
65fd9f7 
Published a `version` function that returns the version as a tuple
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved inline functions to a separate header
8590470 
Moved a bunch of inline function to a separate `api_impl.h` header to
avoid polluting too much the main header.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added function to return the version string
66d4361 
Added a VERSION_STRING() function that gives the version string in
the standard format.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Rename init.cpp -> module.cpp
2f55a7f 
As it's the file that will be defining the main api.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added python implementation of compress
387f10c 
Published compress_default function as a simple python
implementation, and added a test to see if it works.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Specified safer types as source
8be081b 
Source buffers should be const and point to const content.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Specified safer types as source
deef7db 
Source buffers should be const and point to const content.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved info functions out of namespace api
46ea1ec 
Since they just redirect internal constants, no need for them to be
part of the internal "hidden" API.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved info functions out of namespace api
f606f1c 
Functions used are in the base namespace.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved compress from lambda to named func
6eda765 
`compress` was implemented inlined as a lambda in the module
initialization.

Since it wasn't a "trivial" implementation, I think it's better for it
to be a "real" function.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Moved initialization block to main.cpp
88f2de6 
Moved the initialization block that has the `lz4` module declarations
to a new `main.cpp` file that will only contain the delarations, since
the actual implementations remain in `module.cpp`.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Fix of incorrect constant name
d68e263
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Better error messages
3078f51 
Included boost::format for generating formatted strings in the
exception messages.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added implementation of decompress
270f2c2 
Published basic decompress() function. Next in the line is to publish
the actual functions that are needed for compression/decompression
support of files.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Cosmetic change on pytest import
f6039e0
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added a fixture for accessing data files
352d154 
Added a new fixture for using data files on test cases that
require generating/validating execution against 'model' data.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Updated testCompress to use new fixture
f71fbd6 
Added new data files generated by the tool LZ4.exe as model for
checking  the function implementation.

As they are now, the tests don't pass because the tool writes the
output  data with a specific format generated by the 'frame'
functions, instead of the simple ones that I was publishing.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added a testDecompress case
161fbf2 
Test case for the decompress function.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Removed #include not needed anymore
5e4b6de
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added 'model'data for high-compression
3df7ca7 
Data files used for testing the high compression mode.
leandor added a commit that referenced this issue Jul 1, 2017
@leandor
(#18) Added #include for lz4frame.h api
bba9208 
This is the API that is required to compress/decompress files as the
LZ4.exe tool does.
@Infernio Infernio added this to the 0.8.0 milestone Sep 20, 2019
@Infernio Infernio removed papi Issue is primarily related to the Python API. goal Long term goal. May be code-related or a meta goal. labels Sep 20, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@leandor leandor

Labels
enhancement Request to add or enhance a feature.
Projects
None yet
Milestone
0.8.0
Development

No branches or pull requests
3 participants
@leandor @Utumno @Infernio