Question: Easiest Path to using fastcrc with crc-rs CUSTOM_ALG functionality

[gaiuscosades]

Thank you for providing this great library!

I think I will be not alone wondering, what the easiest ways of using this library with a custom CRC algorithm definition would be.
Is it true that inserting all the necessary definitions directly into crc-rs and rebuilding crc-rs from scratch while extending this library by the newly defined algorithms is the only way?

Sorry for the beginner question, but I have never used anything relying on Rust before.

Thank you very much.

[overcat]

Hi @gaiuscosades

Thanks for creating the issue. Currently, since crc-rs requires algorithms to be provided at compile time (see mrhooray/crc-rs#68), I think fastcrc cannot support dynamic algorithms for the time being. There may be other solutions, but since I'm so short on time right now, it may take a while to explore if there are other solutions.

Also I would like to ask what kind of algorithm do you need now, and if it's a fixed algorithm, I'll be happy to add it and release a new version.

[gaiuscosades]

@overcat Thank you for taking the time to respond!

Currently, since crc-rs requires algorithms to be provided at compile time (see mrhooray/crc-rs#68), I think fastcrc cannot support dynamic algorithms for the time being. There may be other solutions, but since I'm so short on time right now, it may take a while to explore if there are other solutions.

Thanks for confirming what I suspected. As far as I understand the calculation tables get initialized a compiletime which makes the calculation as fast as it is, while having minimal codesize(~700kiBiByte) atm, because no python wrapping is needed for this data.

Also I would like to ask what kind of algorithm do you need now, and if it's a fixed algorithm, I'll be happy to add it and release a new version.

Yes they are fixed, with an exotic init, but this is already perfectly abstracted by fastcrc.

For Naming I would use what the Wikipedia Article on CRC puts forward for these polynomials, so that someone elese might find them, while I don't know if they are specified in any norming documents with other names.

const CRC_16_IBM_REFIN: Algorithm<u16> = Algorithm {
    width: 16,
    poly: 0x8005,
    init: 0xf0f0,
    refin: true,
    refout: false,
    xorout: 0x0000
};

const CRC_32K_REVERSED_RECIPROCAL_REFIN: Algorithm<u32> = Algorithm {
    width: 32,
    poly: 0xba0dc66b,
    init: 0xf0f0f0f0,
    refin: true,
    refout: false,
    xorout: 0x0000
};

I would propose that, if someone experienced with Rust would need a similar functionality, that an interface for runtime generating additional calculation tables gets added, which can then be passed into the library as an argument to a unqiue calculation function like the following:

fastcrc.crc16.custom_crc_algorithm(data, configuration_with_calculation_table)
fastcrc.crc32.custom_crc_algorithm(data, configuration_with_calculation_table)
fastcrc.crc64.custom_crc_algorithm(data, configuration_with_calculation_table)

This would give the flexibility of adding custom definitions, sacrificing a little on performance, while not increasing memory footprint for everyone with loads of unneeded algos.

Thank you very much for your efforts and the great library you are providing!

[overcat]

Hi @gaiuscosades

I've released a new version that adds the above two functions, you can install it with the following command, you'd better lock the version in the dependencies file.

pip install fastcrc==0.2.1.dev1

Example:

from fastcrc import crc16, crc32

data = b"123456789"

print(f"crc16: {crc16.ibm_refin(data)}")
print(f"crc16: {crc16.ibm_refin(b'6789', crc16.ibm_refin(b'12345'))}")
print(f"crc32: {crc32.reversed_reciprocal_refin(data)}")

I'll keep this PR open until I have time to find a better way.

[gaiuscosades]

@overcat

Thank you very much for all your effort!
I will test this as soon as all the wheel files are up.

Is there any way I could contribute monetarily to this project (Do you have a kind of "Buy me a coffee" Link)?

I will use this code on two plattforms, one of which worked fine already with the stable versions wheel.
For the other ones I would have to setup the build environment myself. (manylinux2014_armv7l)
Is there a way this could be integrated to the already established automatic building process?

[overcat]

Hi @gaiuscosades, I am not going to accept any donations. It's a tiny project, one of the reasons I built it was to try out PyO3, I'm happy to have users using it and giving feedback, thank you again for your kind words.

Currently I'm using maturin-action to build wheels for multiple platforms, but I'm having some trouble now that it doesn't recognize the Python installed via actions/setup-python@v1 on Linux platforms, I'll see if I can build wheels for as many platforms as possible tomorrow.

You can view existing wheels here https://pypi.org/project/fastcrc/0.2.1.dev5/#files

[gaiuscosades]

@overcat

I am not going to accept any donations

Let me know if it changes, as I suspect that this project could get some traction in the future ;D

Currently I'm using maturin-action to build wheels for multiple platforms, but I'm having some trouble now that it doesn't recognize the Python installed via actions/setup-python@v1 on Linux platforms, I'll see if I can build wheels for as many platforms as possible tomorrow.

I suspected something like this while monitoring pypi. Will be great when most platforms get supported all out of the box. Calculating crc's without overhead is especially an issue on all the arm based embedded platforms out there.

I already tested 0.2.1.dev4-cp310-none-win_amd64 without any noticable issues!

[gaiuscosades]

@overcat

slight correction: Koopman/Wikipedia calls this polynomial: CRC_32K_REVERSED_RECIPROCAL
So I would change the function to be:

crc32.k_reversed_reciprocal_refin(data)

[overcat]

Hi @gaiuscosades

slight correction: Koopman/Wikipedia calls this polynomial: CRC_32K_REVERSED_RECIPROCAL

Fixed

In addition, I added the build for the manylinux_2_24_armv7l platform. If the CI can pass smoothly, it will be automatically released to PyPI, and the new version number is 0.2.1.dev6.

update

0.2.1.dev6 released.

[gaiuscosades]

@overcat

I have now been able to integrate everything into my codebase and it runs perfectly with lightning speed!

Only one legacy embedded platform I am running does not find compatible wheels.
This is most likely because of the old kernel version. Would you mind adding the generic "linux" to the build script also?

$ python3 -m pip debug --verbose
pip version: pip 20.0.2 from /usr/lib/python3.8/site-packages/pip (python 3.8)
sys.version: 3.8.12 (default, Aug 30 2021, 16:42:10)
[GCC 9.3.0]
sys.executable: /usr/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /usr/lib/python3.8/site-packages/pip/_vendor/certifi/cacert.pem
Compatible tags: 49
  cp38-cp38-manylinux2014_armv7l
  cp38-cp38-linux_armv7l
  cp38-abi3-manylinux2014_armv7l
  cp38-abi3-linux_armv7l
  cp38-none-manylinux2014_armv7l
  cp38-none-linux_armv7l
  cp37-abi3-manylinux2014_armv7l
  cp37-abi3-linux_armv7l
  cp36-abi3-manylinux2014_armv7l
  cp36-abi3-linux_armv7l
  cp35-abi3-manylinux2014_armv7l
  cp35-abi3-linux_armv7l
  cp34-abi3-manylinux2014_armv7l
  cp34-abi3-linux_armv7l
  cp33-abi3-manylinux2014_armv7l
  cp33-abi3-linux_armv7l
  cp32-abi3-manylinux2014_armv7l
  cp32-abi3-linux_armv7l
  py38-none-manylinux2014_armv7l
  py38-none-linux_armv7l
  py3-none-manylinux2014_armv7l
  py3-none-linux_armv7l
  py37-none-manylinux2014_armv7l
  py37-none-linux_armv7l
  py36-none-manylinux2014_armv7l
  py36-none-linux_armv7l
  py35-none-manylinux2014_armv7l
  py35-none-linux_armv7l
  py34-none-manylinux2014_armv7l
  py34-none-linux_armv7l
  py33-none-manylinux2014_armv7l
  py33-none-linux_armv7l
  py32-none-manylinux2014_armv7l
  py32-none-linux_armv7l
  py31-none-manylinux2014_armv7l
  py31-none-linux_armv7l
  py30-none-manylinux2014_armv7l
  py30-none-linux_armv7l
  cp38-none-any
  py38-none-any
  py3-none-any
  py37-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

[overcat]

Hi @gaiuscosades

Can you try upgrading pip(>= 20.3 required) to the latest version first and try again?

[gaiuscosades]

@overcat

After some troubles I was able to update pip on this platform also, which enabled me to use the wheels you already built!

I am able to run everything on my windows test environment, also in combination with my complete codebase.
Howerver I encountered a very strange error on the embedded platform.

$ python3
Python 3.8.12 (default, Aug 30 2021, 16:42:10)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastcrc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/fastcrc/__init__.py", line 1, in <module>
    from . import crc16, crc32, crc64
  File "/usr/lib/python3.8/site-packages/fastcrc/crc16.py", line 6, in <module>
    from .fastcrc import crc_16_arc as _crc_16_arc
ModuleNotFoundError: No module named 'fastcrc.fastcrc'
>>>

Do you have any idea why line fastcrc/crc16.py:6 is valid code on one platform and not on the other?

[overcat]

@gaiuscosades

I'm at work, and when I get off work, I'll try to create an ARM virtual machine for testing.

[gaiuscosades]

@overcat

Wow thx. ;D
But you don't have to, just interested if you have any ideas why the include Paths suddenly won't work.

[overcat]

Hi @gaiuscosades, I think it may be due to not reading the fastcrc.cpython-38-arm-linux-gnueabihf.so file correctly, can you run the following script and let me know the output?

import sysconfig
sysconfig.get_config_var("EXT_SUFFIX")

Also which wheel from the list are you currently using? fastcrc-0.2.1.dev6-cp38-cp38-manylinux_2_24_armv7l.whl?

[gaiuscosades]

$ python3
Python 3.8.12 (default, Aug 30 2021, 16:42:10)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-38-arm-linux-gnueabi.so'
>>>

Yes fastcrc-0.2.1.dev6-cp38-cp38-manylinux_2_24_armv7l.whl gets selected and installed.

[overcat]

$ python3
Python 3.8.12 (default, Aug 30 2021, 16:42:10)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-38-arm-linux-gnueabi.so'
>>>

Yes fastcrc-0.2.1.dev6-cp38-cp38-manylinux_2_24_armv7l.whl gets selected and installed.

What is included in fastcrc-0.2.1.dev6-cp38-cp38-manylinux_2_24_armv7l.whl is fastcrc.cpython-38-arm-linux-gnueabihf.so, I think it cannot be invoked on your platform (gnueabihf != gnueabi), let me see if there is any solution.

[overcat]

Hi @gaiuscosades

Can you help me test if this wheel works on your platform?

unzip fastcrc-0.2.1.dev6-cp36-abi3-manylinux_2_24_armv7l.whl.zip
pip install fastcrc-0.2.1.dev6-cp36-abi3-manylinux_2_24_armv7l.whl --force-reinstall

fastcrc-0.2.1.dev6-cp36-abi3-manylinux_2_24_armv7l.whl.zip

[gaiuscosades]

Perfect, works with the provided artifact!

$ python3
Python 3.8.12 (default, Aug 30 2021, 16:42:10)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastcrc
>>> b = b''
>>> print(f"{fastcrc.crc32.k_reversed_reciprocal_refin(b, 0x01020304):08x}")
01020304
>>>

[overcat]

@gaiuscosades

Thanks for your feedback! I'm going to release v0.2.1-dev.7 now, and if everything works fine, I'll release v0.2.1 today.

[gaiuscosades]

slight bug: The initial value for the two algorithms that I provided should be 0x00000000
Only my application uses differing initial values but they can be provided on every call!
Other users might be confused by that behavior.

[gaiuscosades]

@overcat

Thank you, I will test the release once it's up on pypi!

[overcat]

slight bug: The initial value for the two algorithms that I provided should be 0x00000000 Only my application uses differing initial values but they can be provided on every call! Other users might be confused by that behavior.

fixed in v0.2.1-dev.8

[gaiuscosades]

@overcat

I tested v0.2.1-dev.8 using both of my setups. Everything I can tell seems fine and working reliably, as of now.

Thank you very much!

[overcat]

v0.2.1 published.