new release when & ship with libvips? #443
Comments
|
Hi @SteveHawk, pyvips works with any libvips version -- it introspects the libvips binary and exposes the API it finds. pyvips v2.2.1 should work fine with 8.15 and 8.16. Yes, we've talked about including a binary, and you're right it'd be pretty simple. Someone just needs to do the work :( pyvips still uses the prehistoric One slight issue is that pyvips supports ABI (portable but slow and leaky) and API (fast and stable) modes, but with a prebuilt libvips binary we'd be stuck in ABI mode. Maybe the benefits are worth it.
You should just need the |
Oh that's weird. I asked because I tried to use pyvips v2.2.1 with 8.15, but didn't work and throw errors immediately (seems to be related to glib). The master branch works though, that's why I thought v2.2.1 is not compatible with 8.15. Here's how I did it:
After setting up the environment, enter python shell and run: >>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from pyvips import Image
DEBUG:pyvips:Binary module load failed: /usr/local/lib/python3.11/site-packages/_libvips.abi3.so: undefined symbol: vips_path_mode7
DEBUG:pyvips:Falling back to ABI mode
DEBUG:pyvips:Loaded lib <cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x7fa8935a3550>
DEBUG:pyvips:Loaded lib <cffi.api._make_ffi_library.<locals>.FFILibrary object at 0x7fa8936f3710>
DEBUG:pyvips:Inited libvips
>>> Image.new_from_file("/root/lib/test.png")
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = VipsForeignLoadPngFile
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = filename, value = /root/lib/test.png
(process:875): GLib-CRITICAL **: 09:49:38.370: g_datalist_id_set_data_full: assertion 'key_id > 0' failed
(process:875): GLib-GObject-CRITICAL **: 09:49:38.370: g_param_spec_pool_lookup: assertion 'pool != NULL' failed
(process:875): GLib-GObject-WARNING **: 09:49:38.370: g_object_set_is_valid_property: object class '(null)' has no property named 'filename'
Segmentation fault (core dumped)That crashed. I also tried >>> Image.thumbnail_buffer(b"test", 128)
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = thumbnail_buffer
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = buffer, value = b'test'
DEBUG:pyvips.error:Error unsupported gtype for set NULL, fundamental GBoxed
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.11/site-packages/pyvips/vimage.py", line 256, in call_function
return pyvips.Operation.call(name, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 282, in call
op.set(name, intro.details[name]['flags'], match_image, value)
File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 216, in set
super(Operation, self).set(name, value)
File "/usr/local/lib/python3.11/site-packages/pyvips/vobject.py", line 122, in set
gv.set(value)
File "/usr/local/lib/python3.11/site-packages/pyvips/gvalue.py", line 242, in set
raise Error('unsupported gtype for set {0}, fundamental {1}'.
pyvips.error.Error: unsupported gtype for set NULL, fundamental GBoxedWhen I uninstall the pypi version and install the master branch, pip show still reports API mode, and everything works as expected: >>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> from pyvips import Image
DEBUG:pyvips:Loaded binary module _libvips
DEBUG:pyvips:Module generated for libvips 8.15
DEBUG:pyvips:Linked to libvips 8.15
DEBUG:pyvips:Inited libvips
>>> Image.new_from_file("/root/lib/test.png")
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = VipsForeignLoadPngFile
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = filename, value = /root/lib/test.png
DEBUG:pyvips.vobject:VipsObject.get: name = out
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.vobject:VipsObject.get: name = width
DEBUG:pyvips.vobject:VipsObject.get: name = height
DEBUG:pyvips.vobject:VipsObject.get: name = format
DEBUG:pyvips.vobject:VipsObject.get: name = bands
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.voperation:VipsOperation.call: result = <pyvips.Image 1240x1754 uchar, 1 bands, b-w>
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
DEBUG:pyvips.vobject:VipsObject.get: name = width
DEBUG:pyvips.vobject:VipsObject.get: name = height
DEBUG:pyvips.vobject:VipsObject.get: name = format
DEBUG:pyvips.vobject:VipsObject.get: name = bands
DEBUG:pyvips.vobject:VipsObject.get: name = interpretation
<pyvips.Image 1240x1754 uchar, 1 bands, b-w>
>>> Image.thumbnail_buffer(b"test", 128)
DEBUG:pyvips.voperation:VipsOperation.call: operation_name = thumbnail_buffer
DEBUG:pyvips.voperation:VipsOperation.call: match_image = None
DEBUG:pyvips.vobject:VipsObject.set: name = buffer, value = b'test'
DEBUG:pyvips.vobject:VipsObject.set: name = width, value = 128
INFO:pyvips:VIPS: thumbnailing 4 bytes of data
DEBUG:pyvips.error:Error unable to call thumbnail_buffer VipsForeignLoad: buffer is not in a known format
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.11/site-packages/pyvips/vimage.py", line 256, in call_function
return pyvips.Operation.call(name, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/pyvips/voperation.py", line 305, in call
raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call thumbnail_buffer
VipsForeignLoad: buffer is not in a known format
I installed pyvips with the prebuilt libvips-dev from kleisauke/libvips-packaging (with proper LD_LIBRARY_PATH and PKG_CONFIG_PATH set) and
Actually the package manager will download a LOT of dependencies (over 100 using conda, over 200! using apt, including libjpg, libpng etc.), taking up more disk spaces than statically linked libvips-dev package, and also taking more time to download and install. More importantly, package managers like apt usually don't come with newer versions of these libraries, so that would be another down side. |
|
Huh strange, it ought to work. I'll try to make a dockerfile for this.
Did you use |
Ah if you use the libvips-dev precompiled binary it'd work, yes. But that's probably no smaller than just installing |
|
I updated this dockerfile: https://github.com/jcupitt/docker-builds/blob/master/pyvips-alpine/Dockerfile And it seems to work fine. That's libvips 8.15.1 and pyvips 2.2.1 on alpine. I've not tried with those precompiled libvips binaries. If you need a newer libvips than is in your package manager, I'd expect building from source yourself to be much more likely to work, should be smaller (it can use the platform libjpeg etc.) and can be deployed pretty easily with docker or whatever. |
Yes i did, in a Without the
The precompiled libvips-dev package is a 8MB tarball and only 26MB uncompressed on disk. I believe it only includes necessary .so files, so it's actually much smaller in size ;) |
|
The prebuilt binaries available at https://github.com/kleisauke/libvips-packaging would only work with pyvips' API mode since GLib is statically-linked. Here's a Dockerfile to reproduce this issue: Details# Build and run with:
# docker build -t pyvips-libvips-packaging .
# docker run -it --rm pyvips-libvips-packaging
FROM python:3.12-slim-bookworm
LABEL maintainer="Kleis Auke Wolthuizen <[email protected]>"
ARG VIPS_VERSION=8.15.0
# https://gist.github.com/kleisauke/91ef7315adbe1ca7455a38b24d28081e
ARG VIPS_PC_FILE=https://gist.github.com/kleisauke/91ef7315adbe1ca7455a38b24d28081e/raw/8b780945bacb66d7e79efa1ecc1c9a783b2d1f47/vips.pc
WORKDIR /opt/libvips
# Update packages
RUN apt-get update -q \
# Install needed dependencies
&& apt-get install -qy --no-install-recommends \
build-essential \
ca-certificates \
curl \
pkg-config \
# Download pre-built libvips binaries
&& curl -Ls https://github.com/kleisauke/libvips-packaging/releases/download/v$VIPS_VERSION/libvips-$VIPS_VERSION-linux-x64.tar.gz | tar -xzC . \
# Download pkgconfig file
&& mkdir -p lib/pkgconfig \
&& cd lib/pkgconfig \
&& curl -LO $VIPS_PC_FILE
WORKDIR /app
# Update the PKG_CONFIG_PATH environment variable,
# since libvips is installed in a non-standard prefix
ENV PKG_CONFIG_PATH=/opt/libvips/lib/pkgconfig \
# Ensure dynamic linker finds the pre-built libvips binaries
LD_LIBRARY_PATH=/opt/libvips/lib
RUN pip install --verbose --user pyvips \
&& pip show pyvips
CMD ["python"]The salient part of the error log is: So, it tries to fallback to ABI mode since the deprecated |
|
Ah that had popped out of my brain, thanks Kleis. Then I agree, a pyvips 2.2.2 would be useful. I'll do it now. |
|
We might have v2.2.2, any testing very welcome. |
|
Great! I just opened PR #444 to support this in ABI mode. FWIW, looks like v2.2.2 was miss-tagged as v2.2.0: |
I'll have a look at this somewhere tomorrow. |
🤦 fixed. |
Nice! It works well on my codes so far, will do more tests next week. Thanks! |
PR #445 should fix this. |
Hi, I'm just wondering, is there going to be a new release of pyvips soon?
libvips 8.15 introduces highway for simd which is exciting, however current pyvips v2.2.1 on PyPI only supports libvips up to 8.13. Master branch of pyvips does work with 8.15 in my setup, but it would be really nice to have a stable version to lock in.
Oh and, will it be possible for pyvips to ship with prebuilt libvips in the future? As my understanding, net-vips and sharp has been doing this for years, I tried the prebuilt package from kleisauke/libvips-packaging with pyvips and it works like a charm. Using pyvips without needing to install many dozens of system dependencies is a lifesaver in deployment.
Thanks!
The text was updated successfully, but these errors were encountered: