Allow upload of PDFs to SIPI

[gfoo]

Is there any rules for Sipi upload route to accept or not a PDF?

I've an error for example with this file: Programme_Colloque_Bertrand.pdf

Using the binary, the PDF sounds ok:

$ docker run --rm -v $(pwd):/image daschswiss/knora-sipi:v10.1.1 -f "/image/Programme_Colloque_Bertrand.pdf" out-sipi.pdf
pdf has 2 pages

But using the upload route (the call is stuck):

curl -s -X POST -F filename=@Programme_Colloque_Bertrand.pdf http://localhost:1024/upload?token=xxxx

[gfoo]

From the cines web site cited by @mrivoal , this PDF is considered as archivable (see the Validation tab).

I used their tools to try to fix potential errors in the PDF (Correction PDF tab):

• using PDFtk : Programme_Colloque_Bertrand-PDFTk-FIX.pdf
• using Ghostscript : Programme_Colloque_Bertrand-GhostscriptPDFa-FIX.pdf

Both fixed files are now accepted by Sipi.

[gfoo]

we also used Acrobat to convert the file into a PDFa but Sipi does not accept it.

Programme_Colloque_Bertrand-AcrobatPDFa-FIX.pdf

[gfoo]

@lrosenth Any news about that?

1/ is there a test in Sipi to not accept particular PDFs? Because a minima, without any PDF quality considerations I cannot import all the pdfs by using the route http://localhost:1024/upload?token=xxxx

2/ is there a well-known way to convert about 1000 pdfs. In this post I used PDFtk (no so easy to install on linux/mac os systems, but I think it is possible) and Ghostscript does not make the job correctly for all our pdfs.

The Ghostscript solution seems to me more flexible, I used this command line (found somewhere...):
gs -dPDFA -dBATCH -dNOPAUSE -sColorConversionStrategy=UseDeviceIndependentColor -sProcessColorModel=DeviceCMYK -sDEVICE=pdfwrite -dPDFACompatibilityPolicy=2 -sOutputFile=output.pdf input.pdf

Maybe you have some advices about these parameters?

(I'm going to prepare some scripts to test all that and provide to you all our pdfs.)

[subotic]

I think this is the wrong repository for this issue. The docker image of sipi you use contains a completely different set of scripts that are custom to Knora.

The upload script that you are using is this one: https://github.com/dasch-swiss/knora-api/blob/develop/sipi/scripts/upload.lua.

[subotic]

I've moved the issue to the knora-api repo.

[benjamingeer]

I've moved the issue to the knora-api repo.

@subotic Why? I thought this issue was about getting Sipi to validate PDF files.

[subotic]

Isn't it about a problem with uploading PDFs? Since this logic is implemented in a lua script, it could probably be solved in this repo.

[benjamingeer]

Which logic? The Lua script just calls Sipi’s C++ functions to process the file. If Sipi can’t parse a particular PDF file, there’s nothing the Lua script can do about it.

[lrosenth]

OK, I had a look at it. It requires at leastsome major changes in

• knora-api/sipi/scripts/upload.lua
• knora-api/sipi/scripts/store.lua

Currently all upload files are being converted to a sipi image object and then converted to a JPEG2000 file. In order to cope with PDF's (and also .ttl's, .xnml's etc.) we have to build in a switch which checks the MIME type and performs

• the conversion as it used to be if the input MIME-type is of TIFF, PNG,JPEG,J2K etc.
• just copies the file to the proper place (tmp and then the images/project/...)

To rememer, PDF's that are in the images-folder can be served

1. as PDF-download (complete PDF) if the URL looks like this: http://sipi-server/images/file.pdf
2. as image using the full IIIF-url with the addition of the pagenumber http://sipi-server/images/file.pdf@pagenum/full/full/0/default.jpg where pagenum is an interger 0,1,2

[benjamingeer]

Currently all upload files are being converted to a sipi image object and then converted to a JPEG2000 file.

No, upload.lua checks whether the file is an image. If not, it doesn't do any conversion:

https://github.com/dasch-swiss/knora-api/blob/develop/sipi/scripts/upload.lua#L116

[benjamingeer]

Support for uploading PDF files was added in #1206.

[benjamingeer]

These changes were also announced on Discuss DaSCH:

https://discuss.dasch.swiss/t/support-non-image-files-in-knora-api-v2/33/2

[gfoo]

@benjamingeer did you try to reproduce the bug? curl -s -X POST -F filename=@Programme_Colloque_Bertrand.pdf http://localhost:1024/upload?token=xxxx I found this bug on 10 Dec 2019

[benjamingeer]

@gfoo I haven't tried to reproduce it, but if it works with one PDF file and not with another PDF file, I think the problem has to be in the C++ code in Sipi.

[mrivoal]

@lrosenth (or to whom it may concern) any news regarding this issue? We still have a lot of things to fix to be able to release Lumieres.Lausanne but if we can't import PDFs, then we can't release anything.

[benjamingeer]

I've reproduced this and can confirm that it depends on the content of the PDF file. Moved to dasch-swiss/sipi#319.

[gfoo]

@benjamingeer thanks for the test