[Feature Request] Split and merge documents

[ghost]

As a user I would like to merge different scan into one document.

Example: I scan the front and back side of an ID card, it uploads as different documents into paperless. I can merge the 2 documents into one.

[jonaswinkler]

Hi, welcome to GitHub!

As of right now, I don't have any plans to support editing PDF documents. If you really need that, it might be worth giving Papermerge a shot, they do have some editing tools over there. Although I don't know if they support document merging specifically.

This would also be a very big change, since

• I need a new UI to select and reorder documents for merging
• I need need support from the web server to do the actual editing
• Paperless stores exactly one original file for each document and never modifies that. If we want to merge two documents, we modify the original files. This is something I want to avoid.
• What if the user wants to merge an Office .docx and a PDF document? We could simply disable that, but since all documents are displayed as PDF documents in paperless, some users might get confused about why certain documents can be merged, and some cannot.

Not going to happen (anytime soon).

[shamoon]

Agree, feels kinda out of the scope of this app IMHO, and so many tools can do this, even native PDF / image viewers...

[Philmo67]

Just curious : what tools are you using for merging/splitting/rotating pdf documents ?
I tried PDFsam basic, or pdfarranger or even directly after scanning using NAPS2 but I don't find these tools user-friendly enough for these tasks.

[jonaswinkler]

PDFArranger gets the job done, and has everything I need. Apart from that, I usually use gscan2pdf for scanning, and discard unwanted pages with that, or scan multiple pages into a single document. No further editing needed, usually.

[Matthias84]

Some request here. I feed scanned TIFFs from an ADF document scanner. Of course I could do a manual preprocessing outside of paperless, but for non tech users, it might be interesting to get an preprocessing-inbox where you could merge, remove, reorder pages, turn pages, ... before they are fully processed within paperless-ng? But yes, I see the point that it's a lot of work to support it for all the different (multi-page) formats ... 🤔

[Zocker1999NET]

In #426 I had a similar idea (I did not found this already existing issue) about how to implement this:

How could this be implemented on the UI:

1. Select the documents you want:
2. Click on a "Combine" button

What happens in the background:

1. Combine the original documents (not the archived versions!) for example using ImageMagisk: convert "$@" pdf:-
2. Delete all old entries of the selected documents
3. Reprocess the new document as it was simply placed into the consume directory

Known issues with this implementation:

• The original source files maybe cannot be currently handled, so they may be lost. Possible workaround: Before combining the originals to a PDF document, pack them together into a zip/tar archive, store that as "original document" and enable paperless to work with zip/tar archives if possible
• Will most likely not support formats not supported by ImageMagisk like Office documents, however should be able to combine JPEGs/PNGs/PDFs/TIFFs. Possible workaround: Before combining using ImageMagisk them, convert each file not supported by ImageMagisk to a PDF reusing current existing strategies.

The TAR/ZIP approach would allow paperless-ng to prevent losing the old original documents, while, as I see it, allow it "in theory relatively simply" (haven't seen the code of this project yet) to allow this with all existing documents.

[jhass]

I wonder where the need for merging generally comes from.

For me it's because my printer's scan to mail function can't put more than 4-5 pages into a single document. I wonder if it's comparable for most of you. If so, we might not need (much) UI for this or even change much about the one document == one file paradigm. Instead we could allow defining rules for merging at consumption time.

For mail this is "easy", we could have a mail rule to merge all attachments in a mail into one document for example (convert to PDF if needed, sort by name, then merge PDFs).

For manual upload it could be a checkbox or a separate button "upload selected files as one".

API wise that could be a new endpoint, "upload multiple as one", to be integrated into any frontends that want to support this.

The most tricky bit is for the folder drop, since there's no rule system for that yet as far as I noticed. One could imagine something alike the mail filtering system though based on filenames, match all files with pattern, do action on that set of files. The only action at first would be merge of course.

So, to summarize I wonder if "merge at consumption" solves the needs of most people here already.

[Zocker1999NET]

@jhass Your ideas on how to implement this feature seems great.

To answer your question, my printer may has such a feature, but because I'm currently scanning over 600 documents to store even older documents on the computer, I decided not to use such features because I wanted to scan these amounts of pages using a feed scanner without thinking about them in the first place. I wanted to do the sorting/merging of documents only digitally but create an index over my offline documents by sorting them by an incrementing scan id they gain after scanning. I think this is much easier in my case.

[jhass]

Not too different story here, just maybe a little less to go through and I'm also throwing out stuff where I feel fine at retaining only the digital copy, so going by the recommended ASN system :) I'm just running paperless-ng on a server so using the mail function of my feed scanner rather than some scan tool on a PC is easiest. Now my pain is that I have multiple documents for the same ASN 😅 , othewise I'm not too worried about, it's indeed easy to find the "other parts" by date or title.

[jonaswinkler]

Chiming in here and sharing some further ideas and comments. I'm pretty busy right now and don't have all that much time except for critical stuff.

For mail this is "easy", we could have a mail rule to merge all attachments in a mail into one document for example (convert to PDF if needed, sort by name, then merge PDFs).

This would certainly be possible. However, does your scanner actually support sending multiple scanned files in one mail? Also, I'd like to have the merging logic available to all users, not just users who use the mail functionality.

The most tricky bit is for the folder drop, since there's no rule system for that yet as far as I noticed. One could imagine something alike the mail filtering system though based on filenames, match all files with pattern, do action on that set of files. The only action at first would be merge of course.

This is in fact the most tricky part. Once paperless detects new files in the consumption folder, it sends them to the task queue for processing immediately. How does paperless detect when the last document of a batch has arrived? I don't think there's a good solution here.

If we do this, I'd like to have the merging functionality available to everyone, and the consumption folder is still the most commonly used way to upload documents.

---

Therefore, some ideas on what could actually work for everyone, given the current architecture.

• Paperless will always add one file as one document, and one document has exactly one original file (+ optional archived file). That will not change. The implications and required changes are just too big.
• We can add merging support to the UI in a non-intrusive way:
  • On the document detail page, have an additional action in the top right corner "Edit / Add to edit / Merge with others".
  • This would add the document to an "Edit tool / Merge tool", a link to this tool would appear below "Documents" in the sidebar.
  • More documents can be added to that tool in the same way.
  • More documents could also be added by selection on the list (as pictured above)
  • The merge tool would allow reordering and saving the combined document as a new document.
  • The merge tool would also have an option to delete the selected document after merging was successful.
  • The merge tool only appears when documents are selected for merging.
• We need API support for that.
  • A new endpoint "/api/merge" that accepts a "merge plan" on POST, which is essentially the ordered list of documents.
  • We already have python dependencies in paperless that deal with pdf editing.
  • Since paperless also accepts non-PDF documents (images, text, office, ...): Use the original documents if PDF, or the archived versions when original is not a PDF (issue warning). Issue error when no PDF is available.
  • API combines original documents, invokes OCRmyPDF on the merged document again to get the combined archived version, and saves all that as a new document.
  • API optionally deletes all documents in the merge plan.
  • Then there's also metadata. We could either redo metadata detection on the merged document just as we do with new documents, or simply take the metadata from the first document.

After that works, we could also think about adding support for selecting individual pages from the selected documents. (This is something I'd find useful as well, since I've got lots of documents with empty pages that my scanner detected as not empty)

The most critical part is making the backend work, so that should be the focus.

How does that sound? This all is also very isolated functionality and can be added without affecting anything else. If someone wants to take a stab at that, I can give some more detailed instructions on how to do it.

[jhass]

However, does your scanner actually support sending multiple scanned files in one mail? Also, I'd like to have the merging logic available to all users, not just users who use the mail functionality.

No it does not, it doing the opposite, splitting the document into multiple PDFs inside one mail (I think it's one mail, I actually never checked 😅) if it gets too big.

Yes, a merging tool that just creates a new document sounds like a great idea 👍 :)

As a third alternative I think something like a meta-document which groups several documents in a defined order but keeping the members untouched and their own entries could also already help a lot of usecases and might be a little bit less effort.

[jonaswinkler]

and might be a little bit less effort.

We'd still need some UI to define these meta documents, which is about the same as the one for a merge tool. We also need support in the back end for that, documents now have ordered child documents? Also, many components of paperless have to take this new data structure into account (search index should not return documents that are part of a meta document, import+export, metadata matching should use the concatenated content of all documents in a meta document, ...)

Compared to building a feature that uses already existing data structures and abides to already defined contracts (and in doing so is compatible with all existing features), this is actually a lot more work.

And then there's also the test suite. Changing features requires changing associated test cases. Adding a new isolated feature just requires new test cases for that feature.

[jhass]

I didn't mean it to be that invasive, child documents could appear normally still and meta documents could not appear in full text search etc, child documents would "just" provide a quick link to go to the meta document they're part of.

I felt building the merge background task could potentially prove to be quite the rabbit hole 😅

[jonaswinkler]

I didn't mean it to be that invasive, child documents could appear normally still and meta documents could not appear in full text search etc, child documents would "just" provide a quick link to go to the meta document they're part of.

I want to do things properly :)

I felt building the merge background task could potentially prove to be quite the rabbit hole 😅

Actually not, use pikepdf to produce a new pdf document, submit that to the consumer just as we do with other new documents, and optionally delete some documents when done. The consumer will take care of the rest.

The actual merging and editing is very straight forward (https://pikepdf.readthedocs.io/en/latest/topics/pages.html).

[jonaswinkler]

I just put a new API endpoint together, and the actual merging process on the server side is straight forward. Reordering documents, keeping only selected pages, that's all simple. The merged document will appear as a new document to paperless, with notifications and all that.

Now I need to get this implemented properly and figure out how the UI is supposed to work.

[shamoon]

Very cool! Didn’t even realize you were actively working on this. Let me know if / when / where I can help, have some UI ideas

[jonaswinkler]

Well, I don't exactly communicate what I'm actively working on, that's true.

If you want to work out a UI for this, go for it.

I've also got some UI ideas, not sure if they align with yours, here goes.

• Add some button to the document detail page to the top right ("Mark for merge" / "Add to merge" or similar)
• Add a button to the bulk editor that does the same, but with multiple documents
• When documents are selected for merge, A new sidebar link "merge tool" or similar will appear, which in turn opens the merge editor. I feel that when selecting multiple documents for merge, that should open immediately.
• The merge editor has two columns. The left has some sort of list of the selected documents, and changing the order should be possible with drag and drop. Probably some cards with the thumbnail, but not as tall as the large cards.
• The cards may also have some field to select specific pages (maybe a text field that accepts a string such as "1,3-6,8")
• The right column will show a preview. Generating the merged PDF is actually pretty fast, the download will probably take more time than the merge itself. This preview could be reloaded in real time (debounced).
• And well, some buttons to start the process.

These are just ideas. If you got better ideas, go for it, while keeping the following in mind:

• I'd like to keep the UI simple as a whole. Therefore, be sparse with adding buttons that are always visible, even when the tool is not used.
• I can't provide images for individual pages of a document. Just the thumbnail and the PDF itself.

The API will essentially accept an ordered list document of ids, and for each document id an optional page range. I don't have the details down yet. As long as the UI is able to provide that, we're good. It will be possible to specify the same document twice, in case you want to add pages from a document somewhere in the middle of another document.

The API will also have an option to download the resulting document as a preview without actually adding it to paperless.
And some flags to optionally delete source documents on success.

[shamoon]

Yea thats pretty similar to what I imagined. And I agree as this will be a not-every day and probably even not every-user kinda tool the button shouldnt be too prominent / take up too much space, maybe inside a menu or something. And yep exactly what I was thinking about getting there from document detail or bulk edit, and it opens a modal with the UI.

As for the actual UI, definitely agree on visual drag + drop, preview does sound cool too. And then when the user is done do they hit "Save" does it create a new document? And what about metadata? Im sure we'll have to figure out lots of stuff once we dig in. Mobile might be a challenge, etc.

[jonaswinkler]

and it opens a modal with the UI.

Not necessarily a modal, I think this should be a full page view. You may want to go from the merge tool to the list again, and add more documents.

And then when the user is done do they hit "Save" does it create a new document?

It will create a new document. Maybe a checkbox that will cause the source files to be removed on success.

And what about metadata?

Options for either keeping info from the first document (which should be most representative for the resulting file), or run it through the matching algorithms again.

Mobile might be a challenge, etc.

It's okay to have certain functionality not available on mobile.

[shamoon]

Hmm, just now this makes me think about whether it will be frustrating if the actual merge UI has no way to add documents, like a “picker” of some kind. Like if you added 2 docs but realized you need a third you’d have to go back to the list and find the other, hightlight it and add it. A little odd. Then again maybe people will mostly be merging two docs so it’s no big deal?

Just kinda asking / thinking out loud. I personally haven’t needed this so I’m trying to put myself in the mindset of a user of this

[jonaswinkler]

Hmm, just now this makes me think about whether it will be frustrating if the actual merge UI has no way to add documents, like a “picker” of some kind.

The picker would essentially be a list view (with filtering) as well, or something similar, and we already have that.

[shamoon]

The picker would essentially be a list view (with filtering) as well, or something similar, and we already have that.

Yea.

Should be fun challenge, LMK when I can start playing with it.

[ffleischer]

While you're currently working on this topic, would it make sense to directly consider some kind of Staple functionality?

UseCase:
I've got a small mobile document scanner (Doxie GO) which can scan multiple pages but only one-sided. So I need to scan the other sides separately. Their Windows companion app has this Staple functionality.

It generates from 2 documents (D1 & D2) with multiple pages (D1P1, D1P2, .... & D2P1, D2P2, ....) a new document with (D1P1, D2P1, D1P2, D2P2,.....). This mitigates the single side scan a bit.

[jonaswinkler]

• What browser are you using btw?

Chromium.

• When the separator is the first or last item it should remove it. I cant reproduce the bug youre seeing where its sending [], are there specific steps to recreate this? Thats what lines 31-35 in split-merge.service.ts should prevent

I manually dragged a separator to the end after splitting.

• Ill see about document chooser using the view, its pretty basic right now

I think it's okay that way.

[shamoon]

Ok these small things should be addressed, I'll tackle the rest when I have some more time later and reply directly to your code review. Thanks

[rklasen]

Any updates on this? This seems like a fantastic feature.

[jonaswinkler]

I just need to find some time to address a few remaining issues.

[steffe]

First of all a big thank you, amazing Software, was searching since longer a solution for a paperless easy to use tool, now I found it.
Awaiting for this Features, especially the Split feature, if you need some testers, let me know. Best Stefan

[GoldenBatt88]

New to Paperless, just saw this project last week, and I'm very happy with this piece of software, it's great! Just picked up a cheap scanner today, but i also ran into problems that it would make 2 pdf's is it is a double sided letter, no option to merge from te scanner itself... Glad to see there is already a solution being worked on. If you need more people to test, I'm also willing to test some documents here.

[mhupfauer]

Is there an approximate ETA on the availability of this feature on the stable main branch?

[nikor30]

Hi, yes all so here highly wanted :-)

[olvier]

+1 :)

[eSportler89]

+1 :)

[rutgr]

+1 :)

[cardinalfan1]

I’d also like to see this functionality. Especially the ability to merge a pdf with another one in reverse-alternate order. (For those of us who have non-duplex scanners who end up with alternate order with the second pdf being reversed). Right now doing it with pdfsam but would be great to have this built in.

[MBadberg]

Hm, lol. the first function, what i've missed after uploading a few documents - merging :-D

So, here i will give a +1

[lucasmenno]

+1 also from my end :)
Got a WiFi multifunction device. Scanning with bash directly to the input folder.
Merging documents inside paperless-ng would be awesome :3

[buttercheetah]

+1 In my opinion, its the only thing missing from this program.

[Duval23]

I do not urgently need this feature, but I can give a use case case where it comes in handy.

I need to create an account of charges. I used to directly scan to paperless and have not prepared any other 'scan-option'. As new bills occur regularly, they could be scanned to paperless-ng and then appended to the existing document containing all the bills. So this feature helps to keep 'work in progess documents' up to date.

For now, I scan all the bills to paperless and once the document can be finished, I download all bills and merge them locally. Not a big deal, to be honest. But maybe this use case will help for the documentation.

[henfri]

For some of you, this may help:
#457 (comment)
Not anywhere close to the .mov above, of course. But if your scanner is a Brother ADS it should do the trick.

[Jaykob]

First of all, thanks for this great piece of software!

I also miss this feature after using paperless for one hour, as my scanner doesn’t easily support scanning multiple pages to a network share. Had to use a RPi with saned and scanbd to listen for the scanner’s hardware buttons to be pressed. So I’m quite happy that it works after all.

Seeing that it’s been 9 months or so without progress, I’m asking myself where and how we could help to get this thing finished?
I don’t want to sound offensive - just trying to offer help for a feature that for sure many people (including me) would suspect to be available in a project like this.

[henfri]

See here:
#1599

[Jaykob]

Thanks for the heads up!
Hope Jonas is OK 🙏

[maugsburger]

May I leave one more idea: Use OCR and a special page (with a unique text, like, PAPERLESS-SPLIT-PAGE-PAPERLESS-SPLIT-PAGE on it), to automatically split a scan there (and remove the seperator page).

My scanner is really slow on loading the scan preset, but quite fast on the actual scanning. Therefore, I would love to add a bunch of documents at once, but get them split automatically.

[smseidl]

May I leave one more idea: Use OCR and a special page (with a unique text, like, PAPERLESS-SPLIT-PAGE-PAPERLESS-SPLIT-PAGE on it), to automatically split a scan there (and remove the seperator page).

My scanner is really slow on loading the scan preset, but quite fast on the actual scanning. Therefore, I would love to add a bunch of documents at once, but get them split automatically.

@maugsburger - Paperless-NGX just released Version 1.7.0 that include the ability to use a PATCHT file for separating documents into different files. Its disabled by default, but if you read thru the documentation, it's quite easy to turn on.

[clauschristianude]

Hi,

for me it's the opposite: I scan over the year bill's into single documents and in the new year, I would like to select all bill's of the last year and merge them into one big file. So I can handle them better.

c-c