Permission error with NFS volumes in docker

[FleischKarussel]

Hello everyone,

I'm fiddling around this issue for quite some time and don't have any clue why this happens only with paperlessng as I am running a bunch of services in containers side-by-side in the same manner completely fine.
Please see below, it seems like the permissions seem fine when container starts for the first time when initiating the volume but fails afterwards.
I even tried 777 for data and media folder on the nfs server.
Using local volumes works fine.

Is this an issue on my side or a bug in the directory hopping (/../ ?)

Best regards
FK

docker logs:

Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian buster-updates InRelease
Hit:3 http://security.debian.org/debian-security buster/updates InRelease
Reading package lists...
package tesseract-ocr-deu already installed!
creating directory ../data/index
creating directory ../media/documents
creating directory ../media/documents/originals
creating directory ../media/documents/thumbnails
Waiting for PostgreSQL to start...
SystemCheckError: System check identified some issues:

ERRORS:
?: PAPERLESS_DATA_DIR is not writeable
        HINT: Set the permissions of /usr/src/paperless/src/../data to be writeable by the user running the Paperless services
?: PAPERLESS_MEDIA_ROOT is not writeable
        HINT: Set the permissions of /usr/src/paperless/src/../media to be writeable by the user running the Paperless services

Filesystem permission in container:

docker exec -it paperlessng_webserver ls -ln /usr/src/paperless/
total 32
drwxr-xr-x 2 1000 1000 4096 Jan 15 12:27 consume
drwxrwxrwx 1 1000 1000   38 Jan 15 12:34 data
drwxr-xr-x 2 1000 1000 4096 Jan 15 12:27 export
-rw-r--r-- 1 1000 1000 1241 Dec  9 16:52 gunicorn.conf.py
drwxrwxrwx 1 1000 1000   18 Jan 15 12:34 media
-rw-r--r-- 1 1000 1000 2252 Dec  9 16:51 requirements.txt
drwxr-xr-x 1 1000 1000 4096 Dec 22 14:56 src
drwxr-xr-x 1 1000 1000 4096 Dec 22 14:56 static

Filesystem permissions on NFS server:

ls -l paperlessng-data/* paperlessng-media/*
-rw-r--r-- 1 1000 1000  0 Jan 15 13:37 paperlessng-data/migration_lock

paperlessng-data/index:
total 0

paperlessng-media/documents:
total 0
drwxr-xr-x 1 1000 1000 0 Jan 15 13:34 originals
drwxr-xr-x 1 1000 1000 0 Jan 15 13:34 thumbnails

ls -ld paperlessng-data/ paperlessng-media/
drwxrwxrwx 1 1000 1000 38 Jan 15 13:34 paperlessng-data/
drwxrwxrwx 1 1000 1000 18 Jan 15 13:34 paperlessng-media/

[jonaswinkler]

Please try to write to these directories from within the paperless container as the paperless user. If this works, this is an issue with how paperless tries to determine whether or not it can write to a given directory. If not, see below.

Paperless does the volume initialization as root, therefore, it works. Paperless also changes ownership of these files to the paperless system user, so that paperless can write there.

• Check the UID of the paperless user, it seems you're using MAP_UID and MAP_GID as far as i can tell. The ls output show ids instead of usernames, so I figure the docker container did not manage to change ownership there. This would explain the errors (paperless running as a user with an UID different from 1000) Didn't see the -n flag, that explains this.

[FleischKarussel]

Hey @jonaswinkler,

thanks for your reply!
In fact, I am running with UID and GID mapping to 1000:1000.
I manually created the directory that is being mounted/mapped to the volume on the docker host with 1000:1000, just to be sure.
Do the mount options matter in that regard?

The test you described looked like this, and imho worked as expected. I hope I got it right.

docker exec -u 1000 -it paperlessng_webserver touch /usr/src/paperless/data/test.datei

docker exec -it paperlessng_webserver ls -ln /usr/src/paperless/data
total 0
drwxr-xr-x 1 1000 1000 0 Jan 15 12:34 index
-rw-r--r-- 1 1000 1000 0 Jan 15 16:23 migration_lock
-rw-r--r-- 1 1000 1000 0 Jan 15 16:20 test.datei

docker exec -it paperlessng_webserver ls -l /usr/src/paperless/data
total 0
drwxr-xr-x 1 paperless paperless 0 Jan 15 12:34 index
-rw-r--r-- 1 paperless paperless 0 Jan 15 16:23 migration_lock
-rw-r--r-- 1 paperless paperless 0 Jan 15 16:20 test.datei

docker exec -it paperlessng_webserver df -h /usr/src/paperless/data
Filesystem                                         Size  Used Avail Use% Mounted on
:/volume2/container-data/develop/paperlessng-data  855G   93G  762G  11% /usr/src/paperless/data

[jonaswinkler]

In fact, I am running with UID and GID mapping to 1000:1000.

That's the default. No need to configure that, it will change nothing.

The test you described looked like this, and imho worked as expected. I hope I got it right.

That means that the python process in paperless is either not able to write there, or paperless fails to properly identify whether or not it's able to write there. Please do the following tests.

Bring up a python shell inside the container:

$ docker exec -u 1000 -it paperlessng_webserver /usr/local/bin/python3
Python 3.7.9 (default, Oct 13 2020, 21:10:49) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Do the following commands and post the output:

>>> import os
>>> os.access('../data', os.W_OK | os.X_OK)
[Output I'm interested in]
>>> os.stat('../data')
[Output I'm interested in]

Enter exit() or hit CTRL+D to exit.

[FleischKarussel]

Hello again,

thanks for pointing out the UID/GID Mapping settings.
It took me some time to figure out how to prevent the container from restarting. ;)

>>> import os
>>> os.access('../data', os.W_OK | os.X_OK)
True
>>> os.stat('../data')
os.stat_result(st_mode=16895, st_ino=2090429, st_dev=1048660, st_nlink=1, st_uid=1000, st_gid=1000, st_size=58, st_atime=1610739618, st_mtime=1610727603, st_ctime=1610739618)

[jonaswinkler]

I'm puzzled.

os.access('../data', os.W_OK | os.X_OK)

That's exactly the same check that paperless does to see if it can write to that directory. And when executed manually, it aparrently works, but when paperless does that, it fails and yields the error messages you're experiencing.

paperless-ng/src/paperless/checks.py

Line 24 in d8637ff

elif not os.access(directory, os.W_OK | os.X_OK):

[zafffff]

As @FleischKarussel did, I replaced the entrypoint with /usr/bin/sleep 600 and did the tests in python.
I also tested os.W_OK and os.X_OK separately and found os.W_OK returned false but os.X_OK was true.

Replacing
elif not os.access(directory, os.W_OK | os.X_OK):
with
elif not os.access(directory, os.W_OK & os.X_OK):
has given me a container that seems to work.

[jonaswinkler]

os.W_OK & os.X_OK

Replacing bitwise OR with bitwise AND on two values with no shared bits will essentially check for nothing, always return true, and is equal to not checking anything.

I'll make that a warning instead, so that paperless is at least able to start.

[zafffff]

Using that container for more testing with a python shell. I have found that all files report writable:
`>>> os.getcwd()
'/usr/src/paperless/data'

os.access('db.sqlite3',os.W_OK)
True
os.access('migration_lock',os.W_OK)
True
os.access('index/MAIN_WRITELOCK',os.W_OK)
Truebut all directories report not:>>> os.access('.',os.W_OK)
False
os.access('index',os.W_OK)
False
os.access('../media',os.W_OK)
False`
I do not know if this will help anyone find the solution, but at least there is more information.

[G-regL]

I'm having a similar issue as everyone here.

Trying to map an NFS share to the consume directory leads to the container failing to start.

When I enter the container as paperless:paperless (1000:1000), I can create files with touch in the mounted directory, and see the files and modes are fine, but the python test fails.

As with @zafffff's comment above, W_OK returns False, while X_OK returns True:

paperless@9a5b9ed3cd16:~$ whoami
paperless
paperless@9a5b9ed3cd16:~$ id
uid=1000(paperless) gid=1000(paperless) groups=1000(paperless)
paperless@9a5b9ed3cd16:~$ cd /paperless_consume_iso
paperless@9a5b9ed3cd16:/paperless_consume_iso$ ls
blah  blah2  blah3
paperless@9a5b9ed3cd16:/paperless_consume_iso$ touch foobar
paperless@9a5b9ed3cd16:/paperless_consume_iso$ mkdir baz
paperless@9a5b9ed3cd16:/paperless_consume_iso$ ls
baz  blah  blah2  blah3  foobar
paperless@9a5b9ed3cd16:/paperless_consume_iso$ ls -l
total 4
drwxr-xr-x. 2 paperless paperless 4096 Feb  5 02:00 baz
-rw-r--r--. 1 paperless paperless    0 Feb  5 02:00 foobar
paperless@9a5b9ed3cd16:/paperless_consume_iso$ ls -ln
total 4
drwxr-xr-x. 2 1000 1000 4096 Feb  5 02:00 baz
-rw-r--r--. 1 1000 1000    0 Feb  5 02:00 foobar
paperless@9a5b9ed3cd16:/paperless_consume_iso$ python3
Python 3.7.9 (default, Jan 12 2021, 17:38:33) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.access('.', os.W_OK)
False
>>> os.access('.', os.X_OK)
True
os.stat_result(st_mode=16892, st_ino=90178666, st_dev=1048583, st_nlink=3, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1612490416, st_mtime=1612490414, st_ctime=1612490414)

[FleischKarussel]

I replaced the entrypoint with /usr/bin/sleep 600 to have time for the tests.
May I broke the tests by doing this?

[oliver-la]

I can confirm this happens with NFS shares. (why is NFS always the culprit if a container chokes on a volume?)

Manually setting PAPERLESS_CONSUMPTION_DIR to /consume does not make a difference. So the paperless permission check shouldn't be having issues with the ../ in the path.

What strikes me odd is that my installation only had issues with the consumption directory:

webserver_1  | Paperless-ng docker container starting...
webserver_1  | Waiting for PostgreSQL to start...
webserver_1  | Apply database migrations...
webserver_1  | SystemCheckError: System check identified some issues:
webserver_1  |
webserver_1  | ERRORS:
webserver_1  | ?: PAPERLESS_CONSUMPTION_DIR is not writeable
webserver_1  |  HINT: Set the permissions of /usr/src/paperless/src/../consume to be writeable by the user running the Paperless services

PAPERLESS_DATA_DIR and PAPERLESS_MEDIA_ROOT appear to be perfectly writable.

In my case, I have a webdav container to add files to the consume directory.

  webdav:
    image: xama/nginx-webdav
    volumes:
      - ./consume:/var/webdav/public
    environment:
      - "WEBDAV_USERNAME=webdav"
      - "WEBDAV_PASSWORD=webdav"
      - "TZ=Europe/Zurich"
    restart: unless-stopped
    expose:
      - 80

Maybe the error only appears if two containers share the same directory and perhaps put a lock on it?

As a workaround, I switched to a named volume. (default driver)
This made the error disappear.

This issue started to appear when I updated my container from 0.9.14 to latest. Reverting to 0.9.14 did not fix the issue.

I hope my findings are of any help.

Edit; Here's my fstab. Maybe nolock is causing issues?

192.168.0.165:/volume1/docker /volume1/docker nfs _netdev,nolock,nfsvers=3,proto=udp 0 0

I remember that my Plex installation also had loooots of issues with the nfs share because locks do not work properly with NFS. (even with the option removed)

[jonaswinkler]

Checked for related changes between 0.9.14 and 1.0.0 and there's nothing.

PAPERLESS_DATA_DIR and PAPERLESS_MEDIA_ROOT appear to be perfectly writable.

If these are not stored on NFS volumes, that would explain that.

[oliver-la]

If these are not stored on NFS volumes, that would explain that.

They are, but I think this is an issue with NFS & Docker in general. So you're right to close the issue. I mean yeah, you could probably work around this with some python trick or an ENV to disallow permission checking at startup, but that's out of the project's scope IMHO.

[jonaswinkler]

I'd rather like to know whether the test in #458 (comment) works, and when that exact same test is executed by paperless, as the same user, in the same working directory, on the same consumption directory, it suddenly does not work anymore.

[FleischKarussel]

If these are not stored on NFS volumes, that would explain that.

They are, but I think this is an issue with NFS & Docker in general. So you're right to close the issue. I mean yeah, you could probably work around this with some python trick or an ENV to disallow permission checking at startup, but that's out of the project's scope IMHO.

yes that would be nice and as @jonaswinkler commented in #458 (reply in thread) as option to override/unforce the check I would see the risk on my side to have a proper backup if I feel the need to do that. ;)

[FleischKarussel]

@oliver-la
I see you maybe also use a Synology NAS (Xpenology in my case)?
Just guessing because of the path /volume1/...

Just checked, I also have nolock enabled.
Tried without nolock, another error ;)

Get:1 http://deb.debian.org/debian buster InRelease [121 kB]
Get:2 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://security.debian.org/debian-security buster/updates/main amd64 Packages [270 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [7860 B]
Fetched 8424 kB in 2s (3732 kB/s)
Reading package lists...
package tesseract-ocr-deu already installed!
Waiting for PostgreSQL to start...
flock: 200: No locks available

[oliver-la]

Hello fellow Xpenology user. ;)

We could blame it on Xpenology, but apparently NFS doesn't even support file locking. (source: https://docstore.mik.ua/orelly/networking_2ndEd/nfs/ch11_02.htm)

Maybe try forcing the NFS version to 4? Apparently v4 supports file locking. Depending if Synology implemented NFS4 correctly. (and remove nolock option of course)

I've decided to use volumes for now.

[jonaswinkler]

Just looked into the configuration files again; that lock was initially put there in OG paperless, since paperless used to start two different containers for the webserver and the consumer; and these two containers both tried to run the migrations, at the same time. (which is bad without a lock)

With -ng, its now all in one container and I could probably remove this and its okay in 99.9% of all cases. However, this still protects against concurrent migrations should you both start paperless and execute a management command simultaneously (i don't know, mail_fetcher, just about any will do) after an update that contains database migrations, it may still wreck the migration history (=you'll have a bad time).

I'd rather leave that lock in the script.

[FleischKarussel]

I havn't looked into this option and kind of lacking the knowledge if NFSv4 has any other advantages and surprisingly I may not remember having any other problems with other nfs mount options and other services so far.
Even Jellyfin (because Plex has been mentioned) runs fine so far.
Anyway, there seems to be a need to checkout NFSv4 now! :D

[FleischKarussel]

@oliver-la
https://www.kernel.org/doc/html/latest/filesystems/nfs/nfs41-server.html
Apparently, LOCKING in any form, is not implemented in NFSv4(.1) in the Linux kernel at all.
Even though, I don't know if vendors are doing their own NFS implementions.

[zafffff]

@FleischKarussel , @oliver-la And anyone else having issues with this.
Try setting UID and GID to those of a user on the device containing the nfs share instead of the system running paperless-ng.
This has worked for me on a Synology nas unit.

[jonaswinkler]

I've converted the errors about permissions to warnings in the most recent version. Paperless will now start despite the error. This might help people who experienced issues with data / media folders stored on NFS shares.

[FleischKarussel]

Great, thanks!
I'm going to check this out asap! ;)

[FleischKarussel]

I can confirm paperlessng now starts fine with NFS as volume backend!
Thank you very much! :)

[EdgarVerona]

Have you tried importing a document using the consumer folder with this setup? It sounds like my setup is very similar to yours (attempting to use NFS shares, as the share is on my NAS where things get backed up regularly and it's got redundant disks via ZFS), and I got paperless-ng to start up correctly but when I try to save a document I get hit with "Cannot acquire lock after 10 attempts" and the saving fails as a result.

If you got past that file lock error and are still using NFS, I'd be really interested to hear more about how you did it! So far I've had no luck.

[tinfever]

I might have had a similar issue but I was using a CIFS volume.

volumes:
  data:
  media:
  pgdata:
  export:
    driver: local
    driver_opts:
        type: cifs
        device: //10.0.0.3/Scanned_Documents/export
        o: username=paperless-ng,password=redacted,rw
  consume:
    driver: local
    driver_opts:
        type: cifs
        device: //10.0.0.3/Scanned_Documents/consume
        o: username=paperless-ng,password=redacted,rw

docker-compose up -d would appear to work fine but running docker-compose run --rm webserver createsuperuser would just return the following without ever prompting for any input:

user@ubuntu:~/paperless-ng$ docker-compose run --rm webserver createsuperuser
Starting paperless_tika_1      ... done
Starting paperless_db_1        ... done
Starting paperless_gotenberg_1 ... done
Starting paperless_broker_1    ... done
Paperless-ng docker container starting...
Creating directory /tmp/paperless
Adjusting permissions of paperless files. This may take a while.
Waiting for PostgreSQL to start...
Apply database migrations...
SystemCheckError: System check identified some issues:

ERRORS:
?: PAPERLESS_CONSUMPTION_DIR is not writeable
        HINT: Set the permissions of
drwxr-xr-x /usr/src/paperless/src/../consume
 to be writeable by the user running the Paperless services
Executing management command createsuperuser
SystemCheckError: System check identified some issues:

ERRORS:
?: PAPERLESS_CONSUMPTION_DIR is not writeable
        HINT: Set the permissions of
drwxr-xr-x /usr/src/paperless/src/../consume
 to be writeable by the user running the Paperless services

When opening a shell into the container, I could create files in the export and consume dirs just fine as the root user.
Permissions inside the container would appear like this:

root@d6f22262038b:/usr/src/paperless# ls -l
total 32
drwxr-xr-x 2 paperless paperless 4096 Aug  9 00:03 __pycache__
drwxr-xr-x 2 root      root         0 Aug  8 23:59 consume
drwxr-xr-x 4 paperless paperless 4096 Aug  9 00:03 data
drwxr-xr-x 2 root      root         0 Aug  8 23:59 export
-rw-r--r-- 1 paperless paperless 1109 Jun 13 19:39 gunicorn.conf.py
drwxr-xr-x 3 paperless paperless 4096 Aug  9 00:03 media
-rw-r--r-- 1 paperless paperless 3312 Jun 13 19:39 requirements.txt
drwxr-xr-x 1 paperless paperless 4096 Jun 13 19:44 src
drwxr-xr-x 7 paperless paperless 4096 Jun 13 19:44 static

I'm sure there's a logical reason but there isn't actually a paperless user inside the container oddly:

root@9fa8faaa5ead:/usr/src/paperless/src# id -a
uid=0(root) gid=0(root) groups=0(root)

Attempting to chown paperless:paperless consume would appear successful but not make any difference. Apparently this is normal for docker with cifs volumes from what I've read.

After starting up docker-compose, I ran cat /proc/mounts | grep cifs to find how docker was mounting them and saw the following:

user@ubuntu:~/paperless-ng$ sudo cat /proc/mounts | grep cifs
//10.0.0.3/Scanned_Documents/export /var/lib/docker/volumes/paperless_export/_data cifs rw,relatime,vers=3.1.1,cache=strict,username=paperless-ng,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.0.3,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1 0 0
//10.0.0.3/Scanned_Documents/consume /var/lib/docker/volumes/paperless_consume/_data cifs rw,relatime,vers=3.1.1,cache=strict,username=paperless-ng,uid=0,noforceuid,gid=0,noforcegid,addr=10.0.0.3,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1 0 0

The section file_mode=0755,dir_mode=0755 caught my eye so I changed the docker-compose volumes mount options to o: username=paperless-ng,password=redacted,rw,file_mode=0777,dir_mode=0777 and it appears to have fixed the issue.

More testing is required but I thought this might help someone else.

[scoates]

For Synology DSM users: FWIW, I seem to have fixed this (thanks to this post on Reddit) by giving my user (the one running docker-compose up) read + write access to the consume directory in File Station.

[anlicata]

Hi scoates,
I didn't get it. Because I'm a newbie.
Can you explore the steps you did to get it running?
I'm trying since hours but it doesn't work.
Got a Synology with Docker and DSM 7.1 running

Thank you