Unable to Save Database Dump in Docker Compose #1527

WooMai · 2023-07-08T11:49:16Z

WooMai
Jul 8, 2023

Hello Community,

I am facing an issue while using Dragonfly DB in Docker Compose.
I am unable to save a database dump, and the error message I receive is: Failed to perform snapshot Permission denied: Couldn't open file for writing (is direct I/O supported by the file system?)

Here are some details regarding my setup:

Use docker exec to get inside the container and echo "test" > /data/test.txt works as expected
I have attempted to save a database dump using the SAVE command in redis-cli, but also results in the mentioned error.

Here is my docker-compose.yaml

version: '3'
services:
  app:
    build:
      context: ./app
      dockerfile: Dockerfile
    restart: always
    depends_on:
      - dragonfly
    networks:
      - appnet
    environment:
      - FOO=BAR

  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /root/myapp/data/dragonfly:/data

networks:
  appnet:

I would appreciate any guidance or suggestions on how to resolve this issue. Thank you in advance for your help.

Answered by romange

Sep 5, 2023

Fixed by 1.9.0

View full answer

koraykoska · 2023-08-25T08:22:58Z

koraykoska
Aug 25, 2023

Same here. @romange @dranikpg

0 replies

romange · 2023-08-25T08:39:23Z

romange
Aug 25, 2023
Maintainer

this works for me

version: '3'
services:
  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
    ports:
      - "6379:6379"      
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /tmp:/data

networks:
  appnet:

The problem is probably with the host folder that you map inside. Specifically, /root/myapp/data/dragonfly is located on a filesystem that is probably special in some way.

1 reply

WooMai Aug 25, 2023
Author

I don't think it's a host folder issue as I tried before:

Here are some details regarding my setup:

Use docker exec to get inside the container and echo "test" > /data/test.txt works as expected

I have attempted to save a database dump using the SAVE command in redis-cli, but also results in the mentioned error.

koraykoska · 2023-08-25T08:50:44Z

koraykoska
Aug 25, 2023

In my case it’s just the normal Linux filesystem. Ext4 with lvm.

…

On Fri, 25 Aug 2023 at 10:39, Roman Gershman ***@***.***> wrote: this works for me version: '3'services: dragonfly: image: "docker.dragonflydb.io/dragonflydb/dragonfly" command: - "--save_schedule=*:*" ports: - "6379:6379" ulimits: memlock: -1 restart: always networks: - appnet volumes: - /tmp:/data networks: appnet: The problem is probably with the host folder that you map inside. Specifically, /root/myapp/data/dragonfly is located on a filesystem that is probably special in some way. — Reply to this email directly, view it on GitHub <#1527 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACWUTTKNIFPE766HFSNMLJDXXBQEPANCNFSM6AAAAAA2CZSQDE> . You are receiving this because you commented.Message ID: ***@***.***>

2 replies

romange Aug 25, 2023
Maintainer

Does docker compose I provided work for you? can you run redis-cli SAVE ?

koraykoska Aug 25, 2023

@romange Same error with redis-cli SAVE and your docker compose.

WooMai · 2023-08-25T10:34:16Z

WooMai
Aug 25, 2023
Author

this works for me

version: '3'
services:
  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
    ports:
      - "6379:6379"      
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /tmp:/data

networks:
  appnet:

The problem is probably with the host folder that you map inside. Specifically, /root/myapp/data/dragonfly is located on a filesystem that is probably special in some way.

I just created a brand new virtual server on DigitalOcean with Debian 12, and here is the situation:

root@dragonflown:~# cat docker-compose.yaml 
version: '3'
services:
  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
    ports:
      - "6379:6379"
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /tmp/df:/data

networks:
  appnet:

root@dragonflown:~# docker exec root-dragonfly-1 redis-cli GET foo

root@dragonflown:~# docker exec root-dragonfly-1 redis-cli SAVE
ERR Permission denied: Couldn't open file for writing (is direct I/O supported by the file system?)

root@dragonflown:~# docker exec root-dragonfly-1 redis-cli SET foo bar
OK
root@dragonflown:~# docker exec root-dragonfly-1 redis-cli GET foo
bar
root@dragonflown:~# docker exec root-dragonfly-1 redis-cli SAVE
ERR Permission denied: Couldn't open file for writing (is direct I/O supported by the file system?)

root@dragonflown:~# docker compose down
[+] Running 2/2
 ✔ Container root-dragonfly-1  Removed                                                                   0.2s 
 ✔ Network root_appnet         Removed                                                                   0.1s 
root@dragonflown:~# docker compose up -d
[+] Running 2/2
 ✔ Network root_appnet         Created                                                                   0.1s 
 ✔ Container root-dragonfly-1  Started                                                                   0.3s 
root@dragonflown:~# docker exec root-dragonfly-1 redis-cli GET foo

root@dragonflown:~# docker exec root-dragonfly-1 redis-cli KEYS *

root@dragonflown:~#

Looks exactly the same as before.

Then I tried the redis image with the same configuration:

root@dragonflown:~# cat docker-compose.yaml 
version: '3'
services:
  redis:
    image: "redis"
    ports:
      - "6379:6379"
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /tmp/redis:/data

networks:
  appnet:

root@dragonflown:~# docker exec -it root-redis-1 redis-cli SET foo1 bar1
OK
root@dragonflown:~# docker exec -it root-redis-1 redis-cli GET foo1
"bar1"
root@dragonflown:~# docker exec -it root-redis-1 redis-cli SAVE
OK
root@dragonflown:~# docker compose down && docker compose up -d
[+] Running 2/2
 ✔ Container root-redis-1  Removed                                                                       0.3s 
 ✔ Network root_appnet     Removed                                                                       0.1s 
WARN[0000] Found orphan containers ([root-dragonfly-1]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
[+] Running 2/2
 ✔ Network root_appnet     Created                                                                       0.1s 
 ✔ Container root-redis-1  Started
root@dragonflown:~# docker exec -it root-redis-1 redis-cli GET foo1
"bar1"
root@dragonflown:~#

It looks like the above behaviors are all exactly the same as what I encountered at the moment of posting this discussion.

Update: Forgot to check what was in /tmp, so I repeated the steps above and got the same result, with /tmp:

@romange

3 replies

romange Aug 25, 2023
Maintainer

@WooMai Ubuntu can run our binary directly. Can you please check if the issue disappears if running the binary without docker? Another thing, can you run it with '--force_epoll=true' argument?

koraykoska Aug 25, 2023

@romange It's a different error now with force epoll. It's this one:

(error) ERR Permission denied: Couldn't open file for writing

WooMai Aug 25, 2023
Author

Maybe check your fs permission on the host?

koraykoska · 2023-08-25T10:36:26Z

koraykoska
Aug 25, 2023

@romange It's on DigitalOcean for me as well but instead of Debian it's Ubuntu 22.04.

0 replies

WooMai · 2023-08-25T11:27:19Z

WooMai
Aug 25, 2023
Author

@WooMai Ubuntu can run our binary directly. Can you please check if the issue disappears if running the binary without docker? Another thing, can you run it with '--force_epoll=true' argument?

Running Binary without Docker, seems good, using Ubuntu 22.04:

(btw, the latest version 1.8.0's link is not updated to docs)

# wget https://dragonflydb.gateway.scarf.sh/1.8.0/dragonfly-x86_64.tar.gz
# tar zxf dragonfly-x86_64.tar.gz
# mv dragonfly-x86_64 dragonfly

# terminal 1
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# ./dragonfly --logtostderr --dbfilename dump
I20230825 11:16:19.695794  7911 init.cc:69] ./dragonfly running in opt mode.
* Logs will be written to the first available of the following paths:
/tmp/dragonfly.*
./dragonfly.*
* For the available flags type dragonfly [--help | --helpfull]
* Documentation can be found at: https://www.dragonflydb.io/docs
I20230825 11:16:19.696341  7911 dfly_main.cc:666] Starting dragonfly df-v1.8.0-7c99d2d1111e2636556ad8c2afad43c396906d01
I20230825 11:16:19.696520  7911 dfly_main.cc:708] maxmemory has not been specified. Deciding myself....
I20230825 11:16:19.696566  7911 dfly_main.cc:712] Found 969.45MiB available memory. Setting maxmemory to 775.56MiB
I20230825 11:16:19.708907  7912 uring_proactor.cc:158] IORing with 1024 entries, allocated 102720 bytes, cq_entries is 2048
I20230825 11:16:19.712519  7911 proactor_pool.cc:147] Running 1 io threads
I20230825 11:16:19.719470  7911 server_family.cc:171] Data directory is "/root"
I20230825 11:16:19.723155  7912 listener_interface.cc:97] sock[5] AcceptServer - listening on port 6379

# terminal 2
# apt install redis-tools
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# redis-cli
127.0.0.1:6379> GET foo
(nil)
127.0.0.1:6379> SET foo bar
OK
127.0.0.1:6379> GET foo
"bar"
127.0.0.1:6379> SAVE
OK
127.0.0.1:6379> SHUTDOWN
OK

# terminal 1
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# ./dragonfly --logtostderr --dbfilename dump
I20230825 11:18:57.698421  8756 init.cc:69] ./dragonfly running in opt mode.
* Logs will be written to the first available of the following paths:
/tmp/dragonfly.*
./dragonfly.*
* For the available flags type dragonfly [--help | --helpfull]
* Documentation can be found at: https://www.dragonflydb.io/docs
I20230825 11:18:57.698873  8756 dfly_main.cc:666] Starting dragonfly df-v1.8.0-7c99d2d1111e2636556ad8c2afad43c396906d01
I20230825 11:18:57.699062  8756 dfly_main.cc:708] maxmemory has not been specified. Deciding myself....
I20230825 11:18:57.699110  8756 dfly_main.cc:712] Found 969.45MiB available memory. Setting maxmemory to 775.56MiB
I20230825 11:18:57.708930  8757 uring_proactor.cc:158] IORing with 1024 entries, allocated 102720 bytes, cq_entries is 2048
I20230825 11:18:57.712469  8756 proactor_pool.cc:147] Running 1 io threads
I20230825 11:18:57.718014  8756 server_family.cc:171] Data directory is "/root"
I20230825 11:18:57.718302  8756 server_family.cc:698] Loading /root/dump-summary.dfs
I20230825 11:18:57.720664  8757 listener_interface.cc:97] sock[5] AcceptServer - listening on port 6379
I20230825 11:18:57.721403  8757 server_family.cc:743] Load finished, num keys read: 1


# terminal 2
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# redis-cli
127.0.0.1:6379> GET foo
"bar"
127.0.0.1:6379> KEYS *
1) "foo"
127.0.0.1:6379> 
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# ls -lh
total 30M
-rw-r--r-- 1 root root 3.2K Aug  8 19:57 LICENSE.md
-rwxr-xr-x 1 root root  19M Aug  8 20:07 dragonfly
-rw-r--r-- 1 root root  11M Aug  8 21:53 dragonfly-x86_64.tar.gz
-rw-r--r-- 1 root root 4.0K Aug 25 11:18 dump-0000.dfs
-rw-r--r-- 1 root root 4.0K Aug 25 11:18 dump-summary.dfs
drwx------ 3 root root 4.0K Aug 25 11:11 snap
root@ubuntu-s-1vcpu-1gb-sgp1-01:~#

And here is the docker compose with force_epoll option, also Ubuntu 22.04:

root@ubuntu-s-1vcpu-1gb-sgp1-01:~# service docker start
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# mkdir test
root@ubuntu-s-1vcpu-1gb-sgp1-01:~# cd test
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# nano docker-compose.yaml
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# cat docker-compose.yaml 
version: '3'
services:
  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
      - "--force_epoll=true"
    ports:
      - "6378:6379"      
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /tmp:/data

networks:
  appnet:
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker compose up -d
[+] Running 1/1
 ✔ Container test-dragonfly-1  Started                                                                   0.5s 
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker exec -it test-dragonfly-1 redis-cli GET foo
(nil)
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker exec -it test-dragonfly-1 redis-cli SET foo bar
OK
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker exec -it test-dragonfly-1 redis-cli SAVE
OK
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# ls /tmp
dump-2023-08-25T11:24:59-0000.dfs
dump-2023-08-25T11:24:59-summary.dfs
dump-2023-08-25T11:25:00-0000.dfs
dump-2023-08-25T11:25:00-summary.dfs
snap-private-tmp
systemd-private-c82199f95c574144af60a233b4fad267-systemd-logind.service-Jf3qZ7
systemd-private-c82199f95c574144af60a233b4fad267-systemd-resolved.service-bj4kV8
systemd-private-c82199f95c574144af60a233b4fad267-systemd-timesyncd.service-BRICSA
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker compose down && docker compose up -d
[+] Running 2/2
 ✔ Container test-dragonfly-1  Removed                                                                   0.2s 
 ✔ Network test_appnet         Removed                                                                   0.1s 
[+] Running 2/2
 ✔ Network test_appnet         Created                                                                   0.1s 
 ✔ Container test-dragonfly-1  Started                                                                   0.3s 
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test# docker exec -it test-dragonfly-1 redis-cli GET foo
"bar"
root@ubuntu-s-1vcpu-1gb-sgp1-01:~/test#

It's working normally. I'll try this option on Debian later, it looks like the force_epoll thing.

0 replies

WooMai · 2023-08-25T11:36:31Z

WooMai
Aug 25, 2023
Author

It looks like a problem with force_epoll.
It worked fine after I added --force_epoll=true.
Thanks @romange !

root@debian-s-1vcpu-1gb-sfo3-01:~# nano docker-compose.yaml
root@debian-s-1vcpu-1gb-sfo3-01:~# mkdir test
root@debian-s-1vcpu-1gb-sfo3-01:~# cat docker-compose.yaml 
version: '3'
services:
  dragonfly:
    image: "docker.dragonflydb.io/dragonflydb/dragonfly"
    command:
      - "--save_schedule=*:*"
      - "--force_epoll=true"
    ports:
      - "6379:6379"
    ulimits:
      memlock: -1
    restart: always
    networks:
      - appnet
    volumes:
      - /root/test:/data

networks:
  appnet:
root@debian-s-1vcpu-1gb-sfo3-01:~# docker compose up -d
[+] Running 10/10
 ✔ dragonfly 9 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                 6.4s 
   ✔ edaedc954fb5 Pull complete                                                                          2.9s 
   ✔ d5d93e86ae17 Pull complete                                                                          4.6s 
   ✔ dbcbab6ce18a Pull complete                                                                          4.7s 
   ✔ d14c160bf2b6 Pull complete                                                                          4.7s 
   ✔ 4f4fb700ef54 Pull complete                                                                          4.7s 
   ✔ 27e60f37f528 Pull complete                                                                          4.8s 
   ✔ 3cca23c9d5be Pull complete                                                                          4.8s 
   ✔ 14a50fd65771 Pull complete                                                                          4.8s 
   ✔ e3fe35eed737 Pull complete                                                                          5.0s 
[+] Running 2/2
 ✔ Network root_appnet         Created                                                                   0.1s 
 ✔ Container root-dragonfly-1  Started                                                                   0.5s 
root@debian-s-1vcpu-1gb-sfo3-01:~# docker ex
exec    export  
root@debian-s-1vcpu-1gb-sfo3-01:~# docker exec -it root-dragonfly-1 redis-cli
127.0.0.1:6379> GET foo
(nil)
127.0.0.1:6379> SET foo bar
OK
127.0.0.1:6379> SAVE
(error) ERR Permission denied: Couldn't open file for writing
127.0.0.1:6379> 
root@debian-s-1vcpu-1gb-sfo3-01:~# ls
docker-compose.yaml  test
root@debian-s-1vcpu-1gb-sfo3-01:~# chmod 777 test
root@debian-s-1vcpu-1gb-sfo3-01:~# docker exec -it root-dragonfly-1 redis-cli
127.0.0.1:6379> SAVE
OK
127.0.0.1:6379> GET foo
"bar"
127.0.0.1:6379> 
root@debian-s-1vcpu-1gb-sfo3-01:~# docker compose down && docker compose up -d
[+] Running 2/2
 ✔ Container root-dragonfly-1  Removed                                                                   0.2s 
 ✔ Network root_appnet         Removed                                                                   0.1s 
[+] Running 2/2
 ✔ Network root_appnet         Created                                                                   0.1s 
 ✔ Container root-dragonfly-1  Started                                                                   0.4s 
root@debian-s-1vcpu-1gb-sfo3-01:~# docker exec -it root-dragonfly-1 redis-cli
127.0.0.1:6379> GET foo
"bar"
127.0.0.1:6379> 
root@debian-s-1vcpu-1gb-sfo3-01:~#

0 replies

koraykoska · 2023-08-25T16:11:40Z

koraykoska
Aug 25, 2023

@romange as mentioned by @WooMai the issue is solved by adding chmod 777 to the folder. This is far from ideal. What is the reason behind this?

1 reply

romange Aug 25, 2023
Maintainer

can you expand please. What folder? what do you map in docker-compose and what were the permissions before setting 777 ?

romange · 2023-09-05T05:38:09Z

romange
Sep 5, 2023
Maintainer

Fixed by 1.9.0

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to Save Database Dump in Docker Compose #1527

{{title}}

Replies: 9 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Unable to Save Database Dump in Docker Compose #1527

WooMai Jul 8, 2023

Replies: 9 comments · 7 replies

koraykoska Aug 25, 2023

romange Aug 25, 2023 Maintainer

WooMai Aug 25, 2023 Author

koraykoska Aug 25, 2023

romange Aug 25, 2023 Maintainer

koraykoska Aug 25, 2023

WooMai Aug 25, 2023 Author

romange Aug 25, 2023 Maintainer

koraykoska Aug 25, 2023

WooMai Aug 25, 2023 Author

koraykoska Aug 25, 2023

WooMai Aug 25, 2023 Author

WooMai Aug 25, 2023 Author

koraykoska Aug 25, 2023

romange Aug 25, 2023 Maintainer

romange Sep 5, 2023 Maintainer

WooMai
Jul 8, 2023

Replies: 9 comments 7 replies

koraykoska
Aug 25, 2023

romange
Aug 25, 2023
Maintainer

WooMai Aug 25, 2023
Author

koraykoska
Aug 25, 2023

romange Aug 25, 2023
Maintainer

WooMai
Aug 25, 2023
Author

romange Aug 25, 2023
Maintainer

WooMai Aug 25, 2023
Author

koraykoska
Aug 25, 2023

WooMai
Aug 25, 2023
Author

WooMai
Aug 25, 2023
Author

koraykoska
Aug 25, 2023

romange Aug 25, 2023
Maintainer

romange
Sep 5, 2023
Maintainer