[Bug] self-hosted file storage to minio not working effectively

[bdruth]

Bug Description

When adding photos to a page (using photo gallery in this instance) - only one image out of 3 is actually being saved. The photo gallery looks OK on the device creating it, but on sync'd devices, only one image shows.

Note

nginx logs when creating photo gallery

nginx-1  | 192.168.7.198 - - [25/Nov/2024:18:03:44 +0000] "POST /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/create_upload HTTP/1.1" 200 266 "-" "-"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:03:44 +0000] "POST /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/create_upload HTTP/1.1" 200 266 "-" "-"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:03:44 +0000] "POST /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/create_upload HTTP/1.1" 200 266 "-" "-"
nginx-1  | 192.168.7.198 - - [25/Nov/2024:18:03:45 +0000] "PUT /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/upload_part/4646858d%2D0a18%2D4657%2Db8a9%2Dae09b3b8ea45/pziF242UMzFXxb0gIkPFP2V42II48I7_k8awe6kG6ac=.jpeg/YmViMTlmZDEtNzc5ZS00MzEyLTkxOTktMTVhYWQwN2RlMTljLjZiMDYwNDc4LTZjMWQtNGE5Ni04M2RkLWEyNzJkMGRkOTA4MngxNzMyNTU3ODI0MzcxODI1OTYw/1 HTTP/1.1" 200 125 "-" "-"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:03:45 +0000] "PUT /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/complete_upload HTTP/1.1" 200 56 "-" "-"

Note

nginx logs when viewing page from a sync'd device
the first 3 entries are for the blobs for the 3 images, one is a 200 and the other 2 are 404. Those then are retried twice, each.

nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:04:14 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/sBJA8yHV6e5rvemsVYIleOdrhl_ERXQG0PYdWSsQeec=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:04:14 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/UIlOahfVt_v1AHMdb-eSieJpJhGqr7Khm8sqjPMpu2o=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 192.168.7.198 - - [25/Nov/2024:18:04:14 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/pziF242UMzFXxb0gIkPFP2V42II48I7_k8awe6kG6ac=.jpeg HTTP/1.1" 200 647623 "-" "Dart/3.4 (dart:io)"
nginx-1  | 192.168.7.198 - - [25/Nov/2024:18:04:20 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/sBJA8yHV6e5rvemsVYIleOdrhl_ERXQG0PYdWSsQeec=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:04:20 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/UIlOahfVt_v1AHMdb-eSieJpJhGqr7Khm8sqjPMpu2o=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:05:49 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/sBJA8yHV6e5rvemsVYIleOdrhl_ERXQG0PYdWSsQeec=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:05:49 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/UIlOahfVt_v1AHMdb-eSieJpJhGqr7Khm8sqjPMpu2o=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:13:55 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/sBJA8yHV6e5rvemsVYIleOdrhl_ERXQG0PYdWSsQeec=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"
nginx-1  | 172.21.0.1 - - [25/Nov/2024:18:13:55 +0000] "GET /api/file_storage/ad493c12-5712-4e39-a32c-5525d736f5bf/v1/blob/4646858d-0a18-4657-b8a9-ae09b3b8ea45/UIlOahfVt_v1AHMdb-eSieJpJhGqr7Khm8sqjPMpu2o=.jpeg HTTP/1.1" 404 0 "-" "Dart/3.4 (dart:io)"

Important

Minio shows only one blob in the console

How to Reproduce

1. Create a new page
2. Add an photo gallery
3. Add 3 photos to photo gallery
4. View new page from sync'd device

Expected Behavior

Sync'd page looks the same as page on original device that created it.

Operating System

macOS 15.1.1 and iOS 18.1.1

AppFlowy Version(s)

0.7.2?

Screenshots

Additional Context

Using my own minio instance, not the container that's built-in to the docker-compose.yml.

[appflowy]

@bdruth Thank you for reporting. I've moved this issue to appflowy-cloud repo. I will take a look later

[bdruth]

Confirmed that it's still happening w/ macOS v0.7.5 and iOS v0.7.4 (looks like 0.7.5 isn't in the App Store yet). As near as I can tell, the most recently published version of the backend is also present (docker compose pull doesn't get any updates).

[appflowy]

@bdruth I found the bug. Before uploading an image to the server, we calculate the file ID for the given image. If the image is the same, the file ID will also be the same, and duplicate files will not be uploaded. In your scenario, you uploaded three images, but since they are identical, only one image was uploaded. In the frontend application, we should remove duplicate images.

[bdruth]

Or maybe ... don't "calculate" the file ID and just "generate" a file ID for each file? I selected 3 distinct files from my drive, so they were not technically "the same file" - if that matters.

Also, when I reproduced this in 0.7.5/0.7.4, I didn't use identical images and it still happened, fwiw.

[appflowy]

@bdruth sorry to hear that. I can't reproduce it. Can I have your log files?

After open the folder, you will see log.2024-xx-xx