Sometimes osxphotos is killed by SIGABRT

[chengguangnan]

Sometimes osxphotos is killed by SIGABRT and there are no crash log. So I'm not sure what's the cause. Maybe memory leak and out of memory? I'm running the command with Terminal.app, so it's not about requesting MacOS permissions.

[chengguangnan]

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Python [9714]
Path:                  /opt/homebrew/*/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.12.1 (3.12.1)
Code Type:             ARM-64 (Native)
Parent Process:        fish [9643]
Responsible:           Terminal [1424]
User ID:               501

Date/Time:             2024-01-19 20:10:25.0245 +0800
OS Version:            macOS 14.2.1 (23C71)
Report Version:        12
Anonymous UUID:        D925B12E-0CA5-3F29-09D8-7BEBEAEA15EA

Sleep/Wake UUID:       3717A081-C1EB-4F71-A766-FEAB878A5363

Time Awake Since Boot: 180000 seconds
Time Since Wake:       95324 seconds

System Integrity Protection: enabled

Crashed Thread:        2  Dispatch queue: com.apple.photos.assetResources.fileIO

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Termination Reason:    Namespace SIGNAL, Code 6 Abort trap: 6
Terminating Process:   Python [9714]

Application Specific Information:
abort() called


Thread 0::  Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib        	       0x184ced06c __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x184d2a5fc _pthread_cond_wait + 1228
2   Python                        	       0x102e104b0 PyThread_acquire_lock_timed + 248

[RhetTbull]

What version of macOS are you using?
How much RAM does your Mac have?
How large is your Photos library?
What is the size of the .osxphotos_export.db export database in the export directory?
When it crashed was it during a full export/update or were you doing something else?
What was the full osxphotos command you were running?

My guess is out of memory error. osxphotos consumes a lot of memory when working on a large Photos library and consumes even more if using --ramdb. In the current design there is nothing I can do about this. I am working on a redesign of the core of osxphotos (which reads the database) that will be somewhat slower in execution but consume much less memory. Currently, osxphotos loads the entire Photos library into memory (after denormalizing the data in python data structures). Thus if your Photos library database is a couple of GB (not uncommon in a very large library) this all gets loaded into memory. The export database can also exceed 1GB for a large library and this will also get loaded into memory with --ramdb.

I've seen osxphotos consume 8GB+ of memory for a large library. The Apple Silicon Macs generally have less RAM available and despite Apple's assurance that they are more memory efficient, it is not uncommon to see an Apple Silicon Mac run out of memory. In Sonoma in particular, I've seen reports of the window server and other processes crashing due to out of memory.

If you're using --ramdb you could try dropping this. Your export will be much slower, especially if export database is on a NAS or external drive, but will use much less memory. (A partial solution to this is to move the export database to an internal volume and use --exportdb to specify the path to the export database.

If you're using PhotoKit via --use-photokit it's more likely there's a memory leak. That code has to actually allocate memory to use Objective-C objects and it doesn't play well with the Python garbage collector so if there's a memory leak it's most likely there. So one option would be to not use --use-photokit if you're using it.

My time for working on osxphotos is very limited and I'm focused on another (related) project at the moment so I don't know when I'll get to this but I'll open an issue so it doesn't get lost. If you answer the questions above it will be helpful for tracking this down.

[chengguangnan]

I just found that my library's Photos.sqlite is 2.8GB large. I currently have about 111k photos and 2k videos left. That's 2811883520 / 130000 = 21MB 21KB of metadata per file.

My Photo Library was almost using up all 2TB of iCloud, that's when I decied to get off from iCloud. So far my Photo Libaray is about 300GB.

I need to metion that I have tried to use --exportdb, but sometimes it still gets killed even with --exportdb, so I decieded to open a issue here. And reboot the Mac seems temporarily reslove the problem. So yes it seems to be memory related.

Thank you for your very detailed response. I think I can work around this issue by exporting smaller subset. Just try to feedback the problems that I've encountered.

[RhetTbull]

Thanks. The feedback is helpful. If the deleted photos haven't been purged from the "recently deleted" cache (which is kept for 28 days) the metadata is still in the Photos.sqlite which could be part of the issue. It will be purged when Photos permanently deletes the items.

You can use --from-date and --to-date to limit the range of photos exported. From date is inclusive and to date is exclusive so keep that in mind when exporting chunks so you don't miss a day.

[RhetTbull]

For comparison, my library has 31425 assets and an additional 6445 assets in shared albums and the Photos.sqlite is 942MB which comes out to ~26KB per photo of data.