How to speed up --update

[chengguangnan]

See the two exports I did on a large album, the second export skipped about 97% photos but the whole process still takes about 2.5 hours.

This is strange, because I was assuming --update will read the previous .osxphotos_export.db and skipping exported photos should be very fast, no?

Exporting 4291 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 4291 photos, exported: 4340, missing: 47, error: 0
Elapsed time: 2:59:31

Exporting 4291 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 4291 photos, exported: 4, updated: 91, skipped: 4191, updated EXIF data: 0, missing: 101, error: 0
Elapsed time: 2:31:19

[RhetTbull]

This is odd. What's the full export command?

If the export drive is slow (network drive) or you are not using --ramdb I could see that the update would be very slow but for only 4k photos, this seems unusually slow. I routinely run osxphotos to update a backup of my 30K image library to a fast external drive and it takes about 1.5 hours. A lot of this time is reading the disk directory to match which files are there to see if they need to be updated. When run with --update, osxphotos checks that the target file is on disk and verifies its signature matches the export database. This means an ls and a stat occurs which isn't speedy.

[chengguangnan]

Adding --ramdisk seems helpful, with some other changes to my environment, export time reduced to one hour. I was exporting to a external USB 3.0 SATA SSD, which I was not consider it as slow, but now we know.

BTW, right now the --update run is much fast, within a minute. This is related to #1385.

It seems that the cause is that, my photo libaray has many videos with the same original filename, eg IMG_0001.mov (2019) and IMG_0001.mov (2022, new phone). I didn't verify this, but it seems that upon exporting, those videos were overwriting each other and cause some EDITED_SIG_DIFFERENT?

I added --filename "{original_name}_{id:06d}" and now exported files are skipped very fast.

[RhetTbull]

I added --filename "{original_name}_{id:06d}" and now exported files are skipped very fast.

Good, glad you figured out the issue. If the export directory structure is sorted by date, album, etc. then it's unlikely you'll get many collisions. But in the case where there was a collision, osxphotos should have incremented the filename. For example, You should end up with IMG_0001.mov, IMG_0001 (1).mov, IMG_0001 (2).mov, etc. Did this not happen?

If you are using --overwrite then the files would get overwritten but you shouldn't use this except in specific use cases. It's not needed with --update which will auto-increment filenames.

[chengguangnan]

I couldn't reproduce the above issue quickly right now. I think what might happen is:

1. Photos set to "Optimize Mac Storage" and has a low disk space available.
2. osxphotos export --download-missing, the movie is downloaded from iCloud exported and then "optimized".
3. Second export, the file is downoloaded from iCloud again, which produced a different mtime thus a EDITED_SIG_DIFFERENT.

This is just theory, but it explains the slow second export and sig different.

[RhetTbull]

I think you are right. Looking at the code, EDITED_SIG_DIFFERENT occurs when the signature stored in the database of the exported file differs from the signature of the edited file in Photos.

I'll need to think about how it might be possible to fix this. I could compute the fingerprint (hash) of the edited file and store that in the export db. This would be a bit slower but would guarantee no duplicate exports.

There's a modification time in the Photos database and that would be the logical choice to compare. However, on Photos <= version 4.0, the modification date/time was adjusted any time the photo's metadata (e.g. tags) were changed so this wasn't reliable. I've been developing osxphotos since macOS Sierra (Photos 2.0) and it's remained backwards compatible to this day. Adding different checks in the export db for different versions of Photos would add complexity I'd like to avoid.

I'm open to ideas on a better way to check for modification of the edited photo!

Also, if the edited file is missing (likely in your case due to optimization) then the signature is assumed to be different:

        if options.edited and (
            not src or not fileutil.cmp_file_sig(src, file_record.src_sig)
        ):
            # edited file in Photos doesn't match what was last exported
            return ShouldUpdate.EDITED_SIG_DIFFERENT

osxphotos export isn't really ideal for use with optimized libraries though --download-missing provides an option to guarantee you can get an export. I'm wondering if I shouldn't just make "best effort" in this case and optimize for the "normal" case of a "store originals on this Mac" library?

[chengguangnan]

osxphotos export isn't really ideal for use with optimized libraries though --download-missing provides an option to guarantee you can get an export. I'm wondering if I shouldn't just make "best effort" in this case and optimize for the "normal" case of a "store originals on this Mac" library?

That's totaly okay with me. I'm trying to get off from iCloud. I have an iCloud library of 2TB, and I'm exporting and deleting them from iCloud. Once fully exported I will run osxphoto for the local library and this issue will be irrelevant.