Post Completion of Export - What does Missing line Signify ?

[rajscode]

Question: In the pic below on the line that starts with Processed. What is the "missing: 48294" in this case explictly referring to ? EXIF Data on photos or actual photos ?

Thanks

[RhetTbull]

Hi @rajscode sorry this wasn't clear -- it's referring to missing photos. These are photos that are in the library but where the original image could not be located (and thus exported) by osxphotos. The most likely cause of this is that you are using iCloud with "optimize Mac Storage". You can use --download-missing to force Photos to download the image from iCloud so it can be exported.

[rajscode]

I'll try that, but the storage on the library osxphotos is running against is a FULL orignals version library.

But I'll definitely try that option.

[RhetTbull]

Do you have a lot of shared images (in traditional shared albums, not the new iCloud Shared Library)? It could be those as Photos does not download originals for shared images by default even with "keep originals". You could try the following:

$ osxphotos repl
... REPL will load
>>> missing = [p for p in photos if p.ismissing]
>>> len(missing)
>>> shared = [p for p in photos if p.shared]
>>> len(shared)

That will tell you how many photos osxphotos things are missing and how many are shared.

Another possibility is that if you're using referenced images (you manage the files not Photos) and Photos can't find the original referenced image, it would be reported as missing.

If none of the above helps, I can help diagnose some of the missing images to figure out what is going on.

[rajscode]

"shared images" being from shared albums that others have created and I subscribed to over the years ... from the get go of mobileme, etc..

Yes, I've joined quite a few of those from friends and family.

I'll try the repl option and see what happens (tmrw). Thx !

[rajscode]

@RhetTbull I went ahead and tried a few things with repl. Here is what I found for my library.

Prior export results --> missing: 48294 (first pic in this discussion)

Within the Apple Photos App by selecting All Photos at the bottom it shows:
Total of 155239 photo/video items ( 149919 photos, 5320 videos )

Running osxphotos ...
osxphotos repl --db "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"
Done: took 111.32 seconds
Getting photos
Found 196210 photos in 1.49 seconds

• photos: list of PhotoInfo objects for all photos filtered with any query options passed on command line (len=196210)
• all_photos: list of PhotoInfo objects for all photos in photosdb, including those in the trash (len=196215)

Executed

missing = [p for p in photos if p.ismissing]
len(missing)
39563

shared = [p for p in photos if p.shared]
len(shared)
40837

I assume below is correct to see if any referenced type photo or videos in Library

reffy = [p for p in photos if p.isreference]
len(reffy)
0

Seems like zero referenced items - I suspect we can cross that off the list.

Below is get_selected() info from a picture that is in a shared album that I joined as a subscriber.
The following values stood out for that particular picture.

'isreference': False
'shared': True
'isMissing': 1
'localAvailability': -1
'remoteAvailability': 1

I switched over to the Photos App window, right clicked on this shared with me picture and choose IMPORT.

Switched back over to osxphotos and issued another get_selected(). Didn't see any of the values above change. I repeated above steps on another shared picture to the same results.

'isMissing': 1
'localAvailability': -1

I was expecting the values to have changed to:
'isMissing': 0
'localAvailability': 1

I exited osxphotos and also from Photos App. Ran osxphotos repl command again.

Running osxphotos ...
Done: took 234.51 seconds
Getting photos
Found 196212 photos in 1.56 seconds

Load time was 111.32 (1st time) seconds vs 234.51 seconds (2nd time)

Found 196210 photos vs 196212 (the 2 pictures are the ones I actually right clicked in an album that was shared to me and IMPORTED). This I expected.

Checked the isMissing and localAvailability values on the two pictures I'd imported and this time it the values are what I expected. Does that mean after making changes to something in Photos while running osxphotos one would need to exit osxphotos and run it again for the objects in osxphotos be be up to date?

The values for those two downloaded shared photos were now:
'isMissing': 0
'localAvailability': 1

Reran the count on isMissing and shared and got the following results.
ismissing: 1st run 39563 vs 2nd run 39212 (a reduction of 351)
shared: 1st run 40837 vs 2nd run 40837 (identical)

The 351 reduction is puzzling to me as I didn't do any more IMPORTS except the two I mentioned. Only actions I'd taken in Photos earlier was I'd downloaded two photos as aforementioned and prior to that I'd clicked into a few shared albums which showed blank placeholders or a cloud icon and whilst I didn't click on any of those areas as I scrolled up or down the shared album the cloud icons changed into low res photos (most likely preview thumbnail sized downloaded from cloud).

Is there a recommended way to automate the IMPORT of these Shared Album photos if they aren't already imported (no duplicates). Or is it just as simple as trying the following addition to my existing crontab job:

osxphotos export photo_library_backup/ --load-config osxphotos_backup.toml

Instead to:

osxphotos export photo_library_backup/ --download-missing --load-config osxphotos_backup.toml
or
osxphotos export photo_library_backup/ --download-missing --use-photokit --load-config osxphotos_backup.toml

Would solve two things for me. (a) I get to keep the Photos Library "complete" by having it import in shared albums contents so they are stored locally as well as in iCloud. (b) I'd be able to have these "shared" pictures also join the export that I am already doing to a directory structure and thus they too get backed up (in my case Backblaze).

Thanks!

[RhetTbull]

Within the Apple Photos App by selecting All Photos at the bottom it shows:
Total of 155239 photo/video items ( 149919 photos, 5320 videos )

Photos does not show the shared images from shared albums in the count.

Does that mean after making changes to something in Photos while running osxphotos one would need to exit osxphotos and run it again for the objects in osxphotos be be up to date?

Yes. Currently, osxphotos imports the entire database into memory (that's why there's a large memory allocation and the 100+ second start up time in your case). I have a design done for a rewrite that would allow osxphotos to dynamically query the database but its a long term project. (I only have a few hours per week to work on osxphotos).

Load time was 111.32 (1st time) seconds vs 234.51 seconds (2nd time)

This is odd to see load time double. Likely some other process was competing for resources which caused osxphotos to slow down as it was loading the same amount of data (your Photos database at ~/Pictures/Photos Library.photoslibrary/database/Photos.sqlite)

The 351 reduction is puzzling to me as I didn't do any more IMPORTS except the two I mentioned.

Just clicking on a shared album or viewing the contents of an album is enough sometimes to cause Photos to download the images from iCloud. This is likely what happened.

Or is it just as simple as trying the following addition to my existing crontab job:

Yes, adding --download-missing to the command line or download_missing = true to your config file will cause the missing shared photos to be downloaded.

If you're running this as a cron job or with launchd, you may need to tweak permissions of your job so that it can script Photos (this is a separate permission than just being able to access the library). --download-missing uses AppleScript to do the download. See this discussion.

There's another flag, --use-photokit you can add along with --download-missing that uses the native PhotoKit interface from Apple instead of AppleScript to do the download. This is higher performance but some users have found it more problematic due to permissions issues which I've not yet been able to resolve. I recommend you try --download-missing --use-photokit and if that works, stick with it otherwise try without --use-photokit.

There is one other source of "missing" photos on Ventura -- photos that have been shared with you in Messages but not imported to the library still show up in the library. osxphotos can see these photos exist but cannot yet download them. There's an open issue to add support for these (#1054). Until this is implemented, osxphotos will not be able to export these images.

[rajscode]

Thank you! Will do as per your recommendation above.