Google API Bugs

[gilesknap]

I'm creating this Issue to track the Google API issues that affect what we can achieve with gphotos-sync:-

• Some video files get error 500 when accessing their baseUrl. This was reported as fixed and my library no longer sees the issue, but see Bad IDs #125
  • https://issuetracker.google.com/issues/116842164
  • update: I see the new error now and a new ticket covers it:
  • https://issuetracker.google.com/issues/141255600
  • UPDATE AGAIN - It looks like this is fixed - see
    • https://issuetracker.google.com/issues/141255600#comment8
    • BadIds logic has been removed.
    • BadIds logic restored because of Failure downloading , already downloaded #221 (comment)
  • Further Update - Flavian identified that the 500 error happens for Videos that the API reports as still processing see Error 500 while downloading some videos #301
    • https://issuetracker.google.com/issues/181055574
• Original quality images are not downloaded in original quality
  • https://issuetracker.google.com/issues/112096115
• Raw photos are downloaded compressed (Photos downloaded are compressed #111)
• Videos are transcoded to high compression and sometimes lower framerate (Still no workaround for videos not downloading in full quality? #108)
  • https://issuetracker.google.com/issues/80149160
  • https://issuetracker.google.com/issues/113672044
• The GPS info is stripped from Exif (Location data #64)
  • https://issuetracker.google.com/issues/80379228
• No modified date in metadata means we cant check for photos edited in the Web App
  • https://issuetracker.google.com/issues/122737849
• Burst shots are not supported
  • https://issuetracker.google.com/issues/124656564
• Multiple limitations make two-way sync impractical (Add media upload capability (Two Way Sync) #30)
  • https://issuetracker.google.com/issues/109759781
  • https://issuetracker.google.com/issues/132274769
  • cant upload in 'High Quality'
  • https://issuetracker.google.com/issues/135714733
• Mediaitems.list (and .search?) sometimes returns an empty list with a nextToken, if you keep requesting with nextToken, more items are returned eventually, also seen as Intermittent Null response in mediaItems.batchGet. (Indexed only photos in albums #112, Debug logging error after 'Null response in mediaItems.batchGet' message #179, Null response in mediaItems.batchGet #102)
  • https://issuetracker.google.com/issues/151032572
  • https://issuetracker.google.com/issues/129250413

(this issue linked in https://issuetracker.google.com/issues/80149160#comment36)

[LootenPlunder]

Just got a new camera and started shooting raw but the files were coming back incomplete. Seems we're at the mercy of google to fix #111 for this project to work with RAW in any way?

[gilesknap]

@LootenPlunder Yes I'm afraid you are correct. If you require a backup I suggest you use a different service to Google, there as been no movement on this bug for some time. I'm OK with their free space 'high quality' images for my use case, but I'm not OK with what they do to video files when downloaded through the API.

[LootenPlunder]

@LootenPlunder Yes I'm afraid you are correct. If you require a backup I suggest you use a different service to Google, there as been no movement on this bug for some time. I'm OK with their free space 'high quality' images for my use case, but I'm not OK with what they do to video files when downloaded through the API.

Thanks, and thanks again for putting this all together!

[satmandu]

Does Google Takeout download all pics and videos? Is there a way to script that to download incrementally? You can select albums for download there.

[gilesknap]

@satmandu Google Takeout does download everything (including GPS tags in jpgs!) but in a really messy format and there is no incremental backup (yes, it does let you select which albums to download so you could use it with very disciplined album creation, I suppose)

I'm toying with the idea of a python program to take a takeout and create a neat gphotos-sync like file structure from it. But this is to be a one-off to allow me to exit from Google (because of their lack of interest in the above issues which makes it look like deliberately keeping control of your data). I've done some investigation and it would be possible to do this.

[CorneliousJD]

I just started using this tool, and have been importing the downlaoded photos into PhotoPrism, but sadly I'm noticing the EXIF data is incomplete (no GPS or locaiton data).

It looks like the main readme.md file says that there's currently no working way to get location data from the API right now and they are not intersted in implementing?

I am trying to still use Google Photos from my phone to auto-upload every photo it takes, and then download it to my home server with this utility, but not having location data is a big bummer unfortunately.

Hoping there's a good solution somehwere, but not holding my breath with Google.

[gilesknap]

@CorneliousJD if GPS data is important I would look elsewhere than Google for your photos management/storage. I don't think it is likely that we'll ever fix this. I did implement a workaround that used scraping of the GPS via selenium (browser automation) but it got shut down. There is another approach that worked last time I looked using java script but it is pretty klunky and I do not expect it to last long.

If anybody cares to look at this then the beginnings of an out of band photo download was implemented here https://github.com/gilesknap/gphotos-sync-ui but I have just noticed that the original author has deleted the project and I no longer have any enthusiasm to pursue this issue myself.

[CorneliousJD]

Hi @gilesknap thanks for getting back -- a non-location aware photo backup is still better than no backup, so I'll still be using this for now and may supplement bi-yearly or something with takeout data of full-res photos, and then this would be a stop gap for automated downloads between that.

If you do end up creating something that organized takeout photo data for us in the same folder structure that gphotos-sync does, that would be a nice way to let us manually supplement/replace the "high quality, locationless" photos with "original quality, full EXIF" ones from takeout, it would just be a manual process in doing so. Just a thought!

Even without location data this is still an amazing utility that I use every day.

[develar]

Bloody Google — indeed, photos in RAW downloaded with original extension (.arw) but in JPEG format (can be checked using fileinfo command).

It means, that API is useless — you cannot backup your videos, you cannot backup your photos if you shoot in RAW.
This tool works more reliable than https://github.com/mholt/timeliner (rate limit, progress), but both tools affected by silly Google API bugs.

[gilesknap]

@develar yep, I must get around to implementing my exit strategy!

[develar]

Another point — if photo is edited, google takeout provides original file as is and edited with prefix -edited.jpg (one json metafile for all). Via API only one edited version is downloaded.

[develar]

I ended up with using Google Takeout — https://github.com/develar/gphotos-takeout It is not convenient and bad, as to do sync you have to manually select albums to take-out and remove manually already copied files (e.g. if you delete photo from google photos, no easy and robust way to automatically detect it and remove from your backup). But I do not see any other way.
In my case archive via API ~100GB and via Google Takeout 245 GB. Google takeout duplicates files not only in albums, but also in year dirs, so, special tool to correct downloaded data required not only because format of Google takeout is awfull, but also to deduplicate.

I really hope that someday all mentioned bugs will be or fixed, or some workaround will be found. But knowing Google, no such hope.

[ScottESanDiego]

@develar I've been doing the same (Takeouts, then download and untar), and found that rdfind can hardlink the duplicates to keep space under control.

It's annoying, but works.

[gilesknap]

@ScottESanDiego @develar my "exit strategy" is some software to organise takeout into nice structure like gphotos-sync has. The intention is that it would be used once and then I'd move to a different service. However one could stick with Google and do a takeout every so often. Assuming you have the bandwidth!

I'm not sure when I'll get around to this but if there is lots of interest then maybe soon.

[ScottESanDiego]

I like that model @gilesknap (while not ideal, it seems like the best under the current API limitations). In theory one could automate that with the periodic Google Takeout -> download/untar to the local system (cron job checking for when the takeout files appeared somewhere) -> deduplicate/reorganize/whatever tool (aka the theoretical software from your comment)).

The wish-list for "the tool" would include"

1. Act on the raw tarballs to reduce the size of the scratch space needed
2. Be smart about not overwriting files that already exist in the destination (ala, be friendly to my filesystem)
3. Judicious use of hard-links or equivalent to further reduce space requirements

My current kludge for the above (sans the nice reorganization of "the tool") is this bash ugliness:

for x in `find ${TAKEOUTDIR} -name \*.tgz`; do
        tar --extract --file ${x} --skip-old-files --directory=${OUTPUTDIR} --verbose
done

rdfind -makehardlinks true -makeresultsfile false ${OUTPUTDIR}

[satmandu]

Is there a limit on how often one can use Google Takeout?

[gilesknap]

There may be a limit. I expect there will be one soon if we all start using it for regular backups!

[develar]

and found that rdfind can hardlink the duplicates to keep space under control.

@ScottESanDiego Thanks for the link. What's I have discovered — files are duplicated not only for "auto-uploaded album" vs "album" (e.g. 2020-03-20 vs "Trip to Alabama"). But even in one dir — Google Takeout for some reasons can duplicate files with suffix (1) (I double checked — it is not due to user error during decompressing/merging of downloaded takeout archive, but due to some google bug). To speed-up deduplicating (external 2tb hdd drive is pretty slow ;)), not every file is checked, but only if photoTakenTime is the same. And results are pretty good — after that only several duplicates for 245 GB collection (probably because you have uploaded such duplicates using different file names).

one could automate that with the periodic Google Takeout

This tool will be sort of hack, fragile, and very complex to implement (see https://superuser.com/questions/716756/how-to-automate-regular-google-takeout-backups-to-cloud-storage). Because in this case tool should somehow act as a browser and emulate user (no API, not possible to use access token). And the question is — will be ok for you to run such kind of tool that have full access to your Google account... (as you cannot fully trust what's tool does without checking source code). I decided just do manual google takeout each several months (and keep originals for this period as back-up).

[ScottESanDiego]

@develar By "automate" I didn't mean request the Takeout, I meant setup Google do generate a new Takeout "Export every 2 months" feature. Then all we have to do is watch a Google Drive/DropBox/something location for new files to appear.

[karan]

I'm working on https://github.com/karan/gphotos-takeout where the idea is that the program would be stateful and keep a local db, the result of parsing a takeout tgz archive. The idea is that the main program (call it "ingester") would parse a un-tarred archive and store information about photos in a sqlite db. So every few months, you would download your archive, and run the main program on it (again, no untar needed).

Then, we can write auxiliary programs to act on the structured information about photos. For eg, we could easily write a program to walk the database and store individual photos to a target directory, and build album directories with symlinks.

It's not terribly hard code to write, but I don't have a whole lot of time to finish it myself quickly, so definitely would love contributions (see TODOs in the code).

After this is done, we can easily treat this as a link in a chain:

• Takeout creates an archive every 2 months and saves it in Drive/OneDrive etc
• You download your archive using rclone
• You ingest the archive to an existing database
• You run the auxiliary program to de-dupe photos and clean directories

[mrzoops]

Would there be any sort of workaround the bug regarding videos not being downloaded in full resolution/framerate? I know that when downloading a handful of files via the browser you get the true/full versions, so would it be possible to have this program go in and pull videos only outside of the api. Using a hidden browser method but still automatically?

[gilesknap]

@mrzoops one day there could be a workaround - see #271. I just need the energy and motivation to have a go at this. The problem with it is that it is a hack and Google can break it by changing the Web UI.

Next projects on my list are

• https://github.com/gilesknap/mciwb (to teach my son Python)
• https://github.com/gilesknap/IaC-at-home/tree/main/imagebuild (so I can add PoE Raspberry Pis to my Metal as a Service Kubernetes Cluster
• Maybe then look at Use Chrome DevTools Protocol to Work Around Photos API issues. #271

I've also been wondering if we could lobby Google to fix these bugs! Maybe using something like #347