Access to prod memcache

[snarfed]

This would help my local manual datastore edits take effect more quickly. Should be straightforward, except exposing the memcache instance externally, not sure what that will take.

[snarfed]

Added a custom HTTP tunnel ^ through the app itself, works ok (not great, just ok) for manual commands. Local Python shell still isn't connected though.

[snarfed]

Getting more important, this is making manual datastore changes unreliable or not usable.

[snarfed]

This has the most comprehensive set of options I've seen so far for connecting to Memorystore from outside GCP: https://blog.stackademic.com/connect-to-google-cloud-memorystore-redis-from-a-local-machine-and-use-redis-in-next-js-5e5a534d45b6

Currently trying the IAP TCP forwarding route, https://cloud.google.com/iap/docs/tcp-forwarding-overview

[snarfed]

So one catch here is that ideally I'd set up the tunnel directly to the Memorystore instance, not to a generic Compute Engine VM. That seems uncommon, but maybe possible? At least https://stackoverflow.com/a/71428305/186123 says so. Here's what I've tried, based on the docs and posts above and that SO answer:

$ gcloud compute firewall-rules create allow-ingress-from-iap \
  --direction=INGRESS \
  --action=allow \
  --rules=tcp:11211 \
  --source-ranges=35.235.240.0/20

$ gcloud compute networks subnets update default \
  --region=us-central1 \
  --enable-private-ip-google-access

$ gcloud iap tcp dest-groups create memorystore-memcached \
  --region=us-central1 \
  --ip-range-list=10.126.144.0/24

$ gcloud iap tcp dest-groups add-iam-policy-binding \
  --member=user:[email protected] \
  --role=roles/iap.tunnelResourceAccessor \
  --dest-group=memorystore-memcached \
  --region=us-central1

$ gcloud compute start-iap-tunnel 10.126.144.3 11211 \
  --region=us-central1 \
  --network=projects/bridgy-federated/global/networks/default \
  --local-host-port=localhost:11211

...
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4033: 'not authorized'].

...where 10.126.144.3 there is the memcache discovery IP. Not sure that's quite what I want.

https://blog.stackademic.com/securely-connect-to-google-cloud-memorystore-redis-from-your-local-machine-using-vm-and-iap-3556a6d53ab0 is relevant, but assumes I'm tunneling to a Compute Engine VM. google-github-actions/ssh-compute#44 also has this error, but no obvious answer other than IAM roles, which I've added.

[snarfed]

Progress maybe, changed --network and added --dest-group:

gcloud compute start-iap-tunnel 10.126.144.3 11211 \
  --region=us-central1 \
  --network=default \
  --dest-group=memorystore-memcached \
  --local-host-port=localhost:11211

...
ERROR: (gcloud.compute.start-iap-tunnel) While checking if a connection can be made: Error while connecting [4003: 'failed to connect to backend']. (Failed to connect to port 11211)

Sounds like it's setting up the tunnel, but then failing to connect to Memorystore.

[snarfed]

Other ideas:

• add a custom memcached tunnel to atproto-hub
• ssh into atproto-hub and run the Python REPL there, https://cloud.google.com/appengine/docs/flexible/debugging-an-instance#connecting_to_the_instance