Support validating tokens without an outbound network connection

[duncanmmacleod]

Currently one cannot validate a bearer token without an outbound network connection:

$ scitokens-test-access $BEARER_TOKEN https://cilogon.org/ligo ANY read /frames
Failed to deserialize a token: Couldn't connect to server

Is this feasible? It would enable using tokens for jobs on cluster nodes that are not exposed to the network (e.g. on LIGO/Hawk in Cardiff).

[jbasney]

Unlike X.509 CA signing keys, which IGTF updates once per month via RPMs, the SciTokens signing keys need to be updated more frequently, because there are no CRLs in the SciTokens world. Assuming the cluster head node (or management node) has external connectivity, then the head node could update the SciTokens key cache once per hour / once per day and distribute the key cache to the cluster nodes. We'd need a tool that takes a set of trusted issuers and updates the key cache for all the issuers, rather than letting the library do it on demand. This would also require fixing caching lifetimes (#80 & #86).

In scitokens-cpp, the key cache is in $XDG_CACHE_HOME/scitokens/scitokens_cpp.sqllite. See: https://github.com/scitokens/scitokens-cpp/blob/master/src/scitokens_cache.cpp

In Python, the SciTokens key cache is in $XDG_CACHE_HOME/scitokens/scitokens_keycache.sqllite. See: https://github.com/scitokens/scitokens/blob/master/src/scitokens/utils/keycache.py

Seems to me we should try to use the same cache for both C++ and Python.

[bbockelm]

@duncanmmacleod - I don't know if it's helpful, but manipulating the keycache is something I was working on earlier this week:

#99

With that, potentially a nightly cronjob to preload the public keys would suffice?

Beyond that, libcurl is used by scitokens-cpp to download the keys. If you can think of a way to manipulate the environment variables set for the validation process, it might obey https_proxy (mind you, then you'll have to maintain a SSL bump for your cluster ... no easy option!).

Brian

[jbasney]

This feature request came up again in discussions about Fermilab worker nodes being blocked by CILogon rate limiting. If the keycache could be populated on a shared location for the cluster (or in CVMFS?), we could point the worker nodes at $XDG_CACHE_HOME/scitokens/scitokens_cpp.sqllite and $XDG_CACHE_HOME/scitokens/scitokens_keycache.sqllite rather than each worker node trying to populate the keycache for every job. Since every job runs in a temporary home directory, there's no shared cache across jobs.

[DrDaveD]

We could keep a cache updated under /cvmfs/oasis.opensciencegrid.org/mis pretty easily. What would it take to have a pilot job tell the scitokens-cpp library to find its cache there and for scitokens-cpp to quietly accept a read-only cache?

Or, it should probably be the first place to look, and then proceed to the local cache if an issuer is not found there.

[DrDaveD]

We can't just set $XDG_CACHE_HOME because that might be used for other purposes. We would need an environment variable specifically for scitokens.

[DrDaveD]

This is now being discussed as a WLCG-wide standard for key caches in WLCG-AuthZ-WG/common-jwt-profile#75.

[djw8605]

Thanks for the pointer. I have an initial draft of an implementation here: #172

I took the easy way out, and added a new environment variable that points directly to the cache file. Nothing fancy like something in /etc, but it still follows the other cache location rules in addition to the new environment variable. It also builds a new tool to manage the keycache, print the contents of the cache, add a new cache (with a set expiration) and printing the location of the keycache file.

[DrDaveD]

Ah, but the implementation will need to be considerably different if we have a standard that can be conveniently used by other libraries.