@amaltaro I took the liberty to update the description of the issue, according to the discussion in #12081.
@stlammel , feel free to provide additional comments and suggestions here, rather than in the PR linked above, at your convenience.

I think we want to make the stage-out token safe now, i.e. in case a token is in the environment and transfer with token doesn't work, the stage-out doesn't fails. (Right now, if HTCondor makes a token available but token-based transfer doesn't work stage-out may fail.)
Something like:

print date/time
print hostname
print GFAL*, PYTHON*, and LD_* environment
print gfal-copy location
print PFNs

gfal-copy ...
if ( rc == 0 ) done

if ( X509_USER_PROXY is set )
sleep 3 sec
in subprocess {
unsetenv BEARER_TOKEN
unsetenv BEARER_TOKEN_FILE
gfal-copy -v ...
if ( rc == 0 ) done
}
if ( BEARER_TOKEN or BEARER_TOKEN_FILE is set )
sleep 3 sec
in subprocess {
unsetenv X509_USER_PROXY
gfal-copy -v ...
if ( rc == 0 ) done
}

sleep 15 min
print date/time
voms-proxy-info -all
httokendecode
gfal-copy -vvv ...

(token information may change underneath us, so we should print it just before the debug gfal-copy).
'just a thought/suggestion.
Thanks,

• Stephan

Right! We will add the necessary safety mechanism once we integrate tokens in the grid jobs. Thanks Stephan.

Issues I have noticed so far:

• When jobs are submitted via the condor python bindings, inside the WMAgent container, the job is submitted, but /usr/bin/condor_vault_storer does not seem to be triggered.

I am currently working that around by:

1. Executing a condor_submit with a test job once from the host
   Token seems to stay and refresh afterwards.

If cms_readonly scope is used, this does not happen and we need a manual refresh, but for production, we don't need any scope.

maybe @stlammel can comment more on what @khurtado reported

We have not used the python binding in the initial tests. If there is a bug, we'll need to make a test case and submit a bug report.

• Stephan

We have not used the python binding in the initial tests. If there is a bug, we'll need to make a test case and submit a bug report.

I am not sure I understand this comment.
When we were testing these functionalities in a Fermilab agent, we did test it with a workflow going through the WM system (testbed), which also includes job submission by the agent (hence using python bindings).

It is important to notice though that we were using a CC7 node with RPM based deployment. Maybe this is the difference that you are trying to make here? Is the docker solution imposing limitations to this?

I didn't know/realize this Alan. I was only aware of the condor_submit test. Then this is likely a config issue and not bug.
Thanks,

• Stephan

@amaltaro Submission through the python bindings work as long as there is a token present already. This can be achieved by creating the token by hand, or having a client that does this for you (like the host condor_submit).

If the tests at FNAL started with a condor_submit first, and a job submission via WMAgent after, then this issue would not have shown up at all.

The python bindings alone don't seem to invoke any token generation script, but that is probably not a huge deal because the first job usually prompts you to an oauth website for the authentication, and since we submit the jobs automatically via JobSubmitter, we do need to do this interactively beforehand anyway.

I am just noting that as things are now, we need to generate a token at the host, either by hand or with a simple condor_submit test job, in order for WMAgent to submit and use the tokens later on.

I don't think we need to dig too much into this at this point though, because this part of the token generation is going to change eventually, so that this is done without an interactive URL that the user needs to click and authenticate on.

I thought about Kenyi's message about condor_vault_storer to being executed on a submit via python API. The initial authorization forwarding requires interactive execution and the device code URL might be lost on stdin in case of python API.
So, it might be called/executed but prompt for user action not getting through/being lost.
Thanks,

• Stephan