Add support for expressions in context variables

[nuwang]

context

From hrzolix on matrix

[hrzolix]
I'm trying to give our users freely to choose parameters for tools, so far cores and memory work but I've tried to add the option for choosing queue, because we in background we have cpu, cpu_30, cpu_180 (7, 30, 180 days). I'm running into a problem here NameError: name 'params' is not defined

Here is the tpv_rules_local.yml:

global:
  default_inherits: default

tools:
  default:
    abstract: true
    cores: 2
    mem: 4
    params:
      queue: 'cpu_180'
    rules:
      - id: resource_params_defined
        if: |
          param_dict = job.get_param_values(app)
          param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
        cores:  int(job.get_param_values(app)['__job_resource']['cores'])
        mem: int(job.get_param_values(app)['__job_resource']['mem'])
        params:
          queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
  local_env:
    runner: local_runner
    max_accepted_cores: 1
    params:
      tmp_dir: true
  pbs:
    runner: pbs_drmaa
    params:
      native_specification: "-q {params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"

[Nuwan Goonasekera]
Can you try using a context variable instead? Using params within params is possible, but tricky and probably not worth the trouble. Something like this should work:

global:
  default_inherits: default

tools:
  default:
    abstract: true
    cores: 2
    mem: 4
    context:
      queue: 'cpu_180'
    rules:
      - id: resource_params_defined
        if: |
          param_dict = job.get_param_values(app)
          param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
        cores:  int(job.get_param_values(app)['__job_resource']['cores'])
        mem: int(job.get_param_values(app)['__job_resource']['mem'])
        context:
          queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
  local_env:
    runner: local_runner
    max_accepted_cores: 1
    params:
      tmp_dir: true
  pbs:
    runner: pbs_drmaa
    params:
      native_specification: "-q {queue} -l select=1:ncpus={cores}:mem={mem}GB"

[hrzolix]
Hi Nuwan, I tried this and I got an error for queue does not exits, apparently it parses directly: -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=4:mem=8GB

[Nuwan Goonasekera]
Aah yes! I forgot that context variables must be constants. In that case, can you try reverting to your original code, but refer to params as entity.params?:

  pbs:
    runner: pbs_drmaa
    params:
      native_specification: "-q {entity.params['queue']} -l select=1:ncpus={cores}:mem={mem}GB"

In the long run, we should probably consider adding support for computed context variables instead. That makes it much nicer I reckon

[hrzolix]
hm again its the same unfortunately : -q {str(job.get_param_values(app)['__job_resource']['queue'])} -l select=1:ncpus=6:mem=4GB
Here is the job_resource_params_conf.xml if it helps

<parameters>
        <param label="Cores" name="cores" type="integer" min="1" max="128" value="" help="Number of processing cores, 'ncpus' value (1-128). Leave blank to use default value(1)." />
        <param label="Queue" name="queue" type="select" value="" help="Depending on job time, choose a queue for your job(cpu=7days, cpu_30=30 days cpu_180=180 days.)">
                <option value="cpu_180">CPU_180 (default)</option>
                <option value="cpu_30">CPU_30</option>
                <option value="cpu">CPU</option>
        </param>
        <param label="Memory" name="mem" type="integer" min="4" max="480" value="" help="Memory size in gigabytes, 'mem' value (4-256). Leave blank to use default value(3.8GB)." />
</parameters>

[Nuwan Goonasekera]
You're right, it doesn't work that way. This is because of the evaluation order of variables. You can use this instead:

global:
  default_inherits: default

tools:
  default:
    abstract: true
    cores: 2
    mem: 4
    env:
      queue: 'cpu_180'
    rules:
      - id: resource_params_defined
        if: |
          param_dict = job.get_param_values(app)
          param_dict.get('__job_resource', {}).get('__job_resource__select') == 'yes'
        cores:  int(job.get_param_values(app)['__job_resource']['cores'])
        mem: int(job.get_param_values(app)['__job_resource']['mem'])
        env:
          queue: "{str(job.get_param_values(app)['__job_resource']['queue'])}"

destinations:
  local_env:
    runner: local_runner
    max_accepted_cores: 1
    params:
      tmp_dir: true
  pbs:
    runner: pbs_drmaa
    params:
      native_specification: "-q {[env['value'] for env in env if env['name'] == 'queue'][0]} -l select=1:ncpus={cores}:mem={mem}GB"

That'll work for now, but the downside is that it passes a needless environment variable to the tool