Looper process streamlining

[nsheff]

A collection of problems I ran into when trying to use looper for a project:

• the pipeline interface init by looper init-piface shouldn't be named "generic" #417
• the pipeline interface init function specifies an output schema, but doesn't initialize one. #418
• docs for write_sample_yaml don't say how to access it (it's {sample.sample_yaml_path}) #421
• plugins should live in plugins.py #419
• looper is forcing the pipeline file to be executable, even if the command template doesn't execute it #420
• Add tab completion to Looper #422
• there was no clear way to pull from pephub into Python (I guess pephubclient can do this? Possibly just need docs)
• I have lumpn, which is number of commands to batch into one job. Another possibility would be the inverse; I want to batch my samples into m jobs. So, this would just be equivalent to a --lumpn using the number of samples divided by m. Is this worth a new option? I want m jobs, I have s samples; use --lumpn s/m. 100 samples, 8 jobs, so --lumpn 12. 80 samples, 4 jobs, --lumpn 20. (Proposed revision to lump functionality #415)
• I would like to update my PEP with information about a sample: Using pipestat to report results to PEPhub pipestat#125
• looper variable namespaces should be updated to contain what is in the .looper config file #423
• Right now looper.pep_config isn't working if the path given is a pephub path. #424
• how do to parallel-process files with looper locally -- lump them together, this will group several commands in one like: #498
  { echo "str1"; echo "str2"; echo "str3"; } & -- this could become a parallel lump mode.

[donaldcampbelljr]

looper variable namespaces need to be updated; in particular, does the looper namespace get what's in the .looper config file now? -> Q: looper namespace was relative to PEP is it now the looper config? Think everythign in the looper config is in the looper namespace. See #423

looper.pep_config -> command_template -> it works if passing pep that is local, if using a registry path it doesn't appear to work

how do to parallel-process files with looper locally -> originally a divvy idea.
issue: 100 files -> divvy submits to cluster no problem, if local they will run serial. Could run in background process using ampersand. in command shell script with &. So can we lump 100 samples (in 10 background processes), new divvy template to accomplish that.

[donaldcampbelljr]

there was no clear way to pull from pephub into Python (I guess pephubclient can do this? Possibly just need docs)

For using looper with pephub looks like there is documentation:
https://looper.databio.org/en/dev/hello-world-pephub/

However, it does appear that our pephubclient api docs could be expanded. There is currently only a light readme.

[donaldcampbelljr]

Because we've solved the majority of the issues here and the remaining two have child issues tracking them, I will close this parent issue.