Invoke maintenance and features

[kuwv]

@bitprophet as you know, I have submitted a few merge requests. These are only a few that I believe should be implemented. Some of which already align with features you plan to implement.

What I wanted to do here was get your thoughts on some of those before continuing.

1 complete composite implementation for namespaces.

• each namespace should set reference to parent
• implement composite traversal
• plan out task / context visitor

2 split out program into three components using dependency injection instead of composition

• workflow api only capability (no cli)
• parser capabilities for invoke
• current program capabilities w/ components injected assembled by a function for current capability

3 implement type checking to resolve all the 'None' logic errors within invoke

4 data proxy breaks SOLID by worrying about both layering dicts and attr access. Im thinking this could be refactored to inherit chain map.

5 implement 'string.Template' for runners

• split kwargs
• optionally format parametrized command

Anyway, just my thoughts.

[bitprophet]

Thanks for posting! Afraid I don't have a ton of time right now to think hard on these but my off the cuff:

1. What's the 'upsell' for this one? Ie what actual user-facing features would modifying the current namespacing enable? I'm down for making the design smarter as it's very much "what made sense early on" but I'd need to figure out how to prioritize it - just spent tons of time on under-the-hood stuff so I'd like to prioritize shiny/visible features for the next while.
2. Same question here, but also same overall reaction, "sounds like a good idea in general".
3. I'd be curious to see what a typechecking overhaul would look like. It's not something I personally have a lot of experience with yet (not zero, but not much) and once again it strikes me as a "unless this removes significant pain which is preventing implementation of a super cool feature, it should be a couple slots down the list" item.
4. Are there specific changes that the current DataProxy setup is preventing? I gather it is conflating a couple related responsibilities but once again, I want to be sold on why it's worth doing earlier instead of later, or at least shown an example of what an upgraded API would look like for folks using Config objects.
5. I am 99% sure there's open tickets about this feature area (there's at least one about allowing list-style invocations like Popen, as an alternative to the vanilla strings used now; and there's probably also one about templating too honestly.) If you dig them up it might be worth seeing what thoughts others have already had here. I agree that this sort of change could be a useful upgrade / enable some nice quality-of-life additions to common patterns (even just across my own projects' tasks.py, and in Invocations.)

[kuwv]

Yeah, I totally understand the time constraints.

1 This should tie directly into your roadmap for "...calling tasks programmatically from other tasks". Implementing parent and child links would allow the namespace to be its own tree to navigate for accessing tasks from other namespaces. So, if we consider if a context is namespace aware it could make task from task easier to implement. But, I'm not sure what your current plan is here.

Maybe look something like this: ctx('docs.sphinx').deploy(var='test) or ctx('..sphinx').deploy(var='test) from a callable context.

2. This one would free up workflow capabilities to be developed separately from CLI and help solve this here: Easy execution of tasks from within Python itself #170. Ideally, this would allow programmatic chaining of tasks to be done through configuration instead. Also, if done correctly the current capability should stay intact.

3. I think it's summed up best that testing allows a codebase to prevent a specific error from occurring. Formal methods, such as type checking, ensures that a codebase is free of errors.

4. Honestly, the only benefit here is just reducing complexity. no biggie if it's not done. But, it does make it harder to improve attr capabilities when maps take up so much space.

5. Yeah, this issue is the second ever created within the repo: Possible alternative, safer run() interpolation #2.

[bitprophet]

Thanks! I was hoping your ideas were inspired by existing issues/missing-features and sounds like that's a "yes!". Appreciate the details.

FWIW I do know about the benefits of typechecking in general, I had meant more of a "Jeff gut check" - say somebody drops a 100% typechecked PR (there's probably already some out there), will I look at the result and go "yikes, this is unrecognizable now"? Would I try hacking on top of it and think "no, the added friction of dealing with typing is still something I don't enjoy"?

But this is very likely just me being old and crotchety 😇 I have been assuming that typing will arrive in at least some of my projects eventually. (Maybe not Paramiko anytime soon. Ugh.)

[kuwv]

Actually, I've read a few issues and ones that interest me are the longest running ones that seem to be preventing some of invoke use cases. I'm pretty much interested in what @neozenith is looking for: #63 (comment) where invoke could be used as a library to create a lower level workflow engines similar to Airflow or Prefect.

I created an PR so you can see what Mypy would like like in invoke: #906