Usd standalone startup library

[Lypsolon]

Is there an existing issue for this?

• I have searched the existing issues.

Please describe the feature you have in mind and explain the current shortcomings.

let's talk about how we can start Usd standalone tools with

• ayon Usd standalone
• Ayon Usd standalone resolver

In a subprocess:

What is the purpose of this system?

1. we want to execute USD standalone operations, e.g., build a stage, edit layers, debug, etc
2. we do not want to use DCC specific Usd libs to save on licenses
3. the core tool startup system should allow for any env variables and any env setup to be used at startup (this would allow us to eg. start UsdView with karma / Arnold loaded in Houdini)

How would you imagine the implementation of the feature?

How to implement.

Core Startup Lib

• store the environment setup at addon "tray_start ?" so that we know what a "clean" Ayon environment for this system looks like
• add ayon_Usd standalone and ayon_usd_standalone resolver to the environment
• use subprocess.popen with env= to define a new subprocess that, by default, does not inherit the parent environment and have it use the "clean" environment where we added our Usd tools (in this step, we can choose if we want to use the parent env or a manipulated clean env)
• now we do one of 2 things: A: we get the path to an Ayon_Usd_plugin and use the main.py, or we take in the startup command

lib funcs()

• start_standalone_tool (a function that runs an Ayon_Usd_Plugin or individual tool in a subprocess)
• get_ayon_info_from_env (a function that extracts all the ayon related infos from the os.env so that we can use this data on USD standalone startup to augment )

Ayon_Usd plugin

(Usd calls them plugins. Do we want an addon because of Ayon? @dee-ynput )
Structure:

• package_name(parent py module folder)
• main.py (should start up the plugin when run directly )
• bin (holding all the binaries )
• lib (holding all the library code that the runtime code might need)
• src (holding all the runtime code)

Are there any labels you wish to add?

• I have added the relevant labels to the enhancement request.

Describe alternatives you've considered:

No response

Additional context:

No response

[BigRoy]

store the environment setup at addon "tray_start ?" so that we know what a "clean" Ayon environment for this system looks like
add ayon_Usd standalone and ayon_usd_standalone resolver to the environment

Why would we need to store it if we're adding ayon usd apparently 'to the environment' dynamically? Sounds like we should just be able to take the environment that is active in launcher at that point in time just fine. Not sure what else would be polluting the env?

use subprocess.popen with env= to define a new subprocess that, by default, does not inherit the parent environment and have it use the "clean" environment where we added our Usd tools (in this step, we can choose if we want to use the parent env or a manipulated clean env)

So, Web Actions seem to start using the "Launcher" to launch actions/applications (which then by design, take from the environment of the launcher). Could we use that for these "launch me a USD application" as well? So that the environment comes from the launcher by design, not from the host DCC or wherever we run from?

now we do one of 2 things: A: we get the path to an Ayon_Usd_plugin and use the main.py, or we take in the startup command

Preferably we just have some CLI that allows us to launch things with the USD envs. ayon launch -include_env usd /path/to/usdview or whatever. Which kind of sounds like ayon_usd env is just a tool group, and hence it could e.g. use a system we apparently already have for tool group to just do ayon launch --tools ayon_usd/1.0 /path/to/usdview (although I'm not sure it currently takes an argument to append additional tools to a specific context, etc.)

---

get_ayon_info_from_env (a function that extracts all the ayon related infos from the os.env so that we can use this data on USD standalone startup to augment )

What is the Ayon related info here?

start_standalone_tool (a function that runs an Ayon_Usd_Plugin or individual tool in a subprocess)

Yes, seems reasonable - as described it's basically just a ayon launch whatever but with AYON's USD environment variables mixed into it.

[Lypsolon]

Why would we need to store it if we're adding ayon usd apparently 'to the environment' dynamically? Sounds like we should just be able to take the environment that is active in launcher at that point in time just fine. Not sure what else would be polluting the env?

the thought is that if we want to start Ayon_Usd_Standalone from, e.g. inside Houdini, we will inherit the Houdini Usd environment. and this would pull, e.g. Arnold etc. we could "clean" the environment or we could build an env mapping from the ground up, but I belive storing the "startup" env would maybe make this easier? if you have another option, I would love to talk about it because storing at startup is a bit of a mid-solution, I think.

So, Web Actions seem to start using the "Launcher" to launch actions/applications (which then by design, take from the environment of the launcher). Could we use that for these "launch me a USD application" as well? So that the environment comes from the launcher by design, not from the host DCC or wherever we run from?

hmm, god question i guess we could see if we can request the launcher to start it for us.
I'm a bit conflicted, tho tbh.
On the one hand: having the launcher doo it and asking it to launch it would be very clean.
On the other hand:
Having the launcher launch would limit us to the launcher well (I'm not sure how much freedom we have there), so I think ideas like running this Python script with Ayon USD would be out of the question, and we could only run applications.
maybe @iLLiCiTiT can expand on what the pros and cons would be when using the launcher for this.
Here are some thought points

• can we start an "application"? This could be about anything: maybe an exe, maybe an .sh, maybe a Python script that starts an ui
• Can we run "just" a script? I have a Python script. This will do whatever magic in the USD scene, and I can start it with ayon Usd. I imagen to build the ayon_usd_stanlone.start() func so that you could say path="/my/cool/py_script," and you could then also just use it like this.

PS: the reason why I'm thinking about that is because I think we will end up with many situations where we don't want an "application" in that sense but just some process that does something using the ayon_usd env
, or should we maybe separate those 2 concepts? Into, e.g., an application starter and an ayon_standalone runner <- This would then use a background py_app that has everything loaded, and you can request it to execute things (maybe using gRPC / iRPC to talk to it so it's very detached ? ) <- this sounds a bit to complicated to me tbh

Preferably we just have some CLI that allows us to launch things with the USD envs. ayon launch -include_env usd /path/to/usdview or whatever. This kind of sounds like ayon_usd env is just a tool group, and hence it could e.g. use a system we already have for tool group to just do ayon launch --tools ayon_usd/1.0 /path/to/usdview (although I'm not sure it currently takes an argument to append additional tools to a specific context, etc.)

hmm, interesting thought.
I think launching tools completely externally and with as tool groups would be an idea, but I might miss the point of this issue, at least if I understand you right. so the difference I see is as follows: this startup system is more concerned about how we can start essentially arbitrary code to do things for us in an ayon_usd_standalone environment, and I believe that's not the same as starting a tool like Maya Houdini standalone. the reason why I think this is I believe there might be a value to having the option to say: I'm in a Python script somewhere, and I want some Usd related code to be run a bit like I'm doing gRPC but without having to have I an external system running.
but i will say the issue has beewen written with the idea in mind to run tools mainly. so maybe there is a point to see if this has become an mix off ideas instead of an abstract low lvl lib function.

What is the Ayon related info here?

this would be all the env keys the launcher sets. so that if you start some tool or script, it would behave like you started it from the scene you are in. so that you can access Ayon_Project_Name and so on.

Yes, seems reasonable - as described it's basically just a ayon launch whatever but with AYON's USD environment variables mixed into it.

exactly, the target is to have a function that lets you run what you want like you're just calling a function. in the best case, I would also like to be able to say run(/path/to/ayon_usd/lib/lib.py -func -args -kwargs) so that you could kinda call functions in the ayon_usd_env like they are local to what you're doing. im not really sure if i like this idea to much but it would be kinda nice to have some freedom there.

[Lypsolon]

It's just a super quick experiment.
But this would allow running a function in a Process and passing it in an environment.
I don't know if ideas like this would satisfy the requirements, but we are here to see what those requirements are.

def _run_with_env(func, env, queue, *args, **kwargs):
    os.environ.update(env)
    result = func(*args, **kwargs)
    queue.put(result)


def run_external(func, env, *args, **kwargs):
    queue = multiprocessing.Queue()
    process = multiprocessing.Process(
        target=_run_with_env, args=(func, env, queue) + args, kwargs=kwargs
    )
    process.start()
    process.join()
    return queue.get()

[Lypsolon]

In this comment, I will write out what I think the requirements could be; maybe @BigRoy, @antirotor and @dee-ynput have something to say about it.

I start by saying this "system" should replace this function initialize_environment, the idea behind the function was to have the option to call a setup function from within python and then be able to interact with it. But this doesn't work for reasons like LD_LIBARY loading only existing at startup. (PS: the code in the message above would not be able to obtain the LD_LIBARY_PATH correctly, but it serves as an example of what could be nice to have)

Requirements

• allow running any function in an ayon_usd_standalone env (in most cases, this should probably be an entry function to avoid creating multiple processes, but that would be up to the implementer to decide)
• create some syntax sugar to pull in the ayon env variables if wanted so that a tool or function that relies on them will get them from where the tool or function was started and not from ayon_launcher as those 2 might not be the same (e.g., Ayon_project env variable)
• Automatically set the environment so the ayon_usd_standalone environment is loaded, not the environment inside a DCC like Houdini.

That said, I see those comments miss the point of the original issue. The issue focuses more on starting tools like UsdView and less on arbitrary code execution in a separate process.