enable asynchronous Arkouda client requests #2008
Conversation
hokiegeek2
requested review from
Ethan-DeBandi99,
stress-tess,
joshmarshall1 and
jaketrookman
|
example of async connect w/ delay: arkouda-async-connect.mp4example of async ones request w/ delay: arkouda-asynchronous-ones-request.mp4 |
|
just have to fix flake8 errors:
|
|
all tests are now working: |
| <a id="async_client"></a> | ||
| ## Asynchronous Client | ||
|
|
||
| ### Background | ||
|
|
||
| Arkouda has an alpha capability for enabling asynchronous client-server communications that provides feedback to users that a request has been submitted and is being processed within the Arkouda server. The initial asynchronous request capability supports multiuser Arkouda use cases where users may experience delays when the Arkouda server is processing requests by 1..n other users. | ||
|
|
||
| ### Configuration | ||
|
|
||
| To enable asynchronous client communications, set the ARKOUDA_REQUEST_MODE environment variable as follows: | ||
|
|
||
| ``` | ||
| export ARKOUDA_REQUEST_MODE=ASYNC | ||
| ``` | ||
|
|
||
| ### Exiting the Python shell | ||
|
|
||
| As of 01022023, exiting the Python shell in ASYNC request mode requires the following command: | ||
|
|
||
| ``` | ||
| ak.exit() | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Is this information something we would want in GitHub pages? I know that the updates to our documentation are still in progress, but I think this may be something beneficial to broadcast to a larger audience as its deployed.
Not 100% sure what the best place to put it would be, but I am thinking the top level for now.
There was a problem hiding this comment.
@Ethan-DeBandi99 yeah, good point, should be at top-level
| loop = get_event_loop() | ||
|
|
||
| def asyncEligible(cmd: str) -> bool: | ||
| return cmd not in {'delete', 'connect', 'getconfig'} |
There was a problem hiding this comment.
Moving forward, I am assuming we will have more commands if not all supported. Would we need to list every command here or would this be removed if we are supporting all commands?
There was a problem hiding this comment.
I might be wrong but I think this is saying every command except delete, connect, and getconfig is async eligible (i.e. will return True)
This is supported by the fact that one of the mp4s has asynchronous ak.ones which is not in this set of commands
There was a problem hiding this comment.
@Ethan-DeBandi99 ah, sorry, as @pierce314159 this checks to see if a method can be run in async mode. The reasons for the three are as follows:
- delete--since delete invokes del on pdarray, this cannot be run async b/c del is also invoked on Arkouda shell exit() and fails due to the resulting asyncio failing due to modules being deleted
- connect--this is because Python client connect method is itself async-enabled
- getconfig--this is invoked within the async Python client connect method, which, again, is itself async-enabled
|
Here's an example of the "fire-and-forget" approach where the ak.connect() method is invoked and returns immediately. Once the request is actually returned, the user is prompted with the return message. This approach is different and handy in that the user can submit a request, continue working on something else, and then return to the workflow that spawned the request as shown below: arkouda-connect-fire-and-forget.mp4 |
Hey guys,
This is a draft PR that covers the initial proof-of-concept for enabling asynchronous client-server comms in Arkouda utilizing asyncio.
There is currently an issue w/ pytest failures that I am investigating, but the code itself is working.