Enable Tools to Define Execution Model #1023

cescoffier · 2024-10-30T18:28:48Z

This commit introduces support for tools to define their execution model with annotations such as @Blocking, @nonblocking, and @RunOnVirtualThread, or to rely on the method signature to specify the execution model.

Execution constraints are enforced based on the caller thread; for instance, blocking calls are disallowed on the event loop.

For reactive AiService method (streaming), a detection is done at build time to determine if the execution can be done on the event loop or need to be switched to a worker thread. The detection checks the execution model of the various tool methods the AI service method can invoke.

Write documentation
Update sample to show blocking tools when streaming is used
Make sure the duplicated context is propagated

andreadimaio · 2024-10-30T22:36:30Z

I was curious to test these changes in my workspace, but when I use the @RunOnVirtualThread annotation on a @Tool method, I encounter an exception during application startup. However, this issue doesn't occur when I run your tests in the core module, where "virtual thread" methods work fine. It is possible that this is an issue on my side 🤔

Moving on, I would like to ask about a piece of code used within the StreamingChatLanguageModel in watsonx.ai. This code can be removed once this PR is merged, right?

geoand · 2024-10-31T06:22:18Z

I was curious to test these changes in my workspace, but when I use the @RunOnVirtualThread annotation on a @tool method, I encounter an exception during application startup. However, this issue doesn't occur when I run your tests in the core module, where "virtual thread" methods work fine. It is possible that this is an issue on my side 🤔

Can you perhaps add a test into this PR that surfaces the problem?

geoand

Awesome stuff!

I only added a few minor comments, but we should also see what is causing the problem that @andreadimaio mentions

core/deployment/src/main/java/io/quarkiverse/langchain4j/deployment/AiServicesProcessor.java

...eployment/src/main/java/io/quarkiverse/langchain4j/deployment/items/ToolMethodBuildItem.java

core/deployment/src/test/java/io/quarkiverse/langchain4j/test/tools/ToolExecutionModelTest.java

cescoffier · 2024-10-31T07:24:21Z

It should have been a draft - sorry.

andreadimaio · 2024-10-31T08:28:30Z

Can you perhaps add a test into this PR that surfaces the problem?

Yes, I could create something in the integration-tests folder. Let me know if you'd like me to proceed, or if you'd prefer to add the test later.

geoand · 2024-10-31T08:37:43Z

Up to @cescoffier

cescoffier · 2024-10-31T08:40:43Z

@andreadimaio would be great!

I know a few cases that should not have been working but were working that are now rejected (calling an imperative method from the event loop without the @NonBlocking annotation). Still, most of these cases are detected (in the case of streaming), and a switch to a worker thread is automatically done. Because of this, I don't think the emitOn you add is required anymore. But I would need to see the full stack trace.

Now there is still a bit of work do do, as I "forgot" to implement context propagation. It should not take long, but my next coding session will be next week :-(

andreadimaio · 2024-10-31T11:34:43Z

Here you can find the test cescoffier#1.
When I run the test, I get the exception:

The @Blocking, @NonBlocking and @RunOnVirtualThread annotations may only be used on "entrypoint" methods (methods invoked by various frameworks in Quarkus)
Using the @Blocking, @NonBlocking and @RunOnVirtualThread annotations on methods that can only be invoked by application code is invalid
        at io.quarkus.deployment.execannotations.ExecutionModelAnnotationsProcessor.check(ExecutionModelAnnotationsProcessor.java:55)
        at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        at org.jboss.threads.JBossThread.run(JBossThread.java:483)

If I remove the annotations: @Blocking, @NonBlocking and @RunOnVirtualThread, the tests are green.

cescoffier · 2024-10-31T11:44:33Z

Oh oh… that may require a change in Quarkus. I’m wondering why it works in core…

…

On Thu 31 Oct 2024 at 12:35, Andrea Di Maio ***@***.***> wrote: Here you can find the test cescoffier#1 <cescoffier#1>. When I run the test, I get the exception: The @Blocking, @nonblocking and @RunOnVirtualThread annotations may only be used on "entrypoint" methods (methods invoked by various frameworks in Quarkus) Using the @Blocking, @nonblocking and @RunOnVirtualThread annotations on methods that can only be invoked by application code is invalid at io.quarkus.deployment.execannotations.ExecutionModelAnnotationsProcessor.check(ExecutionModelAnnotationsProcessor.java:55) at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733) at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856) at io.quarkus.builder.BuildContext.run(BuildContext.java:256) at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521) at java.base/java.lang.Thread.run(Thread.java:1583) at org.jboss.threads.JBossThread.run(JBossThread.java:483) If I remove the annotations: @Blocking, @nonblocking and @RunOnVirtualThread, the tests are green. — Reply to this email directly, view it on GitHub <#1023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADCG7OP76PQBJGI4YBOQQ3Z6IIWVAVCNFSM6AAAAABQ4XUUEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZGYZTSMJVG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

cescoffier · 2024-10-31T11:47:11Z

There is a package name check… but I should be able to emit the right build item to handle this. On Thu 31 Oct 2024 at 12:44, clement escoffier ***@***.***> wrote:

…

Oh oh… that may require a change in Quarkus. I’m wondering why it works in core… On Thu 31 Oct 2024 at 12:35, Andrea Di Maio ***@***.***> wrote: > Here you can find the test cescoffier#1 > <cescoffier#1>. > When I run the test, I get the exception: > > The @Blocking, @nonblocking and @RunOnVirtualThread annotations may only be used on "entrypoint" methods (methods invoked by various frameworks in Quarkus) > Using the @Blocking, @nonblocking and @RunOnVirtualThread annotations on methods that can only be invoked by application code is invalid > at io.quarkus.deployment.execannotations.ExecutionModelAnnotationsProcessor.check(ExecutionModelAnnotationsProcessor.java:55) > at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733) > at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:856) > at io.quarkus.builder.BuildContext.run(BuildContext.java:256) > at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) > at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516) > at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495) > at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521) > at java.base/java.lang.Thread.run(Thread.java:1583) > at org.jboss.threads.JBossThread.run(JBossThread.java:483) > > If I remove the annotations: @Blocking, @nonblocking and > @RunOnVirtualThread, the tests are green. > > — > Reply to this email directly, view it on GitHub > <#1023 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADCG7OP76PQBJGI4YBOQQ3Z6IIWVAVCNFSM6AAAAABQ4XUUEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBZGYZTSMJVG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

cescoffier · 2024-10-31T13:55:27Z

@andreadimaio I think I fixed it.

andreadimaio · 2024-10-31T14:25:06Z

@andreadimaio I think I fixed it.

Yes, it works!

andreadimaio · 2024-10-31T14:28:00Z

About this code? I think it can be removed and managed with annotations.

andreadimaio · 2024-10-31T15:46:05Z

Uhm... if I remove these lines from the WatsonxChatModel class:

if (tools != null) {
   // Today Langchain4j doesn't allow to use the async operation with tools.
   // One idea might be to give to the developer the possibility to use the VirtualThread.
   mutiny.emitOn(Infrastructure.getDefaultWorkerPool());
}

And I try to call a tool method annotated with @Blocking:

@ToolBox(Calculator.class)
public Multi<String> message(@UserMessage String message);

@ApplicationScoped
public class Calculator {

    @Inject
    EmbeddingModel model;

    @Tool
    @Blocking
    public int sum(int a, int b) {
        model.embed("test");
        return a + b;
    }
}

I am getting java.lang.IllegalStateException: Cannot execute blocking tools on event loop thread.
Looking at the code, the TokenStream is executed in a worker-pool, but the exception is raised in any case.

andreadimaio · 2024-10-31T15:50:09Z

QuarkusToolExecutor #58

case BLOCKING:
    if (io.vertx.core.Context.isOnEventLoopThread()) {
        throw new IllegalStateException("Cannot execute blocking tools on event loop thread");
    }

I think the problem at this point is the Multi created in the WatsonxChatModel.

cescoffier · 2024-10-31T16:38:45Z

It should have done the switch automatically. I will add a few tests.

…

On Thu 31 Oct 2024 at 16:50, Andrea Di Maio ***@***.***> wrote: case BLOCKING: if (io.vertx.core.Context.isOnEventLoopThread()) { throw new IllegalStateException("Cannot execute blocking tools on event loop thread"); } — Reply to this email directly, view it on GitHub <#1023 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADCG7MYNHP6J2GGUJ3KRHTZ6JGUTAVCNFSM6AAAAABQ4XUUEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQGIZDENRQG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

andreadimaio · 2024-10-31T16:42:27Z

Yes, if the switch is the line Infrastructure.getDefaultWorkerPool().execute(tokenStream::start); it is executed, but the exception is raised in any case

cescoffier · 2024-10-31T17:38:37Z

Interesting, can I reproduce it with your test?

cescoffier · 2024-10-31T18:57:39Z

Ok, I know why it happens - I need to think about the proper way to fix it.

andreadimaio · 2024-10-31T19:10:04Z

Interesting, can I reproduce it with your test?

Not yet, but I can try to add it.

Ok, I know why it happens - I need to think about the proper way to fix it.

What's happening?

cescoffier · 2024-10-31T19:35:35Z

Basically, each item is emitted on the event loop. In my tests, I simplified this a bit too much by sending all the items synchronously on subscription (on the event loop).

The current switch only happens at subscription time. So it works for my tests... but only in my tests.

geoand · 2024-11-05T11:58:20Z

@cescoffier I'd like to include this in the next release, but if it still needs plenty of work, we can have in a later one

cescoffier · 2024-11-05T12:01:03Z

I have a plan to fix the remaining issue.

When do you want to cut the release?

geoand · 2024-11-05T12:29:32Z

This week was the plan, but it's not set in stone

cescoffier · 2024-11-05T12:40:40Z

I just got 1 free hour this afternoon, that should be enough

geoand · 2024-11-05T12:46:56Z

💪

@Blocking

This commit introduces support for tools to define their execution model with annotations such as @Blocking, @nonblocking, and @RunOnVirtualThread, or to rely on the method signature to specify the execution model. Execution constraints are enforced based on the caller thread: for instance, blocking calls are disallowed on the event loop. For reactive AiService method (streaming), a detection is done at build time to determine if the execution can be done on the event loop or need to be switched to a worker thread. The detection checks the execution model of the various tool methods the AI service method can invoke.

…and @RunOnVirtualThread

cescoffier · 2024-11-05T15:41:20Z

@geoand how do we modify samples? I need to bump the version of quarkus langchain4j to demonstrate the feature. If I switch to 999-SNAPSHOT would it breaks the world?

…es can use blocking tools

cescoffier · 2024-11-05T15:42:04Z

(@geoand well... let's try)

geoand · 2024-11-06T07:05:28Z

It seems like some of the tests fail

cescoffier · 2024-11-06T07:10:57Z

@geoand @andreadimaio Seems like Watsonx is doing something that now blocks the event loop - I'm having a look.

cescoffier · 2024-11-06T08:01:03Z

Hopefully, it should be better now. The issue was that the initial call was not on a Vert.x context (neither event loop nor worker), which broke the switch (as I had nowhere to switch to).

geoand · 2024-11-06T08:18:23Z

All green!

andreadimaio · 2024-11-06T08:32:43Z

I could do some testing in my local environment if you think it would be useful. But I can do that in a bit.

… done from a Vert.x context

andreadimaio · 2024-11-06T09:19:04Z

It works!! 🚀
@cescoffier I have opened a PR to merge some changes in watsonx, I have removed:

if (tools != null) {
   // Today Langchain4j doesn't allow to use the async operation with tools.
   // One idea might be to give to the developer the possibility to use the VirtualThread.
   mutiny.emitOn(Infrastructure.getDefaultWorkerPool());
}

If you don't want to add this change here, I can open another PR.

Removed emitOn on default worker pool for watsonx.ai

cescoffier requested a review from a team as a code owner October 30, 2024 18:28

cescoffier requested a review from geoand October 30, 2024 18:33

geoand reviewed Oct 31, 2024

View reviewed changes

cescoffier marked this pull request as draft October 31, 2024 07:24

Allows methods annotated with @tool to used @Blocking / @nonblocking …

b33c558

…and @RunOnVirtualThread

cescoffier force-pushed the tools-execution-model branch from 56e1d7c to b33c558 Compare November 5, 2024 15:40

cescoffier marked this pull request as ready for review November 5, 2024 15:40

Modify the fraud detection sample to demonstrate how streamed respons…

00cd5ef

…es can use blocking tools

geoand approved these changes Nov 5, 2024

View reviewed changes

Handle thread switch for tools execution when the initial call in not…

bc7d6c0

… done from a Vert.x context

cescoffier force-pushed the tools-execution-model branch from 5f1eea1 to bc7d6c0 Compare November 6, 2024 08:55

Removed emitOn on default worker pool for watsonx.ai

612241f

Merge pull request #2 from andreadimaio/tools-execution-model

790932f

Removed emitOn on default worker pool for watsonx.ai

cescoffier merged commit b9d826a into quarkiverse:main Nov 6, 2024
60 checks passed

cescoffier deleted the tools-execution-model branch November 18, 2024 12:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable Tools to Define Execution Model #1023

Enable Tools to Define Execution Model #1023

cescoffier commented Oct 30, 2024 •

edited

Loading

andreadimaio commented Oct 30, 2024 •

edited

Loading

geoand commented Oct 31, 2024

geoand left a comment

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 •

edited

Loading

geoand commented Oct 31, 2024

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

cescoffier commented Oct 31, 2024 via email

cescoffier commented Oct 31, 2024 via email

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 •

edited

Loading

cescoffier commented Oct 31, 2024 via email

andreadimaio commented Oct 31, 2024

cescoffier commented Oct 31, 2024

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 •

edited

Loading

cescoffier commented Oct 31, 2024 •

edited

Loading

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 6, 2024

cescoffier commented Nov 6, 2024

cescoffier commented Nov 6, 2024

geoand commented Nov 6, 2024

andreadimaio commented Nov 6, 2024

andreadimaio commented Nov 6, 2024

Enable Tools to Define Execution Model #1023

Enable Tools to Define Execution Model #1023

Conversation

cescoffier commented Oct 30, 2024 • edited Loading

andreadimaio commented Oct 30, 2024 • edited Loading

geoand commented Oct 31, 2024

geoand left a comment

Choose a reason for hiding this comment

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 • edited Loading

geoand commented Oct 31, 2024

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

cescoffier commented Oct 31, 2024 via email

cescoffier commented Oct 31, 2024 via email

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 • edited Loading

cescoffier commented Oct 31, 2024 via email

andreadimaio commented Oct 31, 2024

cescoffier commented Oct 31, 2024

cescoffier commented Oct 31, 2024

andreadimaio commented Oct 31, 2024 • edited Loading

cescoffier commented Oct 31, 2024 • edited Loading

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 5, 2024

cescoffier commented Nov 5, 2024

cescoffier commented Nov 5, 2024

geoand commented Nov 6, 2024

cescoffier commented Nov 6, 2024

cescoffier commented Nov 6, 2024

geoand commented Nov 6, 2024

andreadimaio commented Nov 6, 2024

andreadimaio commented Nov 6, 2024

cescoffier commented Oct 30, 2024 •

edited

Loading

andreadimaio commented Oct 30, 2024 •

edited

Loading

andreadimaio commented Oct 31, 2024 •

edited

Loading

andreadimaio commented Oct 31, 2024 •

edited

Loading

andreadimaio commented Oct 31, 2024 •

edited

Loading

cescoffier commented Oct 31, 2024 •

edited

Loading