Command LLM #186

whilefoo · 2024-11-03T21:41:48Z

Resolves #166

whilefoo · 2024-11-03T22:02:51Z

I have a dilemma about the command interface.
I added a command property in the input object that has command name and command parameters. The name and parameters are given by the LLM when the user tags the bot.

I'm guessing we still wanna keep the /command something 10 so I'm thinking between

letting the LLM also infer this way of calling a command but it might not always be successful because there is less context than a user using natural language to express what they want
leaving the command input empty and the plugin itself has to decode the command from the comment payload like it has done now
removing this way of calling the command but if the LLM can't figure out how to call a command, you basically can't do anything

First option seems better because there's only way to receive commands and it's already parsed by the LLM so the plugin doesn't need to do any parsing, but might be inaccurate at times
Second option makes sure that the direct way of calling a command is robust and not prone to hallucinations by LLM

@0x4007 @gentlementlegen

gentlementlegen · 2024-11-04T02:35:30Z

Second option seems proper to me because it is what we use for all the plugins so the parser is ready without any change, making maintenance easier.

0x4007 · 2024-11-04T11:02:20Z

@sshivaditya2019 perhaps you can offer some pointers related to function calling

whilefoo · 2024-11-05T16:43:36Z

I will leave it for now since every plugin already supports parsing, we can add later if we want

whilefoo · 2024-11-07T22:44:04Z

tests/commands.test.ts

+const dispatchWorkflow = jest.fn();
+jest.mock("../src/github/utils/workflow-dispatch", () => ({
+  //...(jest.requireActual("../src/github/utils/workflow-dispatch") as object),
+  getDefaultBranch: async () => "main",
+  dispatchWorkflow: dispatchWorkflow,
+}));


@gentlementlegen I've been fighting with Jest for hours...do you might know why is mocking working only outside the test function. In sdk.test.ts I put it in the test function and it worked but not here.

Also getDefaultBranch returns undefined when it gets called by issueCommentCreated and I don't understand why

We have moved to ESM so I'd suggest trying unstable_mock instead. As such, imports should happen after the mock, since it is hoisted.
https://jestjs.io/docs/ecmascript-modules

(And I second you, it's a pain in the ass to deal with all this CJS ESM compatibility, I am having nightmares within text-conversation-rewards`)

Update: it broke in the SDK when I moved it too. It is usually easier to fake network calls than the actual package.

I'm not even sure if kernel tests are running with CJS or ESM because if I add experimental vm modules and unstable mock, it says that import must be used

even though the module issue-comment-created is dynamically imported in the test function, it's still importing original dispatchWorkflow and not mocked

This can happen if the module you are trying to lock is previously imported by another module first, in such case the mock won't work. I've seen such scenario within the SDK.

It seems the issue was that jest caches imported modules and I imported issue-comment-created in 3 test cases but only mocked dispatchWorkflow in the 2nd test case, but jest cached the first import so the mock didn't work. I had to use jest.resetModules() to reset the cache.
Thanks for the help!

Glad you figured it out. Jest has been giving me headaches a lot lately particularly when ESM came into play.

whilefoo · 2024-11-10T10:53:20Z

@gentlementlegen I've just realised that separating SDK makes development more difficult, I made changes to Manifest but now it's in SDK package so I need to use bun link and I have to build the SDK every time I make a change.
Also decoding manifest schema gives some TS errors about types not matching even though Typebox version is the same in kernel and SDK

gentlementlegen · 2024-11-10T11:21:05Z

@whilefoo The types not matching was there before somehow, not sure about the cause.

What I usually do is to build with a --watch so I don't have to think of it. If that is too burdensome, we can consider merging it back, but it has to be perfectly separated otherwise we will have the circular reference again which will break ncc compilation.

whilefoo · 2024-11-10T22:21:02Z

QA: ubiquibot-whilefoo-testing/testing#7

gentlementlegen · 2024-11-10T22:34:37Z

src/github/handlers/issue-comment-created.ts

+  }
+
+  const toolCalls = response.choices[0].message.tool_calls;
+  if (!toolCalls || toolCalls.length === 0) {


Suggested change

if (!toolCalls || toolCalls.length === 0) {

if (!toolCalls?.length) {

gentlementlegen · 2024-11-10T22:43:55Z

@whilefoo Looks cool. I wanted to try some commands, I guess the bot was not able to execute them which is ok but I got no feedback on them: ubiquibot-whilefoo-testing/testing#7 (comment) should there be some error message displayed in such case?

whilefoo · 2024-11-11T09:05:28Z

@whilefoo Looks cool. I wanted to try some commands, I guess the bot was not able to execute them which is ok but I got no feedback on them: ubiquibot-whilefoo-testing/testing#7 (comment) should there be some error message displayed in such case?

I was running it locally on my laptop and I think it went to sleep so it stopped replying

0x4007 · 2024-11-12T01:10:55Z

QA: ubiquibot-whilefoo-testing/testing#7

This is amazing for demos to see let's merge it if you think it's in a good spot to.

Really looking forward to this natural language interface either on the copilot chat window on GitHub web desktop / iOS and/or our telegram bot.

0x4007 · 2024-11-12T01:11:44Z

package.json

0x4007 · 2024-11-12T01:15:19Z

src/github/handlers/issue-comment-created.ts

-  const body = context.payload.comment.body.trim();
-  if (/^\/help$/.test(body)) {
+  const body = context.payload.comment.body.trim().toLowerCase();
+  if (body.startsWith(`@ubiquityos`)) {


I wonder if we should offer a short hand syntax.

As I understand @UbiquityOS wouldn't automatically populate on the GitHub UI.

Mixed thoughts on this idea, but it might be more ergonomic to use with something like @U or @os

Also this next idea could be out of scope but wondering if we should intercept all comments and run commands on behalf. Would be perfect if we could handle those assigns for the newcomers asking to work on tasks for example.

I wonder if we should offer a short hand syntax.

As I understand @UbiquityOS wouldn't automatically populate on the GitHub UI.

Mixed thoughts on this idea, but it might be more ergonomic to use with something like @U or @os

Yes, it's a bit cumbersome to always type out @UbiquityOS so a short hand tag would be great but we will be tagging another person :D

Also this next idea could be out of scope but wondering if we should intercept all comments and run commands on behalf. Would be perfect if we could handle those assigns for the newcomers asking to work on tasks for example.

Do you mean without the tag? For example if the users says "how can I start this task?" the router should run the start command?
It sounds good in theory but in practice it might trigger it even when not intended to and will consume OpenAI API a lot which will result in higher cost.

It might be interesting to have some type of local logic (perhaps non LLM) to determine if the comment is likely asking for some type of action. Accuracy can be pretty low and in theory it could still cut quite a bit of costs.

Also with mini models costs might already be cheap enough for it to be feasible. We would need to run projections I suppose.

I agree with @whilefoo that tagging another person is not good. Maybe we should create a @UbiquityOS account to get the auto-completion when typing the name.

0x4007 · 2024-11-12T01:17:48Z

src/github/handlers/issue-comment-created.ts

+  - **Tagged Natural Language**: Interpret the "comment" field provided in JSON. Users will mention you with "@UbiquityOS", followed by their request. Infer the intended command and parameters based on the "comment" content.
+
+- **Action**: Map the user's intent to one of your available functions. When responding, use the "author", "repositoryOwner", "repositoryName", and "issueNumber" fields as context if relevant.
+`,


This is a high quality prompt good job

whilefoo · 2024-11-12T21:43:47Z

This is amazing for demos to see let's merge it if you think it's in a good spot to.

I have to modify and make PR for each plugin to accommodate for the new interface so only then we can merge it

feat: initial impl

c5b77f3

feat: more context and tests

e90a93d

whilefoo commented Nov 7, 2024

View reviewed changes

whilefoo added 5 commits November 9, 2024 19:50

fix: tests

164f247

feat: merge

6d24748

fix: tests

b58f0f0

fix: remove test command

fcb4078

chore: merge

aed0974

fix: imports

01854e7

Keyrxng mentioned this pull request Nov 10, 2024

Properly set the metadata value of LogReturn ubiquity-os/ubiquity-os-logger#46

Open

whilefoo marked this pull request as ready for review November 10, 2024 22:19

whilefoo requested a review from gentlementlegen November 10, 2024 22:21

feat: additional properties and required

639e4e3

gentlementlegen reviewed Nov 10, 2024

View reviewed changes

0x4007 approved these changes Nov 12, 2024

View reviewed changes

gentlementlegen approved these changes Nov 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Command LLM #186

Command LLM #186

whilefoo commented Nov 3, 2024

whilefoo commented Nov 3, 2024

gentlementlegen commented Nov 4, 2024

0x4007 commented Nov 4, 2024

whilefoo commented Nov 5, 2024

whilefoo Nov 7, 2024

gentlementlegen Nov 8, 2024 •

edited

Loading

gentlementlegen Nov 8, 2024

whilefoo Nov 8, 2024

whilefoo Nov 8, 2024

gentlementlegen Nov 9, 2024

whilefoo Nov 9, 2024 •

edited

Loading

gentlementlegen Nov 10, 2024

whilefoo commented Nov 10, 2024

gentlementlegen commented Nov 10, 2024

whilefoo commented Nov 10, 2024

gentlementlegen Nov 10, 2024

gentlementlegen commented Nov 10, 2024

whilefoo commented Nov 11, 2024

0x4007 commented Nov 12, 2024 •

edited

Loading

0x4007 Nov 12, 2024

0x4007 Nov 12, 2024

whilefoo Nov 12, 2024 •

edited

Loading

0x4007 Nov 13, 2024

gentlementlegen Nov 13, 2024

0x4007 Nov 12, 2024

whilefoo commented Nov 12, 2024 •

edited

Loading

	if (!toolCalls \|\| toolCalls.length === 0) {
	if (!toolCalls?.length) {

Command LLM #186

Are you sure you want to change the base?

Command LLM #186

Conversation

whilefoo commented Nov 3, 2024

whilefoo commented Nov 3, 2024

gentlementlegen commented Nov 4, 2024

0x4007 commented Nov 4, 2024

whilefoo commented Nov 5, 2024

Choose a reason for hiding this comment

gentlementlegen Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whilefoo Nov 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whilefoo commented Nov 10, 2024

gentlementlegen commented Nov 10, 2024

whilefoo commented Nov 10, 2024

Choose a reason for hiding this comment

gentlementlegen commented Nov 10, 2024

whilefoo commented Nov 11, 2024

0x4007 commented Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whilefoo Nov 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

whilefoo commented Nov 12, 2024 • edited Loading

gentlementlegen Nov 8, 2024 •

edited

Loading

whilefoo Nov 9, 2024 •

edited

Loading

0x4007 commented Nov 12, 2024 •

edited

Loading

whilefoo Nov 12, 2024 •

edited

Loading

whilefoo commented Nov 12, 2024 •

edited

Loading