Refactor Assistant builder: use Vaults, use Context #6768

Fraggle · 2024-08-14T11:42:07Z

Description

This PR is about using Vaults (& DataSourceViews) in the assistant builder flow.

The main changes to make it work with DataSourceView:

AssistantBuilderDataSourceModal store configurations mapping by dataSourceView.sId instead of dataSource.sId
Use LightContentNode (returned by Vault's api) instead of ContentNode (old api).
Use DataSourceViewType everywhere in builder and in the configurations.
Use useVaultDataSourceViewContent instead of useConnectorPermissions.
Refactored global assistants to use datasourceview

Note: it will break using "@dust" locally.

Risk

Break assistant builder

Deploy Plan

Deploy front-edge, test, deploy front

front/components/DataSourceResourceSelectorTree.tsx

front/components/assistant_builder/AssistantBuilderDataSourceModal.tsx

front/hooks/useParentResourcesById.ts

front/lib/resources/data_source_resource.ts

front/lib/resources/data_source_view_resource.ts

front/lib/resources/vault_resource.ts

Fraggle · 2024-08-19T14:23:37Z

@flvndvd feedback adressed, feel free to re-review.

front/components/DataSourceResourceSelectorTree.tsx

front/lib/swr.ts

front/components/DataSourceResourceSelectorTree.tsx

front/components/ConnectorPermissionsTree.tsx

flvndvd · 2024-08-22T12:54:47Z

front/components/assistant/AssistantDetails.tsx

@@ -357,7 +357,7 @@ function DataSourcesSection({
    <div className="flex flex-col gap-1">
      <ManagedDataSourceDocumentModal


Note to self, move to DataSourceViewDocumentModal once this PR is merged.

front/components/assistant_builder/AssistantBuilder.tsx

front/components/assistant_builder/AssistantBuilderContext.tsx

front/components/assistant_builder/AssistantBuilderDataSourceModal.tsx

front/lib/swr.ts

front/pages/w/[wId]/builder/data-sources/public-urls.tsx

front/scripts/helpers.ts

flvndvd · 2024-08-22T13:34:24Z

types/src/front/api_handlers/internal/agent_configuration.ts

@@ -62,7 +62,7 @@ const RetrievalActionConfigurationSchema = t.type({
  dataSources: t.array(
    t.type({
      dataSourceId: t.string,
-      dataSourceViewId: t.union([t.string, t.null]),
+      dataSourceViewId: t.string,


This will break the assistant builder for users already on the builder when we release.

What do you propose ?

flvndvd

I left a few minor comments, but they aren't critical. Let's go ahead and merge this PR since it is holding up several others. We can address those comments in a subsequent PR.

flvndvd · 2024-08-23T12:22:56Z

front/components/assistant/AssistantDetails.tsx

@@ -446,7 +444,7 @@ function DataSourceSelectedNodes({

  return (
    <>
-      {dataSourceSelectedNodes.nodes.map((node: ContentNode) => (
+      {dataSourceSelectedNodes.nodes.map((node: LightContentNode) => (


Given that we have not yet changed useDataSourceContentNodes this should still be a ContentNode

flvndvd · 2024-08-23T12:23:53Z

front/components/assistant_builder/AssistantBuilder.tsx

@@ -322,7 +322,7 @@ export default function AssistantBuilder({
      });
    } else {
      await mutate(
-        `/api/w/${owner.sId}/data_sources/${slackDataSource?.name}/managed/slack/channels_linked_with_agent`
+        `/api/w/${owner.sId}/data_sources/${slackDataSource?.dataSource.name}/managed/slack/channels_linked_with_agent`


nit but slackDataSource should be named slackDataSourceView here.

front/components/assistant_builder/AssistantBuilderTablesModal.tsx

flvndvd · 2024-08-23T12:34:15Z

front/components/assistant_builder/PickDataSource.tsx

-import { CONNECTOR_CONFIGURATIONS } from "@app/lib/connector_providers";
+import { AssistantBuilderContext } from "@app/components/assistant_builder/AssistantBuilderContext";
+import { orderDatasourceViewByImportance } from "@app/lib/assistant";
+import { getConnectorProviderLogoWithFallback } from "@app/lib/connector_providers";
 import { getDisplayNameForDataSource } from "@app/lib/data_sources";

 export default function PickDataSource({


Follow up item, renamed this component to PickDataSourceView.

flvndvd · 2024-08-23T12:36:08Z

front/components/vaults/EditVaultManagedDatasourcesViews.tsx

@@ -143,7 +143,9 @@ export function EditVaultManagedDataSourcesViews({
        }}
        owner={owner}
        systemVaultDataSourceViews={systemVaultDataSourceViews.filter(
-          (ds) => ds.connectorProvider && ds.connectorProvider !== "webcrawler"
+          (dsv) =>


Follow up item we can use isManaged here.

flvndvd · 2024-08-23T12:37:39Z

front/lib/api/assistant/global_agents.ts

+  const defaultVaults = [await VaultResource.fetchWorkspaceGlobalVault(auth)];
+  const dataSourceViews = await DataSourceViewResource.listByVaults(
+    auth,
+    defaultVaults
+  );


nit

Suggested change

const defaultVaults = [await VaultResource.fetchWorkspaceGlobalVault(auth)];

const dataSourceViews = await DataSourceViewResource.listByVaults(

auth,

defaultVaults

);

const globalVault = await VaultResource.fetchWorkspaceGlobalVault(auth);

const dataSourceViews = await DataSourceViewResource.listByVaults(

auth,

[globalVault]

);

flvndvd · 2024-08-23T12:38:10Z

front/lib/api/assistant/global_agents.ts

 // Used when returning an agent with status 'disabled_by_admin'
 const dummyModelConfiguration = {
  providerId: GPT_4O_MODEL_CONFIG.providerId,
  modelId: GPT_4O_MODEL_CONFIG.modelId,
  temperature: 0,
 };

+type PrefetchedDataSourcesType = {


Follow up item, rename.

flvndvd · 2024-08-23T12:39:38Z

front/lib/connector_providers.ts

@@ -13,6 +13,8 @@ import type {
  DataSourceViewType,
  WhitelistableFeature,
 } from "@dust-tt/types";
+import type { LucideIcon } from "lucide-react";


Wait we should not use any of those icons 🤔

flvndvd · 2024-08-23T12:41:24Z

front/pages/w/[wId]/assistant/labs/transcripts/index.tsx

    }
    await makePatchRequest(
      transcriptConfigurationId,
      {
-        dataSourceId: dataSource ? dataSource.name : null,
+        dataSourceId: dataSource ? dataSource.dataSource.name : null,


Follow up item rename to dataSourceView.

* Refactor Assistant builder: use Vaults, use Context * Fix: wrong key for actions initial configurations * Fix: force close when picking a table from a folder

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch from fb3d3f0 to 88abec2 Compare August 16, 2024 16:50

Fraggle requested a review from flvndvd August 19, 2024 08:33

Fraggle marked this pull request as ready for review August 19, 2024 10:15

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch from 48fa4be to 3c3285a Compare August 19, 2024 10:16

flvndvd reviewed Aug 19, 2024

View reviewed changes

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch from 15eaf10 to c6a9600 Compare August 19, 2024 14:33

flvndvd reviewed Aug 20, 2024

View reviewed changes

front/components/DataSourceResourceSelectorTree.tsx Outdated Show resolved Hide resolved

front/lib/swr.ts Outdated Show resolved Hide resolved

front/components/DataSourceResourceSelectorTree.tsx Outdated Show resolved Hide resolved

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch 5 times, most recently from 7d4149c to 88f9ca6 Compare August 22, 2024 10:25

flvndvd reviewed Aug 22, 2024

View reviewed changes

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch 5 times, most recently from 581a3d1 to cecd6d8 Compare August 23, 2024 08:50

Fraggle added 2 commits August 23, 2024 14:14

Refactor Assistant builder: use Vaults, use Context

f104db5

Fix: wrong key for actions initial configurations

ba1c0ff

Fraggle force-pushed the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch from cecd6d8 to ba1c0ff Compare August 23, 2024 12:14

flvndvd approved these changes Aug 23, 2024

View reviewed changes

Fix: force close when picking a table from a folder

a0aae30

Fraggle merged commit 26baf3f into main Aug 23, 2024
3 checks passed

Fraggle deleted the 1149-groups-ui-update-assistant-builder-ui-to-use-vaults branch August 23, 2024 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Assistant builder: use Vaults, use Context #6768

Refactor Assistant builder: use Vaults, use Context #6768

Fraggle commented Aug 14, 2024 •

edited

Loading

Fraggle commented Aug 19, 2024

flvndvd Aug 22, 2024

flvndvd Aug 22, 2024

Fraggle Aug 22, 2024

flvndvd left a comment

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

flvndvd Aug 23, 2024

		@@ -357,7 +357,7 @@ function DataSourcesSection({
		<div className="flex flex-col gap-1">
		<ManagedDataSourceDocumentModal

Refactor Assistant builder: use Vaults, use Context #6768

Refactor Assistant builder: use Vaults, use Context #6768

Conversation

Fraggle commented Aug 14, 2024 • edited Loading

Description

Risk

Deploy Plan

Fraggle commented Aug 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flvndvd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Fraggle commented Aug 14, 2024 •

edited

Loading