Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DOCS-2899: Edit code samples from Data client QA #3457

Merged
merged 7 commits into from
Sep 19, 2024
Merged

DOCS-2899: Edit code samples from Data client QA #3457

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3f4d351
edit code samples
sguequierre Sep 13, 2024
c7e5ec1
Merge branch 'main' into DOCS-2899/data-client-qa
sguequierre Sep 13, 2024
3069a9d
Make edits from code review
sguequierre Sep 16, 2024
6e04c8a
apply suggestions from code review
sguequierre Sep 18, 2024
0992e4b
Merge branch 'main' into DOCS-2899/data-client-qa
sguequierre Sep 18, 2024
d6198b3
Fixup
sguequierre Sep 19, 2024
8ba7210
Apply suggestions from code review
sguequierre Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 62 additions & 36 deletions static/include/app/apis/generated/data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ You can also find your tabular data under the **Sensors** subtab of the app's [*
- `limit` ([int](https://docs.python.org/3/library/stdtypes.html#numeric-types-int-float-complex)) (optional): The maximum number of entries to include in a page. Defaults to 50 if unspecified.
- `sort_order` ([viam.proto.app.data.Order.ValueType](https://python.viam.dev/autoapi/viam/proto/app/data/index.html#viam.proto.app.data.Order)) (optional): The desired sort order of the data.
- `last` ([str](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str)) (optional): Optional string indicating the object identifier of the last-returned data. This object identifier is returned by calls to `TabularDataByFilter` as the last value. If provided, the server will return the next data entries after the last object identifier.
- `count_only` ([bool](https://docs.python.org/3/library/stdtypes.html#boolean-type-bool)) (required): Whether to return only the total count of entries.
- `include_internal_data` ([bool](https://docs.python.org/3/library/stdtypes.html#boolean-type-bool)) (required): Whether to return the internal data. Internal data is used for Viam-specific data ingestion, like cloud SLAM. Defaults to False.
- `count_only` ([bool](https://docs.python.org/3/library/stdtypes.html#boolean-type-bool)) (optional): Whether to return only the total count of entries.
- `include_internal_data` ([bool](https://docs.python.org/3/library/stdtypes.html#boolean-type-bool)) (optional): Whether to return the internal data. Internal data is used for Viam-specific data ingestion, like cloud SLAM. Defaults to False.
- `dest` ([str](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str)) (optional): Optional filepath for writing retrieved data.

**Returns:**
Expand All @@ -26,13 +26,15 @@ You can also find your tabular data under the **Sensors** subtab of the app's [*
from viam.utils import create_filter

my_data = []
my_filter = create_filter(component_name="motor-1")
last = None
my_filter = create_filter(component_name="left_motor")
while True:
tabular_data, count, last = await data_client.tabular_data_by_filter(my_filter, last)
tabular_data, count, last = await data_client.tabular_data_by_filter(my_filter, last=last)
if not tabular_data:
break
my_data.extend(tabular_data)

print(f"My data: {my_data}")
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.tabular_data_by_filter).
Expand Down Expand Up @@ -88,17 +90,22 @@ Obtain unified tabular data and metadata, queried with MQL.
```python {class="line-numbers linkable-line-numbers"}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused with this, this feels like it's become a much more involved code sample. And we now no longer show how it works with pymongo?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was required to make it work and I am still using pymongo-- it wasn't working for me otherwise

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you are indeed using pymongo. Which means the code now doesn't work for the bson package.
I just tested both code snippets again and they do work.

Did you run pip uninstall pymongo and pip install bson when you were using the bson package code? Can you try these again and let me know if they work? If not lets look into it together. The code snippets as they are are a lot shorter and cleaner so if possible I'd like to stick with them

Maybe we can make that clearer like this?

# using bson package (pip install bson)
import bson
tabular_data = await data_client.tabular_data_by_mql(org_id="<your-org-id>", mql_binary=[
    bson.dumps({ '$match': { 'location_id': '<location-id>' } }),
    bson.dumps({ '$limit': 5 })
])

# using pymongo package (pip install pymongo)
import bson
tabular_data = await data_client.tabular_data_by_mql(org_id="<your-org-id>", mql_binary=[
    bson.encode({ '$match': { 'location_id': '<location-id>' } }),
    bson.encode({ '$limit': 5 })
])

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the first option to work with organization_id. The second option as far as i can tell with pymongo still needs to be bson.BSON to work properly. Editing as such

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah true I missed the organization_id bit when copying - I'll send you a code snippet.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bson.BSON.encode works, but weirdly so does running it without. I'd love to see what error you're getting but this is fine to merge

# using bson
sguequierre marked this conversation as resolved.
Show resolved Hide resolved
import bson
tabular_data = await data_client.tabular_data_by_mql(org_id="<your-org-id>", mql_binary=[
bson.dumps({ '$match': { 'location_id': '<location-id>' } }),
bson.dumps({ "$limit": 5 })

# using bson package (pip install bson)
tabular_data = await data_client.tabular_data_by_mql(organization_id="<YOUR-ORG-ID>", mql_binary=[
bson.dumps({ '$match': { 'location_id': '<YOUR-LOCATION-ID>' } }),
bson.dumps({ '$limit': 5 })
])

# using pymongo
import bson
tabular_data = await data_client.tabular_data_by_mql(org_id="<your-org-id>", mql_binary=[
bson.encode({ '$match': { 'location_id': '<location-id>' } }),
bson.encode({ "$limit": 5 })
print(f"Tabular Data 1: {tabular_data}")

# using pymongo package (pip install pymongo)
tabular_data = await data_client.tabular_data_by_mql(organization_id="<YOUR-ORG-ID>", mql_binary=[
bson.BSON.encode({ '$match': { 'location_id': '<YOUR-LOCATION-ID>' } }),
bson.BSON.encode({ "$limit": 5 })
])

print(f"Tabular Data 2: {tabular_data}")
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.tabular_data_by_mql).
Expand Down Expand Up @@ -135,17 +142,19 @@ You can also find your binary data under the **Images**, **Point clouds**, or **
from viam.utils import create_filter
from viam.proto.app.data import Filter, TagsFilter, TagsFilterType


# Get data captured from camera components
my_data = []
last = None
my_filter = create_filter(component_name="camera")
my_filter = create_filter(component_name="camera-1")

while True:
data, count, last = await data_client.binary_data_by_filter(my_filter, last)
data, count, last = await data_client.binary_data_by_filter(my_filter, limit=1, last=last)
if not data:
break
my_data.extend(data)

print(f"My data: {my_data}")

# Get untagged data from a dataset
npentrel marked this conversation as resolved.
Show resolved Hide resolved

my_untagged_data = []
Expand Down Expand Up @@ -194,8 +203,12 @@ You can also find your binary data under the **Images**, **Point clouds**, or **

```python {class="line-numbers linkable-line-numbers"}
from viam.proto.app.data import BinaryID
from viam.utils import create_filter

binary_metadata, _, _ = await data_client.binary_data_by_filter(
my_filter = create_filter(component_name="camera-1", organization_ids=["<YOUR-ORG-ID>"])
binary_metadata, count, last = await data_client.binary_data_by_filter(
filter=my_filter,
limit=20,
include_binary_data=False
)

Expand Down Expand Up @@ -237,12 +250,10 @@ Delete tabular data older than a specified number of days.
**Example:**

```python {class="line-numbers linkable-line-numbers"}
from viam.utils import create_filter

my_filter = create_filter(component_name="left_motor")
days_of_data_to_delete = 10
tabular_data = await data_client.delete_tabular_data(
org_id="a12b3c4e-1234-1abc-ab1c-ab1c2d345abc", days_of_data_to_delete)
organization_id="<YOUR-ORG-ID>",
delete_older_than_days=150
)
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.delete_tabular_data).
Expand All @@ -259,7 +270,7 @@ Filter and delete binary data.

**Parameters:**

- `filter` ([viam.proto.app.data.Filter](https://python.viam.dev/autoapi/viam/proto/app/data/index.html#viam.proto.app.data.Filter)) (optional): Optional Filter specifying binary data to delete. Passing an empty Filter will lead to all data being deleted. Exercise caution when using this option.
- `filter` ([viam.proto.app.data.Filter](https://python.viam.dev/autoapi/viam/proto/app/data/index.html#viam.proto.app.data.Filter)) (optional): Optional Filter specifying binary data to delete. Passing an empty Filter will lead to all data being deleted. Exercise caution when using this option. You must specify any organization ID with "organization_ids" when using this option.

**Returns:**

Expand All @@ -270,7 +281,7 @@ Filter and delete binary data.
```python {class="line-numbers linkable-line-numbers"}
from viam.utils import create_filter

my_filter = create_filter(component_name="left_motor")
my_filter = create_filter(component_name="my-webcam", organization_ids=["<YOUR-ORG-ID>"])
res = await data_client.delete_binary_data_by_filter(my_filter)
```

Expand Down Expand Up @@ -302,8 +313,12 @@ Filter and delete binary data by ids.

```python {class="line-numbers linkable-line-numbers"}
from viam.proto.app.data import BinaryID
from viam.utils import create_filter

binary_metadata, _, _ = await data_client.binary_data_by_filter(
my_filter = create_filter(component_name="camera-1", organization_ids=["<YOUR-ORG-ID>"])
binary_metadata, count, last = await data_client.binary_data_by_filter(
filter=my_filter,
limit=20,
include_binary_data=False
)

Expand Down Expand Up @@ -350,10 +365,14 @@ Add tags to binary data by ids.

```python {class="line-numbers linkable-line-numbers"}
from viam.proto.app.data import BinaryID
from viam.utils import create_filter

tags = ["tag1", "tag2"]

binary_metadata, _, _ = await data_client.binary_data_by_filter(
my_filter = create_filter(component_name="camera-1", organization_ids=["<YOUR-ORG-ID>"])
binary_metadata, count, last = await data_client.binary_data_by_filter(
filter=my_filter,
limit=20,
include_binary_data=False
)

Expand Down Expand Up @@ -435,10 +454,15 @@ Remove tags from binary by ids.

```python {class="line-numbers linkable-line-numbers"}
from viam.proto.app.data import BinaryID
from viam.utils import create_filter

tags = ["tag1", "tag2"]

binary_metadata, _, _ = await data_client.binary_data_by_filter(
my_filter = create_filter(component_name="camera-1")

binary_metadata, count, last = await data_client.binary_data_by_filter(
filter=my_filter,
limit=50,
include_binary_data=False
)

Expand Down Expand Up @@ -556,12 +580,12 @@ Add a bounding box to an image specified by its BinaryID.
from viam.proto.app.data import BinaryID

MY_BINARY_ID = BinaryID(
file_id=your-file_id,
organization_id=your-org-id,
location_id=your-location-id
file_id="<YOUR-FILE-ID>",
organization_id="<YOUR-ORG-ID>",
location_id="<YOUR-LOCATION-ID>"
)

bbox_label = await data_client.add_bounding_box_to_image_by_id(
bbox_id = await data_client.add_bounding_box_to_image_by_id(
binary_id=MY_BINARY_ID,
label="label",
x_min_normalized=0,
Expand All @@ -570,7 +594,7 @@ bbox_label = await data_client.add_bounding_box_to_image_by_id(
y_max_normalized=.3
)

print(bbox_label)
print(bbox_id)
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.add_bounding_box_to_image_by_id).
Expand Down Expand Up @@ -639,6 +663,8 @@ from viam.utils import create_filter
my_filter = create_filter(component_name="my_camera")
bounding_box_labels = await data_client.bounding_box_labels_by_filter(
my_filter)

print(bounding_box_labels)
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.bounding_box_labels_by_filter).
Expand All @@ -664,7 +690,7 @@ Get a connection to access a MongoDB Atlas Data federation instance.
**Example:**

```python {class="line-numbers linkable-line-numbers"}
data_client.get_database_connection(org_id="a12b3c4e-1234-1abc-ab1c-ab1c2d345abc")
hostname = await data_client.get_database_connection(organization_id="<YOUR-ORG-ID>")
```

For more information, see the [Python SDK Docs](https://python.viam.dev/autoapi/viam/app/data_client/index.html#viam.app.data_client.DataClient.get_database_connection).
Expand Down Expand Up @@ -693,8 +719,8 @@ It can also be used to reset the password of the existing database user.

```python {class="line-numbers linkable-line-numbers"}
await data_client.configure_database_user(
organization_id="<your-org-id>",
password="your_password"
organization_id="<YOUR-ORG-ID>",
password="Your_Password@1234"
)
```

Expand Down Expand Up @@ -725,8 +751,8 @@ This BinaryData will be tagged with the VIAM_DATASET\_{id} label.
```python {class="line-numbers linkable-line-numbers"}
from viam.proto.app.data import BinaryID

binary_metadata, _, _ = await data_client.binary_data_by_filter(
include_binary_data=False
binary_metadata, count, last = await data_client.binary_data_by_filter(
include_file_data=False
)

my_binary_ids = []
Expand Down
2 changes: 1 addition & 1 deletion static/include/app/apis/generated/data_sync.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ file_id = await data_client.streaming_data_capture_upload(
component_type='motor',
component_name='left_motor',
method_name='IsPowered',
data_request_times=[(time_requested, time_received)],
data_request_times=[time_requested, time_received],
npentrel marked this conversation as resolved.
Show resolved Hide resolved
tags=["tag_1", "tag_2"]
)
```
Expand Down
18 changes: 9 additions & 9 deletions static/include/app/apis/generated/dataset.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ Create a new dataset.

```python {class="line-numbers linkable-line-numbers"}
dataset_id = await data_client.create_dataset(
name="<dataset-name>",
organization_id="<your-org-id>"
name="<DATASET-NAME>",
organization_id="<YOUR-ORG-ID>"
)
print(dataset_id)
```
Expand All @@ -38,7 +38,7 @@ Delete a dataset.

**Parameters:**

- `id` ([str](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str)) (required): The ID of the dataset.
- `id` ([str](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str)) (required): The ID of the dataset. You can retrieve this by navigating to the **DATASETS** sub-tab of the **DATA** tab, clicking on the dataset, clicking the **...** menu and selecting **Copy dataset ID**.

**Returns:**

Expand All @@ -48,7 +48,7 @@ Delete a dataset.

```python {class="line-numbers linkable-line-numbers"}
await data_client.delete_dataset(
id="abcd-1234xyz-8765z-123abc"
id="<YOUR-DATASET-ID>"
)
```

Expand Down Expand Up @@ -77,8 +77,8 @@ Rename a dataset specified by the dataset ID.

```python {class="line-numbers linkable-line-numbers"}
await data_client.rename_dataset(
id="abcd-1234xyz-8765z-123abc",
name="<dataset-name>"
id="<YOUR-DATASET-ID>",
name="MyDataset"
)
```

Expand All @@ -105,8 +105,8 @@ Get the datasets in an organization.
**Example:**

```python {class="line-numbers linkable-line-numbers"}
datasets = await data_client.list_dataset_by_organization_id(
organization_id=[""a12b3c4e-1234-1abc-ab1c-ab1c2d345abc""]
datasets = await data_client.list_datasets_by_organization_id(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

organization_id="<YOUR-ORG-ID>"
)
print(datasets)
```
Expand Down Expand Up @@ -135,7 +135,7 @@ Get a list of datasets using their IDs.

```python {class="line-numbers linkable-line-numbers"}
datasets = await data_client.list_dataset_by_ids(
ids=["abcd-1234xyz-8765z-123abc"]
ids=["<YOUR-DATASET-ID-1>, <YOUR-DATASET-ID-2>"]
)
print(datasets)
```
Expand Down
Loading