From 06c4a28a33ab9e23365ae538bcb85f079ea55c6d Mon Sep 17 00:00:00 2001 From: johndmulhausen Date: Sat, 13 Sep 2025 16:10:34 -0400 Subject: [PATCH] Add documentation for CustomChart API and query syntax - Added comprehensive documentation for using wr.CustomChart in reports - Created dedicated query syntax guide for custom charts - Updated edit-a-report.md with CustomChart examples - Enhanced custom-charts index with Reports API integration details --- .../en/guides/core/reports/edit-a-report.md | 39 ++ .../app/features/custom-charts/_index.md | 74 +++- .../features/custom-charts/query-syntax.md | 340 ++++++++++++++++++ 3 files changed, 448 insertions(+), 5 deletions(-) create mode 100644 content/en/guides/models/app/features/custom-charts/query-syntax.md diff --git a/content/en/guides/core/reports/edit-a-report.md b/content/en/guides/core/reports/edit-a-report.md index 5f591db711..da93cd8353 100644 --- a/content/en/guides/core/reports/edit-a-report.md +++ b/content/en/guides/core/reports/edit-a-report.md @@ -75,6 +75,45 @@ report.save() For more information about available plots and charts you can add to a report programmatically, see `wr.panels`. +### Adding custom charts + +Use `wr.CustomChart` to add visualizations that query data from custom tables: + +```python +import wandb_workspaces.reports.v2 as wr + +# Query data from a summary table +custom_chart = wr.CustomChart( + query={ + 'summaryTable': { + 'tableKey': 'eval_results', + 'tableColumns': ['metric', 'value'] + } + }, + chart_name='wandb/bar/v0', + chart_fields={'x': 'metric', 'y': 'value'} +) + +# Add to report with other panels +report = wr.Report( + project="my-project", + title="Analysis Report" +) + +panel_grid = wr.PanelGrid( + panels=[ + wr.LinePlot(x="step", y=["loss"]), + custom_chart # Add the custom chart + ], + runsets=[wr.RunSet(entity="my-entity", project="my-project")] +) + +report.blocks = [panel_grid] +report.save() +``` + +For detailed query syntax and more examples, see the [Custom charts query syntax guide]({{< relref "/guides/models/app/features/custom-charts/query-syntax.md" >}}). + {{% /tab %}} {{< /tabpane >}} diff --git a/content/en/guides/models/app/features/custom-charts/_index.md b/content/en/guides/models/app/features/custom-charts/_index.md index 91f7f3c2f6..7eb1391f9a 100644 --- a/content/en/guides/models/app/features/custom-charts/_index.md +++ b/content/en/guides/models/app/features/custom-charts/_index.md @@ -224,7 +224,7 @@ You can log the following data types from your script and use them in a custom c * **Config**: Initial settings of your experiment (your independent variables). This includes any named fields you've logged as keys to `wandb.Run.config` at the start of your training. For example: `wandb.Run.config.learning_rate = 0.0001` * **Summary**: Single values logged during training (your results or dependent variables). For example, `wandb.Run.log({"val_acc" : 0.8})`. If you write to this key multiple times during training via `wandb.Run.log()`, the summary is set to the final value of that key. * **History**: The full time series of the logged scalar is available to the query via the `history` field -* **summaryTable**: If you need to log a list of multiple values, use a `wandb.Table()` to save that data, then query it in your custom panel. +* **summaryTable**: If you need to log a list of values, use a `wandb.Table()` to save that data, then query it in your custom panel. * **historyTable**: If you need to see the history data, then query `historyTable` in your custom chart panel. Each time you call `wandb.Table()` or log a custom chart, you're creating a new table in history for that step. ### How to log a custom table @@ -248,6 +248,33 @@ Add a new custom chart to get started, then edit the query to select data from y {{< img src="/images/app_ui/customize_chart.gif" alt="Custom chart creation" max=width="90%" >}} +### Query syntax for different data tables + +W&B provides several ways to query your data: + +- **summary**: Access final metric values (for example, `{"summary": ["val_acc", "val_loss"]}`) +- **history**: Access time-series data (for example, `{"history": ["loss", "accuracy"]}`) +- **summaryTable**: Access custom tables with specific keys: + ```json + { + "summaryTable": { + "tableKey": "eval_results", + "tableColumns": ["precision", "recall"] + } + } + ``` +- **historyTable**: Access tables logged at multiple steps: + ```json + { + "historyTable": { + "tableKey": "predictions", + "tableColumns": ["x", "y", "prediction"] + } + } + ``` + +For detailed query examples and patterns, see the [Query syntax guide]({{< relref "/guides/models/app/features/custom-charts/query-syntax.md" >}}). + ### Custom visualizations Select a **Chart** in the upper right corner to start with a default preset. Next, select **Chart fields** to map the data you're pulling in from the query to the corresponding fields in your chart. @@ -270,12 +297,49 @@ To set a default value for a field, use this syntax: `"${field::}}). + ## Articles and guides -1. [The W&B Machine Learning Visualization IDE](https://wandb.ai/wandb/posts/reports/The-W-B-Machine-Learning-Visualization-IDE--VmlldzoyNjk3Nzg) -2. [Visualizing NLP Attention Based Models](https://wandb.ai/kylegoyette/gradientsandtranslation2/reports/Visualizing-NLP-Attention-Based-Models-Using-Custom-Charts--VmlldzoyNjg2MjM) -3. [Visualizing The Effect of Attention on Gradient Flow](https://wandb.ai/kylegoyette/gradientsandtranslation/reports/Visualizing-The-Effect-of-Attention-on-Gradient-Flow-Using-Custom-Charts--VmlldzoyNjg1NDg) -4. [Logging arbitrary curves](https://wandb.ai/stacey/presets/reports/Logging-Arbitrary-Curves--VmlldzoyNzQyMzA) +1. [Query syntax for custom charts]({{< relref "/guides/models/app/features/custom-charts/query-syntax.md" >}}) +2. [The W&B Machine Learning Visualization IDE](https://wandb.ai/wandb/posts/reports/The-W-B-Machine-Learning-Visualization-IDE--VmlldzoyNjk3Nzg) +3. [Visualizing NLP Attention Based Models](https://wandb.ai/kylegoyette/gradientsandtranslation2/reports/Visualizing-NLP-Attention-Based-Models-Using-Custom-Charts--VmlldzoyNjg2MjM) +4. [Visualizing The Effect of Attention on Gradient Flow](https://wandb.ai/kylegoyette/gradientsandtranslation/reports/Visualizing-The-Effect-of-Attention-on-Gradient-Flow-Using-Custom-Charts--VmlldzoyNjg1NDg) +5. [Logging arbitrary curves](https://wandb.ai/stacey/presets/reports/Logging-Arbitrary-Curves--VmlldzoyNzQyMzA) ## Common use cases diff --git a/content/en/guides/models/app/features/custom-charts/query-syntax.md b/content/en/guides/models/app/features/custom-charts/query-syntax.md new file mode 100644 index 0000000000..81417bf58e --- /dev/null +++ b/content/en/guides/models/app/features/custom-charts/query-syntax.md @@ -0,0 +1,340 @@ +--- +description: Learn how to build queries for custom charts and access different data tables +menu: + default: + identifier: query-syntax + parent: custom-charts +title: Query syntax for custom charts +weight: 10 +--- + +Learn how to build queries for custom charts to access different data tables in W&B. This guide covers both UI-based custom charts and programmatic chart creation using the Reports API. + +## Understanding data tables + +W&B provides different data tables you can query in custom charts: + +### Summary data +Access final values from your runs using `summary`: +- Contains the last logged value for each metric +- Useful for comparing final results across runs +- Example: final accuracy, best validation score + +### History data +Access time-series data using `history`: +- Contains all logged values over time +- Useful for plotting training curves +- Example: loss over epochs, metrics over steps + +### Summary tables +Access custom tables logged to summary using `summaryTable`: +- Contains `wandb.Table()` objects logged once per run +- Useful for structured data like evaluation results +- Requires specifying `tableKey` and `tableColumns` + +### History tables +Access custom tables logged during training using `historyTable`: +- Contains `wandb.Table()` objects logged at multiple steps +- Each `wandb.log()` with a table creates a new entry +- Useful for progressive results like per-epoch predictions + +## Query syntax examples + +### Accessing summary metrics + +To query summary metrics: + +```json +{ + "summary": ["val_loss", "val_acc", "train_loss"] +} +``` + +### Accessing history metrics + +To query time-series data: + +```json +{ + "history": ["loss", "accuracy", "learning_rate"] +} +``` + +### Accessing summary tables + +To query a custom table logged to summary: + +```json +{ + "summaryTable": { + "tableKey": "evaluation_results", + "tableColumns": ["precision", "recall", "f1_score"] + } +} +``` + +### Accessing history tables + +To query tables logged at different time steps: + +```json +{ + "historyTable": { + "tableKey": "predictions_per_epoch", + "tableColumns": ["epoch", "image", "prediction", "ground_truth"] + } +} +``` + +## Using CustomChart with the reports API + +When creating reports programmatically, use `wr.CustomChart` to add custom visualizations that query your logged data. + +### Basic CustomChart examples + +```python +import wandb_workspaces.reports.v2 as wr + +# Create a chart from summary data +custom_chart = wr.CustomChart( + query={'summary': ['val_loss', 'val_acc']}, + chart_name='wandb/scatter/v0', + chart_fields={'x': 'val_loss', 'y': 'val_acc'} +) + +# Add to a report +report = wr.Report( + project="my-project", + title="Custom Charts Report" +) + +panel_grid = wr.PanelGrid( + panels=[custom_chart], + runsets=[wr.RunSet(entity="my-entity", project="my-project")] +) + +report.blocks = [panel_grid] +report.save() +``` + +### Querying summary tables + +To visualize data from a logged `wandb.Table`: + +```python +# First, log a table in your training script +wandb.init() +eval_table = wandb.Table( + columns=["class", "precision", "recall"], + data=[ + ["cat", 0.95, 0.92], + ["dog", 0.89, 0.94] + ] +) +wandb.log({"eval_results": eval_table}) + +# Then, create a custom chart in your report +custom_chart = wr.CustomChart( + query={ + 'summaryTable': { + 'tableKey': 'eval_results', + 'tableColumns': ['class', 'precision', 'recall'] + } + }, + chart_name='wandb/bar/v0', + chart_fields={'x': 'class', 'y': 'precision'} +) +``` + +### Querying history tables + +For tables logged at different time steps: + +```python +# In your training script +for epoch in range(num_epochs): + predictions_table = wandb.Table( + columns=["x", "y", "prediction"], + data=[[x, y, pred] for x, y, pred in epoch_predictions] + ) + wandb.log({"predictions": predictions_table}) + +# In your report +custom_chart = wr.CustomChart( + query={ + 'historyTable': { + 'tableKey': 'predictions', + 'tableColumns': ['x', 'y', 'prediction'] + } + }, + chart_name='wandb/scatter/v0', + chart_fields={'x': 'x', 'y': 'y', 'color': 'prediction'} +) +``` + +### Combining data sources + +You can query different tables in a single chart: + +```python +custom_chart = wr.CustomChart( + query={ + 'summary': ['final_score'], + 'summaryTable': { + 'tableKey': 'class_scores', + 'tableColumns': ['class', 'score'] + } + }, + chart_name='wandb/custom/v0', + chart_fields={ + 'x': 'class', + 'y': 'score', + 'title': 'Class Performance' + } +) +``` + +## Importing existing charts to reports + +To programmatically add charts that you've already created in the UI to a report: + +### Method 1: Import charts via UI then access programmatically + +```python +# Step 1: Create a report with imported charts in the UI +# Use "Create Report > Import charts" to add existing charts + +# Step 2: Load the report and extract panels +import wandb_workspaces.reports.v2 as wr + +# Load the report containing imported charts +source_report = wr.Report.from_url("https://wandb.ai/entity/project/reports/...") +imported_panels = source_report.blocks[0].panels + +# Step 3: Add specific panels to a new report +target_report = wr.Report( + project="my-project", + title="Report with Imported Charts" +) + +# Select the panel you want (by index or by filtering) +selected_panel = imported_panels[0] # First panel + +# Add to new report +panel_grid = wr.PanelGrid( + panels=[selected_panel], + runsets=[wr.RunSet(entity="my-entity", project="my-project")] +) + +target_report.blocks = [panel_grid] +target_report.save() +``` + +### Method 2: Clone and update existing panels + +```python +# Load a report as a template +template_report = wr.Report.from_url("https://wandb.ai/entity/project/reports/...") + +# Extract and modify panels +for block in template_report.blocks: + if isinstance(block, wr.PanelGrid): + # Access existing panels + existing_panels = block.panels + + # Modify or filter panels as needed + modified_panels = [] + for panel in existing_panels: + if isinstance(panel, wr.CustomChart): + # You can access and modify the query + panel.query['summaryTable']['tableKey'] = 'new_table_name' + modified_panels.append(panel) + + # Create new panel grid with modified panels + new_panel_grid = wr.PanelGrid( + panels=modified_panels, + runsets=block.runsets + ) + +# Save the modified report +new_report = wr.Report( + project="my-project", + title="Modified Report" +) +new_report.blocks = [new_panel_grid] +new_report.save() +``` + +## Common query patterns + +### Filtering and grouping + +When building queries, you can filter runs and group data: + +```python +# Filter runs by config values +custom_chart = wr.CustomChart( + query={ + 'summary': ['accuracy'], + 'config': ['model_type', 'learning_rate'] + }, + chart_name='wandb/scatter/v0', + chart_fields={ + 'x': 'learning_rate', + 'y': 'accuracy', + 'color': 'model_type' + } +) +``` + +### Multi-axis charts + +Create charts with two y-axes: + +```python +custom_chart = wr.CustomChart( + query={'history': ['train_loss', 'val_loss', 'learning_rate']}, + chart_name='wandb/line/v0', + chart_fields={ + 'x': '_step', + 'y': ['train_loss', 'val_loss'], + 'y2': 'learning_rate' # Secondary y-axis + } +) +``` + +## Best practices + +1. **Use appropriate table types**: + - Use `summaryTable` for final results that don't change + - Use `historyTable` for data that evolves during training + +2. **Limit table size**: Keep tables under 10,000 rows for optimal performance + +3. **Name keys consistently**: Use descriptive, consistent names for table keys across your project + +4. **Test queries in UI first**: Build and test your query in the UI custom chart editor, then transfer to code + +5. **Cache imported panels**: When importing charts, load the source report once and reuse the panels + +## Troubleshooting + +### Data not appearing + +If your data doesn't appear in the custom chart: +- Verify the table key matches exactly what you logged +- Check that selected runs contain the logged tables +- Ensure column names in `tableColumns` match your table's column names + +### Query errors + +Common query issues and solutions: +- **Missing quotes**: Ensure all strings in the query are properly quoted +- **Wrong field type**: `summaryTable` requires an object with `tableKey` and `tableColumns`, not an array +- **Typos in field names**: Double-check spelling of table keys and column names + +## See also + +- [Custom charts walkthrough]({{< relref "/guides/models/app/features/custom-charts/walkthrough.md" >}}) +- [Reports API reference]({{< relref "/ref/python/wandb_workspaces/reports.md" >}}) +- [Logging plots and media]({{< relref "/guides/models/track/log/plots.md" >}})