Merge branch 'release/v0.5.0'

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Marcelle changelog
 
+## Next
+
+- `modelParameters` was improved to support other types
+- `genericChart` was improved to support lazy iterators (e.g. for datasets)
+- New component `onnxModel` using onnruntime-web with a wasm backend by default
+- New widgets (components): `number` and `numberArray`
+- `textField` was updated to `textInput` and its API was updated (`$value` stream instead of `$text`)
+- `button`, `text`, `toggle`, `select`: minor API changes (simplified constructors)
+- `slider` was improved with better defaults for pips and a continuous mode
+
 ## 0.4.4
 
 - Various minor bugfixes

README.md

@@ -16,14 +16,7 @@ Marcelle is still experimental and is currently under active development. Breaki
 
 ## About
 
-Marcelle is an interactive machine learning toolkit that has been designed to allow ML practitioners, with various levels of expertise, to explore ML and build applications embedding ML models. In particular, Marcelle aims to address the following use cases:
-
-1. interactively designing ML-based web applications with small datasets
-2. teaching ML to an audience without specific skills in ML or CS
-3. learning about ML concepts through exploratory interactive training and testing
-4. discovering and exploring expressive (big) ML models.
-
-Marcelle is a web-based reactive toolkit facilitating the design of custom ML pipelines and personalized user interfaces enabling user interactions on the pipeline's constitutive elements.
+Marcelle is a modular open source toolkit for programming interactive machine learning applications. Marcelle is built around components embedding computation and interaction that can be composed to form reactive machine learning pipelines and custom user interfaces. This architecture enables rapid prototyping and extension. Marcelle can be used to build interfaces to Python scripts, and it provides flexible data stores to facilitate collaboration between machine learning experts, designers and end users.
 
 ## Installation
 
@@ -72,10 +65,30 @@ To generate the typescript declaration files (useful for the rollup example), ru
 - [Prettier - Code formatter](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode)
 - [Tailwind CSS IntelliSense](https://marketplace.visualstudio.com/items?itemName=bradlc.vscode-tailwindcss)
 
-## ✍️ Authors
+## Credits
+
+Marcelle is a research project led by [Jules Françoise](https://www.julesfrancoise.com/) (CNRS researcher at [LISN](https://www.lisn.upsaclay.fr/)) and [Baptiste Caramiaux](https://baptistecaramiaux.com/) (CNRS researcher at [ISIR](https://hci.isir.upmc.fr/)).
+
+This research was supported by the [ELEMENT project](https://element-project.ircam.fr/) (ANR-18-CE33-0002) from the French National Research Agency.
+
+### ✍️ Authors
+
+- [Jules Françoise](https://www.julesfrancoise.com/) (CNRS researcher at [LISN](https://www.lisn.upsaclay.fr/))
+- [Baptiste Caramiaux](https://baptistecaramiaux.com/) (CNRS researcher at [ISIR](https://hci.isir.upmc.fr/)).
+- [Téo Sanchez](https://teo-sanchez.github.io/) (PhD Student at [LISN](https://www.lisn.upsaclay.fr/))
+
+### Citing this work
+
+Please cite the following publication when refering to Marcelle in academic publications:
+
+> Jules Françoise, Baptiste Caramiaux, Téo Sanchez. **Marcelle: Composing Interactive Machine Learning Workflows and Interfaces.** Annual ACM Symposium on User Interface Software and Technology (UIST ’21), Oct 2021, Virtual. DOI: [10.1145/3472749.3474734](https://doi.org/10.1145/3472749.3474734).<br> > [PDF](https://hal.archives-ouvertes.fr/hal-03335115/document)
+
+### Supporting institutions
 
-- Jules Françoise ([@JulesFrancoise](https://github.com/JulesFrancoise/)), LISN, Université Paris-Saclay, CNRS.
-- Baptiste Caramiaux ([@bcaramiaux](https://github.com/bcaramiaux/)), ISIR, Sorbonne Université, CNRS.
+- [CNRS](https://www.cnrs.fr)
+- [Université Paris Saclay](https://www.universite-paris-saclay.fr/)
+- [Sorbonne Université](https://www.sorbonne-universite.fr/)
+- [Inria](https://www.inria.fr/)
 
 ## 🔨 Built Using
 

docs/api/components/model-interfaces.md

@@ -35,7 +35,7 @@ The component will automatically display all parameters with appropriate GUI Wid
   <img src="./images/model-parameters.png" alt="Screenshot of the parameters component">
 </div>
 
-### Example
+### Examples
 
 ```js
 const classifier = marcelle.mlp({ layers: [64, 32], epochs: 20 });
@@ -44,6 +44,24 @@ const params = marcelle.parameters(classifier);
 dashboard.page('Training').use(params);
 ```
 
+```js
+const parametrable = {
+  parameters: {
+    int: new Stream(12, true),
+    float: new Stream(-0.0000045, true),
+    intArray: new Stream(Array.from(Array(3), () => Math.floor(100 * Math.random()))),
+    floatArray: new Stream(Array.from(Array(3), () => Math.random())),
+    string: new Stream('test'),
+    menu: new Stream('three'),
+    bool: new Stream(false),
+  },
+};
+
+const p = modelParameters(parametrable, {
+  menu: { type: 'menu', options: ['one', 'two', 'three'] },
+});
+```
+
 ## trainingProgress
 
 ```tsx

docs/api/components/models.md

@@ -268,6 +268,83 @@ const $embedding = input.$images.map((img) => m.process(img)).awaitPromises();
 const $prediction = input.$images.map((img) => m.predict(img)).awaitPromises();
 ```
 
+## onnxModel
+
+```tsx
+onnxModel({
+  inputType: 'image' | 'generic';
+  taskType: 'classification' | 'generic';
+  segmentationOptions?: {
+    output?: 'image' | 'tensor';
+    inputShape: number[];
+  };
+}): OnnxModel;
+```
+
+This component allows to make predictions using pre-trained models in the ONNX format, using [`onnxruntime-web`](https://github.com/microsoft/onnxruntime/tree/master/js/web). The default backend for inference is `wasm`, as it provides a wider operator support.
+
+The implementation currently supports tensors as input, formatted as nested number arrays, and two types of task (classification, generic prediction). Pre-trained models can be loaded either by URL, or through file upload, for instance using the [`fileUpload`](/api/components/widgets.html#fileupload) component.
+
+Such generic models cannot be trained.
+
+### Methods
+
+#### .loadFromFile()
+
+```tsx
+async loadFromFile(file: File): Promise<void>
+```
+
+Load a pre-trained ONNX model from a `*.onnx` file.
+
+#### .loadFromUrl()
+
+```tsx
+async loadFromUrl(url: string): Promise<void>
+```
+
+Load a pre-trained ONNX model from a URL.
+
+#### .predict()
+
+```tsx
+async predict(input: InputType): Promise<OutputType>
+```
+
+Make a prediction from an input instance, which type depends on the `inputType` specified in the constructor. The method is asynchronous and returns a promise that resolves with the results of the prediction.
+
+Input types can be:
+
+- `ImageData` if the model was instanciated with `inputType: 'image'`
+- `TensorLike` (= array) if the model was instanciated with `inputType: 'generic'`
+
+Output types can be:
+
+- `ClassifierResults` if the model was instanciated with `taskType: 'classification'`
+- `TensorLike` if the model was instanciated with `taskType: 'generic'`
+
+Where classifier results have the following interface:
+
+```ts
+interface ClassifierResults {
+  label: string;
+  confidences: { [key: string]: number };
+}
+```
+
+### Example
+
+```js
+const source = imageUpload();
+const classifier = tfjsModel({
+  inputType: 'image',
+  taskType: 'classification',
+});
+classifier.loadFromUrl();
+
+const predictionStream = source.$images.map(async (img) => classifier.predict(img)).awaitPromises();
+```
+
 ## tfjsModel
 
 ```tsx