John Friction Log

[jig925]

Notes while working on Breadboard.

[jig925]

Schema Defaults do not get used when running a board programmatically. Given a schema:

  type: "string",
  title: "Story ID",
  default: "39788322",
  description: "HackerNews Story ID to extract"  

The board does not use the default value of "39788322" when running the board programmatically without providing the storyID.

[Mearman]

I also encountered this disjoint from the other side whereby TS param default values not being used by the web app means the board behaves differently between the Runner and Web.
There is also the environment of the CLI and Harness.

[dglazkov]

Filed #1183

[dglazkov]

Yup, they will behave differently currently. @PaulKinlan has been working to bring them to coherence.

[PaulKinlan]

Is the "story ID" required? It was mentioned a while ago that default values should only be used on fields that are not marked required.

[jig925]

Breadboard Web does not allow non-required inputs to be left empty. This can be seen in the following board:

import { board, base } from "@google-labs/breadboard";
import { templates } from "@google-labs/template-kit";
import path from "path";
import fs from "fs";

const inputSchema = {
    type: "number",
    title: "my Number",
    default: "1",
    description: "myNumber",
};


const serialized = await board(() => {
    const input = base.input({
        $id: "input",
        schema: {
            title: "My Inputs",
            properties: {
                myNumber: inputSchema,
            },
        },
        type: "number",
    })

    const urlTemplate = templates.urlTemplate({
        $id: "urlTemplate",
        template: "https://hn.algolia.com/api/v1/items/{myNumber}",
        myNumber: input.myNumber
    });

    return {output : urlTemplate.url}
}).serialize({
    title: "Board Not Required Bug",
    description: "Board which demonstrates a bug where breadboard web wont continue without a non required variable"
})


fs.writeFileSync(
	path.join(".", "board.json"),
	JSON.stringify(serialized, null, "\t")
);

Breadboard Web Link:

https://breadboard-ai.web.app/?mode=build&board=https%3A%2F%2Fgist.githubusercontent.com%2Fjig925%2F6ebc264a23506b5458b3cee90a5bbc75%2Fraw%2Fc528fa3494e11432c5b437b36a26cd87ac1117f3%2Fnon-required-bug.json

[jig925]

https://gist.github.com/jig925/6ebc264a23506b5458b3cee90a5bbc75

[dglazkov]

Love the gists! Filed #1409

[jig925]

Breadboard web does not use defaults when no inputs are provided. It was mentioned that defaults are used for non-required inputs, so this should be using the default value of 1. urlTemplate does not error if there is a parameter missing.

import { board, base } from "@google-labs/breadboard";
import { templates } from "@google-labs/template-kit";
import path from "path";
import fs from "fs";

const inputSchema = {
    type: "number",
    title: "my Number",
    default: "1",
    description: "myNumber",
};

const serialized = await board(() => {
    const input = base.input({
        $id: "input",
        schema: {
            title: "My Inputs",
            properties: {
                myNumber: inputSchema,
            },
        },
        type: "number",
    })

    const urlTemplate = templates.urlTemplate({
        $id: "urlTemplate",
        template: "https://hn.algolia.com/api/v1/items/{myNumber}",
    });

    input.to(urlTemplate)

    return {output : urlTemplate.url}
}).serialize({
    title: "Board Defaults Not Used. urlTemplate Error",
    description: "Board which demonstrates a bug where breadboard web does not use default values and urlTemplate does not error if parameters are missing"
})

fs.writeFileSync(
	path.join(".", "board.json"),
	JSON.stringify(serialized, null, "\t")

Breadboard Web Link:
https://breadboard-ai.web.app/?mode=build&board=https%3A%2F%2Fgist.githubusercontent.com%2Fjig925%2F6ebc264a23506b5458b3cee90a5bbc75%2Fraw%2Fc528fa3494e11432c5b437b36a26cd87ac1117f3%2Fdefault-template-bug.json

[jig925]

https://gist.github.com/jig925/6ebc264a23506b5458b3cee90a5bbc75

[dglazkov]

Very very interesting.

[dglazkov]

Filed #1410 and #1411

[jig925]

Core-Kit fetch will always stringify the request body:

  invoke: async (
    { url, method, body, headers, raw, stream },
    { signal }: NodeHandlerContext
  ) => {
    if (!url) throw new Error("Fetch requires `url` input");
    const init: RequestInit = {
      method,
      headers,
      signal,
    };
    // GET can not have a body.
    if (method !== "GET") {
      init.body = JSON.stringify(body);
    } else if (body) {
      throw new Error("GET requests can not have a body");
    }
    ...
}

This may not be a desired behaviour for certain API endpoints. For example, see : https://huggingface.co/docs/api-inference/detailed_parameters?code=js#image-classification-task

Using fetch results in a successful API call, but core.fetch results in an error because of the mentioned request body being stringified.

Hugging Face API key can be generated in https://huggingface.co/settings/tokens

import fs from "fs";
import { core } from "@google-labs/core-kit";

// fileName can be any image on local file system
async function query(filename: string) {
    const data = fs.readFileSync(filename);

    const response = await fetch(
        "https://api-inference.huggingface.co/models/google/vit-base-patch16-224",
        {
            headers: { Authorization: `Bearer myHuggingFaceAPIKey` },
            method: "POST",
            body: data,
        }
    );

    const result = await response.json();

    return result;
}

query("cat.jpg").then((response) => {
    console.log(JSON.stringify(response));
});

// OUTPUT Example

[{"score":0.6734483242034912,"label":"lynx, catamount"},{"score":0.18709655106067657,"label":"tabby, tabby cat"},{"score":0.06381674110889435,"label":"Egyptian cat"},{"score":0.05342644825577736,"label":"tiger cat"},{"score":0.004754822701215744,"label":"Siamese cat, Siamese"}]

// uses core.fetch, ERROR HAPPENS BECAUSE CORE KIT STRINGIFIES BODY
async function query2(filename: string) {
    const data = fs.readFileSync(filename);

    const response = await core.fetch({
        headers: { Authorization: `Bearer myHuggingFaceAPIKey` },
        method: "POST",
        body: data,
        url: "https://api-inference.huggingface.co/models/google/vit-base-patch16-224",

    })

    return response;
}

query2("cat.jpg").then((response) => {
    // write to file because error message is too long
    fs.writeFileSync("./response.txt", JSON.stringify(response))
    console.log(JSON.stringify(response));
});
// OUTPUT Example
{"$error":{"error":["Error in `inputs`: Invalid image: {'type': 'Buffer', 'data': [255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 255, 219, 0, 67, 0, 3, 2 ...]}}

[jig925]

I noticed this hotfix was reverted. Any ETA on when a new fix will be pushed? No rush though! Just out of curiosity.

[dglazkov]

Ah yes! I neglected to update this thread. It turns out that we need to be more careful here: we still need to stringify inputs that aren't a Blob, an ArrayBuffer, a TypedArray, a DataView, a FormData, a URLSearchParams, , or a ReadableStream object.

So this needs to be a more elaborate check, at least.

Also -- at the moment, I can't see how any of these types would be appearing as inputs to the fetch node. Do you have an example of a graph in mind where this problem happens?

[jig925]

I wanted to create a new toolworker board that could accept an audio file and then uses: https://huggingface.co/docs/api-inference/detailed_parameters#automatic-speech-recognition-task to produce a transcription of that audio file. That's how I discovered the issue because it was stringifying my audio file.

[dglazkov]

This is a really good use case. I am pretty sure we don't know how to handle binary data over the wire at the moment. It's all converted into text (base64 for images/audio).

[dglazkov]

Filed #1672

[jig925]

For context I am currently in the process of migrating the following board : https://github.com/ExaDev/breadboard/blob/develop/packages/example-boards/src/boards/playground/board-for-each.ts.

Currently trying to figure out how to achieve the same effect using the new passthrough node:

const passthrough = core.passthrough({
  $metadata: {
    title: "Passthrough",
  },
  board: input.board,
});
passthrough.board.to(passthrough);

const passthroughOutput = passthrough({ board: boardJSON });

I was able to manually do this looping on the visual editor by wiring the output back into the input, so it seems stil possible to do so. But Still trying to figure out how to do it in code.

[dglazkov]

I believe that can be done with loopback?
https://github.com/breadboard-ai/breadboard/tree/231244329176b873f67b3525dac87f4fa9641441/packages/build#cycles--loopbacks

@aomarks to confirm.

[aomarks]

Yep, loopback is the correct way to express a cycle like this with the build API.

[jig925]

Awesome, thanks for that!

[jig925]

I've attempted to migrate the Tour-guide-writer board to the build API. I've migrated it without using the core.curry and core.map as I am not sure if there are plans to migrate/deperecate them. I just wondered how I can format the output of the board? I was expecting the text prefixed with # to be in bold, just like the old board, but that doesn't seem to work!

code:
https://gist.github.com/jig925/2336df1776d0415eb6e4608748b7fb2e

[dglazkov]

Please do use curry and map. They are here to stay. What gave you a sense that they might be deprecated? Did I forget to update the code comments?

To format the output of the board, change the "format" of the output port to "Markdown".

[jig925]

Thanks I shall look into that. I was hesitant to use it as it was still part of the core kit, i.e core.map, core.curry and not like how core.invoke() is now just invoke(). But I'll look into using curry and map now.