tutorials/notebooks/3_Streamer-python-to-Bela.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# pybela Tutorial 3: Streamer – python to Bela\n",
        "This notebook is a tutorial for the Streamer class in the pybela python library. You can use the Streamer to stream data from Bela to python or viceversa. The complete documentation for the pybela library can be found in [https://belaplatform.github.io/pybela/](https://belaplatform.github.io/pybela/).\n",
        "\n",
        "In this tutorial we will be looking at sending data from python to Bela. There is only one method available in the Streamer class for this purpose: `send_buffer()`. This method sends a buffer of a certain type and size to Bela. \n",
        "\n",
        "To run this tutorial, first copy the `bela-code/bela2python2bela` project onto Bela. If your Bela is connected to your laptop, you can run the cell below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!rsync -rvL ../bela-code/bela2python2bela root@bela.local:Bela/projects"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Then you can compile and run the project using either the IDE or by running the following command in the Terminal:\n",
        "```bash\n",
        "ssh root@bela.local \"make -C Bela stop Bela PROJECT=bela2python2bela run\" \n",
        "```\n",
        "(Running this on a jupyter notebook will block the cell until the program is stopped on Bela.) \n",
        "\n",
        "This program expects two analog signals in channels 0 and 1, you can keep using the potentiometer setup from the previous tutorials (check the schematic in `1_Streamer-Bela-to-python.ipynb`)\n",
        "\n",
        "In this example we will be sending the values of the two potentiometers from Bela to python. Once received in python, we will send them immediately back to Bela. The values received in Bela will be used to modulate the amplitude of two sine waves. It is admittedly an overly complicated way to modulate two sine waves in Bela, as you could of course use the potentiometer values directly, without having to send them to python and back. However, this example can serve as a template for more complex applications where you can process the data in python before sending it back to Bela. \n",
        "\n",
        "## Understanding the Bela code\n",
        "If you are not familiar with auxiliary tasks and circular buffers, we recommend you follow first [Lesson 11](https://youtu.be/xQBftd7WNY8?si=ns6ojYnfQ_GVtCQI) and [Lesson 17](https://youtu.be/2uyWn8P0CVg?si=Ymy-NN_HKS-Q3xL0) of the C++ Real-Time Audio Programming with Bela course. \n",
        "\n",
        "Let's first take a look at the Bela code. The `setup()` function initializes the Bela program and some necessary variables. First, we set up the Watcher with the `Bela_getDefaultWatcherManager()` function. We then calculate the inverse of some useful variables (multiplying by the inverse is faster than dividing, so we precompute the inverse in `setup` and use it later in `render`). We then initialize the GUI buffers (these are the internal buffers Bela uses to receive the data) and the `circularBuffers`. The `circularBuffers` are used to store the parsed data from the GUI buffers, and are the variables we will use in `render` to access the data we have sent from python. We also set up the `binaryDataCallback` function, which will be called when Bela receives a buffer from python. \n",
        "\n",
        "\n",
        "```cpp\n",
        "bool setup(BelaContext* context, void* userData) {\n",
        "\n",
        "    Bela_getDefaultWatcherManager()->getGui().setup(context->projectName);\n",
        "    Bela_getDefaultWatcherManager()->setup(context->audioSampleRate); // set sample rate in watcher\n",
        "\n",
        "    gAudioFramesPerAnalogFrame = context->audioFrames / context->analogFrames;\n",
        "    gInvAudioFramesPerAnalogFrame = 1.0 / gAudioFramesPerAnalogFrame;\n",
        "    gInverseSampleRate = 1.0 / context->audioSampleRate;\n",
        "\n",
        "    // initialize the Gui buffers and circular buffers\n",
        "    for (int i = 0; i < NUM_OUTPUTS; ++i) {\n",
        "        Bela_getDefaultWatcherManager()->getGui().setBuffer('f', MAX_EXPECTED_BUFFER_SIZE);\n",
        "        circularBuffers[i].resize(circularBufferSize, 0.0f);\n",
        "        // the write index is given some \"advantage\" (prefillSize) so that the read pointer does not catch up the write pointer\n",
        "        circularBufferWriteIndex[i] = prefillSize % circularBufferSize;\n",
        "    }\n",
        "\n",
        "    Bela_getDefaultWatcherManager()->getGui().setBinaryDataCallback(binaryDataCallback);\n",
        "\n",
        "    // vars and preparation for parsing the received buffer\n",
        "    receivedBufferHeaderSize = sizeof(receivedBuffer.bufferId) + sizeof(receivedBuffer.bufferType) + sizeof(receivedBuffer.bufferLen) + sizeof(receivedBuffer.empty);\n",
        "    totalReceivedCount = 0;\n",
        "    receivedBuffer.bufferData.reserve(MAX_EXPECTED_BUFFER_SIZE);\n",
        "\n",
        "    return true;\n",
        "}\n",
        "```\n",
        "\n",
        "Let's now take a look at the `render()` function. The render function is called once per audio block, so inside of it we iterate over the audio blocks. Since the potentiometers are analog signals, and in Bela the analog inputs are typically sampled at a lower rate than the audio, we read the potentiometers once every 2 audio frames (in the code, `gAudioFramesPerAnalogFrame` is equal to 2 if you are using the default 8 audio channels). Since the variables `pot1` and `pot2` are in the Watcher, these will be streamed to python if we run `start_streaming()` in python.\n",
        "\n",
        "Next, we check if the variable `totalReceivedCount` is greater than 0, which means that we have received at least a buffer from python. If we have received buffers and the read pointer has not caught up with the write pointer, we advance the read pointer in the circular buffer. The reason why we check if we have received a buffer first, is because we don't want to advance the read pointer if we haven't received any data yet, as then the read pointer would catch up with the write pointer. \n",
        "\n",
        "Finally, we read the values from the circular buffer and use them to modulate the amplitude of two sine waves. We then write the output to the audio channels.\n",
        "\n",
        "\n",
        "\n",
        "```cpp\n",
        "\n",
        "void render(BelaContext* context, void* userData) {\n",
        "    for (unsigned int n = 0; n < context->audioFrames; n++) {\n",
        "        uint64_t frames = context->audioFramesElapsed + n;\n",
        "\n",
        "        if (gAudioFramesPerAnalogFrame && !(n % gAudioFramesPerAnalogFrame)) {\n",
        "            Bela_getDefaultWatcherManager()->tick(frames * gInvAudioFramesPerAnalogFrame); // watcher timestamps\n",
        "\n",
        "            // read sensor values and put them in the watcher\n",
        "            pot1 = analogRead(context, n / gAudioFramesPerAnalogFrame, gPot1Ch);\n",
        "            pot2 = analogRead(context, n / gAudioFramesPerAnalogFrame, gPot2Ch);\n",
        "\n",
        "            // read the values sent from python (they're in the circular buffer)\n",
        "            for (unsigned int i = 0; i < NUM_OUTPUTS; i++) {\n",
        "\n",
        "                if (totalReceivedCount > 0 && (circularBufferReadIndex[i] + 1) % circularBufferSize != circularBufferWriteIndex[i]) {\n",
        "                    circularBufferReadIndex[i] = (circularBufferReadIndex[i] + 1) % circularBufferSize;\n",
        "                } else if (totalReceivedCount > 0) {\n",
        "                    rt_printf(\"The read pointer has caught the write pointer up in buffer %d – try increasing prefillSize\\n\", i);\n",
        "                }\n",
        "            }\n",
        "        }\n",
        "\n",
        "        float amp1 = circularBuffers[0][circularBufferReadIndex[0]];\n",
        "        float amp2 = circularBuffers[1][circularBufferReadIndex[1]];\n",
        "\n",
        "        float out = amp1 * sinf(gPhase1) + amp2 * sinf(gPhase2);\n",
        "\n",
        "        for (unsigned int channel = 0; channel < context->audioOutChannels; channel++) {\n",
        "            audioWrite(context, n, channel, out);\n",
        "        }\n",
        "\n",
        "        gPhase1 += 2.0f * (float)M_PI * gFrequency1 * gInverseSampleRate;\n",
        "        if (gPhase1 > M_PI)\n",
        "            gPhase1 -= 2.0f * (float)M_PI;\n",
        "        gPhase2 += 2.0f * (float)M_PI * gFrequency2 * gInverseSampleRate;\n",
        "        if (gPhase2 > M_PI)\n",
        "            gPhase2 -= 2.0f * (float)M_PI;\n",
        "\n",
        "    }\n",
        "}\n",
        "```\n",
        "\n",
        "Let's now run the python code:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pybela import Streamer\n",
        "streamer = Streamer()\n",
        "streamer.connect()\n",
        "\n",
        "variables = [\"pot1\", \"pot2\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The `send_buffer` function takes 4 arguments: the buffer id, the type of the data that goes in the buffer, the buffer length and the buffer data. Since we will be sending back the buffers we receive from Bela, we can get the type and length of the buffer through the streamer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "buffer_type = streamer.get_prop_of_var(\"pot1\", \"type\")\n",
        "buffer_length = streamer.get_prop_of_var(\"pot1\", \"data_length\")\n",
        "\n",
        "buffer_type, buffer_length\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Here we will be using the `block_callback` instead of the `buffer_callback`, as the `block` callback is more efficient. It should be noted that we are receiving and sending blocks of data every 1024/22050 = 0.05 seconds, and the maximum latency is given by the `prefillSize` variable in the Bela code (which is set to 2.5*1024/22050 = 0.12 seconds), so using functions is crucial to meet the real-time deadlines."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "def callback(block):\n",
        "    \n",
        "    for buffer in block:\n",
        "    \n",
        "        _var = buffer[\"name\"]\n",
        "        timestamp = buffer[\"buffer\"][\"ref_timestamp\"]\n",
        "        data = buffer[\"buffer\"][\"data\"]\n",
        "        \n",
        "        buffer_id = 0 if _var == \"pot1\" else 1\n",
        "\n",
        "        print(buffer_id, timestamp)\n",
        "        # do some data processing here...\n",
        "        processed_data = data\n",
        "        \n",
        "        # send processed_data back\n",
        "        streamer.send_buffer(buffer_id, buffer_type,\n",
        "                                        buffer_length, processed_data)\n",
        "\n",
        "streamer.start_streaming(\n",
        "    variables, saving_enabled=False, on_block_callback=callback)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you plug in your headphones to the audio output of Bela, you should hear two sine waves modulated by the potentiometers. The modulation (the amplitude change) is given by the value sent by python, not the analog input directly on Bela. As mentioned before, this is an overly complicated way to modulate two sine waves, but it can serve as a template for more complex applications where you can process the data in python before sending it back to Bela."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "streamer.stop_streaming()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}