feat(sc): enable ReSpeaker Lite support and fix sample streaming #2

Wkstr · 2025-09-18T08:39:19Z

Overview:

This PR delivers support for the Seeed ReSpeaker Lite board within the sound_classification example, addressing issues across the entire data pipeline from sensor to output.

Key changes:

Sensor Driver Adaptation (SensorI2SXMOS):
- The I2S slot configuration has been updated.This ensures the driver correctly captures audio from the appropriate channel on the ReSpeaker's stereo XMOS chip, resolving issues with silent or noisy recordings.
Sample Rate Mismatch for Inference:
- The ReSpeaker Lite's microphone operates at 16kHz, while the feature extraction pipeline expects a 44.1kHz input. This is resolved by modifying the SpeechCommandsPreprocess node. When compiled with BOARD_RESPEAKER_LITE, the node now performs memory-efficient, real-time, chunk-by-chunk upsampling from 16kHz to 44.1kHz. This is done on the stack within the processing loop, avoiding large heap allocations while correctly maintaining the overlapping window state required for accurate feature extraction.
Opus Encoding Failure for Sampling:
- The AT+START=sample command failed to produce valid audio on the ReSpeaker Lite. The root cause was that the TaskSC::Sample task was incorrectly configuring the Opus encoder to 48kHz instead of the sensor's native 16kHz.

iChizer0 · 2025-09-18T08:44:50Z

components/acoustics-porting/CMakeLists.txt

    endif()
 endif()

+if(SOUND_BOARD STREQUAL "respeaker_lite")


BOARD_RESPEAKER_LITE -> PORTING_BOARD_MODEL_XXX
e.g. PORTING_BOARD_MODEL_RESPEAKER_LITE, PORTING_BOARD_MODEL_XIAO_S3

iChizer0 · 2025-09-18T08:46:17Z

examples/sound_classification/main/CMakeLists.txt

+set(SOUND_BOARD "" CACHE STRING "Board selection: empty(default)|respeaker_lite")
+if(SOUND_BOARD STREQUAL "respeaker_lite")
+    set(BOARD_RESPEAKER_LITE ON CACHE BOOL "Enable ReSpeaker Lite board support" FORCE)
+    message(STATUS "Board for sound_sampler: ReSpeaker Lite enabled")
+else()
+    set(BOARD_RESPEAKER_LITE OFF CACHE BOOL "Enable ReSpeaker Lite board support" FORCE)
+    message(STATUS "Board for sound_sampler: default (XIAO ESP32S3 mic)")
+endif()
+


e.g. ACOUSTICS_PORTING_BOARD_MODEL_RESPEAKER_LITE

…rsion Refactor the I2S driver configuration to perform the 32-bit to 16-bit audio sample conversion directly in hardware via DMA.

iChizer0 · 2025-09-24T12:15:50Z

components/acoustics-porting/CMakeLists.txt

+if(NOT DEFINED PORTING_BOARD_MODEL_RESPEAKER_LITE)
+    set(PORTING_BOARD_MODEL_RESPEAKER_LITE 0)
+endif()
+if(NOT DEFINED PORTING_BOARD_MODEL_XIAO_S3)
+    set(PORTING_BOARD_MODEL_XIAO_S3 1)  
+endif()
+
+if(PORTING_BOARD_MODEL_RESPEAKER_LITE)
+    message(STATUS "acoustics-porting: ReSpeaker Lite board support ENABLED.")
+elseif(PORTING_BOARD_MODEL_XIAO_S3)
+    message(STATUS "acoustics-porting: XIAO ESP32S3 board support ENABLED (default).")
+else()
+    message(STATUS "acoustics-porting: No specific board model enabled.")
+endif()


这里这样写缺少 mutual exclusivity 和 scalability，之后改用 menuconfig 来控制吧，也可以参考下面这种写法：

if(NOT DEFINED PORTING_BOARD_MODEL) set(PORTING_BOARD_MODEL "XIAO_S3" CACHE STRING "Select the target board model (e.g., XIAO_S3, RESPEAKER_LITE)") endif() set(SUPPORTED_BOARDS "XIAO_S3" "RESPEAKER_LITE") if(NOT ${PORTING_BOARD_MODEL} IN_LIST SUPPORTED_BOARDS) message(FATAL_ERROR "Unsupported board model: '${PORTING_BOARD_MODEL}'. " "Supported models are: ${SUPPORTED_BOARDS}") endif() if(${PORTING_BOARD_MODEL} STREQUAL "RESPEAKER_LITE") message(STATUS "${PROJECT_NAME}: ReSpeaker Lite board support ENABLED.") # ... set flags for ReSpeaker Lite elseif(${PORTING_BOARD_MODEL} STREQUAL "XIAO_S3") message(STATUS "${PROJECT_NAME}: XIAO ESP32S3 board support ENABLED (default).") # ... set flags for XIAO S3 endif()

iChizer0 · 2025-09-24T12:18:51Z

components/acoustics-porting/porting/device_esp32s3.cpp

+#if defined(PORTING_BOARD_MODEL_RESPEAKER_LITE) && PORTING_BOARD_MODEL_RESPEAKER_LITE
+static constexpr const char DEVICE_NAME[] = "ReSpeaker Lite (XIAO ESP32S3)";
+#elif defined(PORTING_BOARD_MODEL_XIAO_S3) && PORTING_BOARD_MODEL_XIAO_S3
 static constexpr const char DEVICE_NAME[] = "XIAO ESP32-S3";
+#else
+static constexpr const char DEVICE_NAME[] = "XIAO ESP32-S3";
+#endif


这的默认的逻辑建议直接 #error，另外组织形式建议修改一下：
新建 board 文件夹，里面仅仅包含不同型号开发板命名的头文件，头文件中存放配置信息，配置信息外予以宏保护

iChizer0 · 2025-09-24T12:34:39Z

components/acoustics-porting/porting/sensor/i2sxmos.hpp

整体上看了下，idf 驱动接口能应对目前的需求，主要是初始化的地方不一样，那这个实现可以和 i2smic 合并：

采用类似上述分 board 的思路，为不同的开发板提供不同的默认配置信息

参考 1，为不同的开发版提供不同的 i2s_channel 初始化函数
另外需要检查确认一下 available 接口的行为是不是正常的

iChizer0 · 2025-09-24T12:35:31Z

components/acoustics-porting/porting/sensor_esp32s3.cpp

+#if defined(PORTING_BOARD_MODEL_RESPEAKER_LITE) && PORTING_BOARD_MODEL_RESPEAKER_LITE
+#include "sensor/i2sxmos.hpp"
+#endif

 namespace bridge {

 void __REGISTER_SENSORS__()
 {
    [[maybe_unused]] static porting::SensorLIS3DHTR sensor_lis3dhtr;
+#if defined(PORTING_BOARD_MODEL_RESPEAKER_LITE) && PORTING_BOARD_MODEL_RESPEAKER_LITE
+    [[maybe_unused]] static porting::SensorI2SXMOS sensor_i2sxmos;
+#elif defined(PORTING_BOARD_MODEL_XIAO_S3) && PORTING_BOARD_MODEL_XIAO_S3
    [[maybe_unused]] static porting::SensorI2SMic sensor_i2smic;
+#else
+    [[maybe_unused]] static porting::SensorI2SMic sensor_i2smic;
+#endif


这里驱动更新后也请相应修改一下

iChizer0 · 2025-09-24T12:37:55Z

components/acoustics/api/v1/task_sc.hpp

+static inline void resampleLinearChunk(const int16_t *in_data, size_t in_size, int16_t *out_data, size_t out_start_pos,
+    size_t out_chunk_size, size_t in_rate, size_t out_rate) noexcept
+{
+    if (!in_data || !out_data || !in_size || !out_chunk_size || !in_rate || !out_rate)
+        return;
+
+    double ratio = static_cast<double>(in_rate) / static_cast<double>(out_rate);
+
+    for (size_t i = 0; i < out_chunk_size; ++i)
+    {
+        size_t out_pos = out_start_pos + i;
+        double in_pos = static_cast<double>(out_pos) * ratio;
+        size_t index1 = static_cast<size_t>(in_pos);
+        size_t index2 = index1 + 1;
+
+        if (index1 >= in_size)
+        {
+            out_data[i] = 0;
+            continue;
+        }
+
+        float fraction = static_cast<float>(in_pos - static_cast<double>(index1));
+        int16_t s1 = in_data[index1];
+        int16_t s2 = (index2 < in_size) ? in_data[index2] : s1;
+        int32_t interp = static_cast<int32_t>(std::roundf((1.0f - fraction) * s1 + fraction * s2));
+        out_data[i] = static_cast<int16_t>(std::clamp(interp, static_cast<int32_t>(std::numeric_limits<int16_t>::min()),
+            static_cast<int32_t>(std::numeric_limits<int16_t>::max())));
+    }
+}
+


这个函数我之后会可能会整理到 utils 中

iChizer0 · 2025-09-24T12:46:17Z

components/acoustics/api/v1/task_sc.hpp

    inline constexpr const float overlap_ratio_max = 0.6f;
    inline std::atomic<float> overlap_ratio = 0.5f;

+    inline std::atomic<size_t> sensor_sample_rate = 0;


sensor_sample_rate 建议用 int 类型，如没有与其他组件共享存在并发问题，没有必要使用 atomic，放在使用方内部即可

iChizer0 · 2025-09-24T12:48:12Z

components/acoustics/api/v1/task_sc.hpp

                available = head - tail;
-                if (available < required) [[unlikely]]
+
+                const size_t sensor_sample_rate = shared::sensor_sample_rate.load();


这里是一个 shared 的变量，使用前建议先检查变量的合法性（即使有其它 flag 帮你规避了并发问题）

iChizer0 · 2025-09-25T02:00:51Z

components/acoustics/api/v1/task_sc.hpp

+                if (sensor_sample_rate == 44100)
                {
-                    return executor.submit(getptr(), shared::invoke_pull_ms);
+                    if (available < required) [[unlikely]]
+                    {
+                        return executor.submit(getptr(), shared::invoke_pull_ms);
+                    }
+                    for (size_t i = 0; i < required; ++i)
+                    {
+                        _input->data<int16_t>()[i] = shared::buffer[(tail + i) & shared::buffer_size_mask];
+                    }
+                    shared::buffer_tail = tail + std::min(to_discard, available);
                }
-                for (size_t i = 0; i < required; ++i)
+                else
                {
-                    _input->data<int16_t>()[i] = shared::buffer[(tail + i) & shared::buffer_size_mask];
+                    const size_t required_sensor = (required * sensor_sample_rate) / 44100;
+                    const size_t to_discard_sensor
+                        = static_cast<size_t>(std::round(required_sensor * (1.f - shared::overlap_ratio.load())));
+
+                    if (available < required_sensor) [[unlikely]]
+                    {
+                        return executor.submit(getptr(), shared::invoke_pull_ms);
+                    }
+
+                    std::vector<int16_t> temp_sensor_data(required_sensor);
+                    for (size_t i = 0; i < required_sensor; ++i)
+                    {
+                        temp_sensor_data[i] = shared::buffer[(tail + i) & shared::buffer_size_mask];
+                    }
+
+                    const size_t chunk_size = 1024;
+                    for (size_t chunk_start = 0; chunk_start < required; chunk_start += chunk_size)
+                    {
+                        size_t current_chunk_size = std::min(chunk_size, required - chunk_start);
+                        resampleLinearChunk(temp_sensor_data.data(), required_sensor,
+                            _input->data<int16_t>() + chunk_start, chunk_start, current_chunk_size, sensor_sample_rate,
+                            44100);
+                    }
+
+                    shared::buffer_tail = tail + std::min(to_discard_sensor, available);


这里存在比较严重的问题：

在 executor 的循环 loop 中避免 temp_sensor_data 这种需要频繁分配内存的方式

数据拷贝到 temp_sensor_data 后，剩余的操作都是不存在并发问题的，但是你剩余的操作却是在 buffer 的锁里完成的

chunk 的设计是何意味？这里这样写不仅没有减少内存占用，反而还增加了计算量

避免多处 44100 这种 hard code 的写法，用宏或者静态常量表示算法需要的样本数量（其实这里都没必要写成两个分支）

Wkstr added 2 commits September 18, 2025 16:14

feat(sc): add ReSpeaker Lite‘s support

273d2fa

fix(sc): adapt sample stream

c9208da

iChizer0 reviewed Sep 18, 2025

View reviewed changes

Wkstr added 3 commits September 19, 2025 11:51

chore: rename compile options

87243b3

feat(SensorI2SXMOS): implement hardware-accelerated 32-to-16bit conve…

ccd5f07

…rsion Refactor the I2S driver configuration to perform the 32-bit to 16-bit audio sample conversion directly in hardware via DMA.

refactor: move upsampling to API

648cbfa

iChizer0 reviewed Sep 24, 2025

View reviewed changes

iChizer0 reviewed Sep 25, 2025

View reviewed changes

merge: sync with main to incorporate latest updates

0b16075

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(sc): enable ReSpeaker Lite support and fix sample streaming #2

feat(sc): enable ReSpeaker Lite support and fix sample streaming #2

Uh oh!

Wkstr commented Sep 18, 2025

Uh oh!

iChizer0 Sep 18, 2025

Uh oh!

iChizer0 Sep 18, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 24, 2025

Uh oh!

iChizer0 Sep 25, 2025

Uh oh!

Uh oh!

feat(sc): enable ReSpeaker Lite support and fix sample streaming #2

Are you sure you want to change the base?

feat(sc): enable ReSpeaker Lite support and fix sample streaming #2

Uh oh!

Conversation

Wkstr commented Sep 18, 2025

Overview:

Key changes:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!