I2S + INMP441 microphone + no_std + embassy + ESP32S3: Digital chirps on Rust esp-hal?

[brainstorm]

First of all, thanks in advance to anybody reading this for your patience, support and expertise. Here's what I'm playing with recently:

And here's the example code, it essentially reads the data from I2S0 and dumps it to the UART port:

#![no_std]
#![no_main]

use embassy_executor::Spawner;
use embedded_io::Write;
use esp_backtrace as _;
use esp_hal::{
    clock::ClockControl,
    dma::{Dma, DmaPriority},
    dma_buffers, embassy,
    gpio::Io,
    i2s::{asynch::I2sReadDmaAsync, DataFormat, I2s, Standard},
    peripherals::Peripherals,
    prelude::*,
    system::SystemControl,
    timer::timg::TimerGroup,
    uart::{config::Config, TxRxPins, Uart},
};

// DMA buffer size
const I2S_BYTES: usize = 4096;

#[main]
async fn main(_spawner: Spawner) {
    // Prepare all the peripherals and clocks
    let peripherals = Peripherals::take();
    let system = SystemControl::new(peripherals.SYSTEM);
    let clocks = ClockControl::boot_defaults(system.clock_control).freeze();
    let timg0 = TimerGroup::new_async(peripherals.TIMG0, &clocks);
    let io = Io::new(peripherals.GPIO, peripherals.IO_MUX);
    let dma = Dma::new(peripherals.DMA);
    let dma_channel = dma.channel0;

    embassy::init(&clocks, timg0);

    // Set DMA buffers
    let (_, mut tx_descriptors, dma_rx_buffer, mut rx_descriptors) = dma_buffers!(0, I2S_BYTES * 4);

    // I2S settings
    let i2s = I2s::new(
        peripherals.I2S0,
        Standard::Philips,
        DataFormat::Data16Channel16,
        16000u32.Hz(),
        dma_channel.configure_for_async(
            false,
            &mut tx_descriptors,
            &mut rx_descriptors,
            DmaPriority::Priority0,
        ),
        &clocks,
    );

    let i2s_rx = i2s
        .i2s_rx
        .with_bclk(io.pins.gpio2)
        .with_ws(io.pins.gpio4)
        .with_din(io.pins.gpio5)
        .build();

    // UART settings to egress audio bytes as fast as possible (vs the ::default() 115200 baudrate)
    let uart_config = Config::default().baudrate(921_600);
    // On Lolin S3 1.0.0 board, UART port wired to USB2TTL IC, not OTG.
    let uart_pins = Some(TxRxPins::new_tx_rx(io.pins.gpio43, io.pins.gpio44));
    let uart0 = Uart::new_async_with_config(peripherals.UART0, uart_config, uart_pins, &clocks);
    let (mut uart_tx, _rx) = uart0.split();

    // I2S transactions to DMA buffers
    let mut i2s_data = [0u8; I2S_BYTES];
    let mut transaction = i2s_rx.read_dma_circular_async(dma_rx_buffer).unwrap();

    loop {
        let _i2s_bytes_read = transaction.pop(&mut i2s_data).await.unwrap();
        uart_tx.write_all(&i2s_data).unwrap();
    }
}

The command line I use to dump that data is just cargo run --release > data/mic_espressif.data, then I just import it to Audacity, taking into account to skip ~3000 bytes from the ESP32S3 bootlog, before it starts to spill the I2S data from the DMA buffer. The following Audacity screenshot contains 3 captured waveforms from 3 different sources/sessions:

1. Google translate's TTS saying "Espressif" on a loop.
2. An arduino sketch from ThatProject which is calling ESP-IDF underneath and transmitting the 1024 byte I2S samples through WebSockets. I recovered and stitched together the UDP packets via Wireshark (follow conversation).
3. The esp_hal Rust snippet outlined above.

So to be clear, 2 and 3 represent the same audio from 1 on a loop: as pedestrian of a setup as putting the microphone close to the computer's speaker as the "Espressif" TTS is going on.

Thus, comparing the last two waveforms and spectogram, you can easily tell that the data produced by the Arduino sketch replays the recorded message and the last one, produced by esp-hal snippet, is mostly digital noise.

So looking at the bytes with Radare2 at an arbitrary offset in the I2S stream (representative sample), there are clearer byte-level differences between both implementations:

As you can see in the Rust snippet above, I'm using the Standard::Philips mode, which the Arduino sketch is also using via its .communication_format ESP-IDF struct attribute I2S_COMM_FORMAT_STAND_I2S in this line... I've also matched 16 bits and 16KHz to compare apples with apples... interestingly, I've set the Arduino sketch settings to 16KHz, but I need to load it at 8KHz in audacity, however that does not affect much to the actual bytes/data, it just sounds faster/slower when there's a sample rate mismatch, so that's not the issue.

So, having established that the Arduino sketch works as intended with the hardware setup and wiring (ruling that PEBKAC out)... what am I doing wrong the the equivalent esp-hal+embassy snippet above?

Could it be related with how the UART spits out the raw bytes on the wire, not respecting the raw bytes from the DMA buffer as-is? My TTY adding some major garbage to the raw data when doing that > redirection? Some embassy scheduling issue? Wrong buffer sizes on my DMA buffers or indexing? None of the above?

Happy to share the raw data files on request if that helps.

Thanks again in advance!

[bjoernQ]

I did the same thing some time ago successfully but maybe something broke since then - needs checking

[bjoernQ]

Sorry it took me so long to finally get to it. I unfortunately deleted my former experiments so I had to redo it

First of all I have no idea why the Arduino code works for you. But I finally have something that works for me now.

Here is my code running on ESP32-C3 (S3 shouldn't make a difference). This is non-async code however. It's basically the i2s_read example with some modifications. Before it starts to stream the data via UART it waits for a "go" from the host (see the host side code below)

//! BCLK    GPIO2
//! WS      GPIO4
//! DIN     GPIO5

//% CHIPS: esp32 esp32c3 esp32c6 esp32h2 esp32s2 esp32s3

#![no_std]
#![no_main]

use esp_backtrace as _;
use esp_hal::{
    clock::ClockControl,
    dma::{Dma, DmaPriority},
    dma_buffers,
    gpio::Io,
    i2s::{DataFormat, I2s, I2sReadDma, Standard},
    peripherals::Peripherals,
    prelude::*,
    system::SystemControl, uart::{config::Config, TxRxPins, Uart},
};
use esp_println::println;

#[entry]
fn main() -> ! {
    let peripherals = Peripherals::take();
    let system = SystemControl::new(peripherals.SYSTEM);
    let clocks = ClockControl::max(system.clock_control).freeze();

    let io = Io::new(peripherals.GPIO, peripherals.IO_MUX);

    let uart_config = Config::default().baudrate(921_600);
    let uart_pins = Some(TxRxPins::new_tx_rx(io.pins.gpio21, io.pins.gpio20));
    let mut uart0 = Uart::new_with_config(peripherals.UART0, uart_config, uart_pins, &clocks, None);


    let dma = Dma::new(peripherals.DMA);
    #[cfg(any(feature = "esp32", feature = "esp32s2"))]
    let dma_channel = dma.i2s0channel;
    #[cfg(not(any(feature = "esp32", feature = "esp32s2")))]
    let dma_channel = dma.channel0;

    let (_, mut tx_descriptors, mut rx_buffer, mut rx_descriptors) = dma_buffers!(0, 4 * 4092);

    let i2s = I2s::new(
        peripherals.I2S0,
        Standard::Philips,
        DataFormat::Data32Channel32,
        8000.Hz(),
        dma_channel.configure(
            false,
            &mut tx_descriptors,
            &mut rx_descriptors,
            DmaPriority::Priority0,
        ),
        &clocks,
    );

    let mut i2s_rx = i2s
        .i2s_rx
        .with_bclk(io.pins.gpio2)
        .with_ws(io.pins.gpio4)
        .with_din(io.pins.gpio5)
        .build();

    let r = nb::block!(uart0.read_byte()).unwrap();

    let mut transfer = i2s_rx.read_dma_circular(&mut rx_buffer).unwrap();

    loop {
        let avail = transfer.available();

        if avail > 0 {
            let mut rcv = [0u8; 4092*2];
            transfer.pop(&mut rcv[..avail]).unwrap();
            uart0.write_bytes(&rcv[..avail]);
            uart0.flush_tx();
        }
    }
}

On the host side the only dependency I added is serialport = "4.3.0"

use std::{io::Write, time::Duration};

fn main() {
    let mut file = std::fs::File::create("./out.raw").unwrap();

    // change "com24" to match your port
    let mut port = serialport::new("com24", 921_600).timeout(Duration::from_millis(500)).open().unwrap();

    for _ in 0..2 {
        let mut serial_buf: Vec<u8> = vec![0; 15000];
        let len = port.read(serial_buf.as_mut_slice());
    }

    port.write(&[b'S']).unwrap();

    loop {
        let mut serial_buf: Vec<u8> = vec![0; 15000];
        let len = port.read(serial_buf.as_mut_slice()).expect("Found no data!");
        file.write_all(&serial_buf[..len]).unwrap();
        println!("{len}");
    }
}

I did the "wait for the host" thing because knowing where the useable data starts is not that easy otherwise

From INMP441's datasheet:

I really don't know why the Arduino code works with 16 bits but using Data16Channel16 and 16 kHz looks really bad in the logic analyzer trace:

See how the right channel (which should be silent) contains garbage.

I experimented a bit but only could get the INMP441 to produce useful signal on DIN with Data32Channel32 at 8KHz:

Seems to me the esp-hal's driver is doing the right thing given what the signals look like

[brainstorm]

Amazing, thanks for your efforts @bjoernQ!

I just tested your sync version and I'm having exactly the same issues, I hope it's not related to OSX USB2TTL nonsense :/

To be sure, I'll change the host (I'm doing all this on a Mac, will try with a Linux PC, hopefully tomorrow). I have a couple of followup questions before that:

1. How exactly are you running this? Just cargo run --release on two separate terminals? I'm running into "resource busy" issues with the serial port if I do that :-S
2. Did you put a 100KOhm resistor between SD and GND as the datasheet suggests? My waveforms looked really bad on the oscilloscope before doing that (CLK and WS were fine, just SD).

PS: Also I tried to run it on my ESP32-C3 board for completeness but I have an old v0.2 of the silicon (pre-prod giveaway from your CEO), so bad luck... I wish I could apply the old linker esp32c3.rom.ld fix to bypass that validation easily in esp-hal but I don't know how except, I guess, re-compiling the toolchain (and its secondary bootloader silicon rev validation check)... at the moment my C3 board just bootloops like this after flashing:

I (34) boot: ESP-IDF v5.1.2-342-gbcf1645e44 2nd stage bootloader
I (35) boot: compile time Dec 12 2023 10:50:58
I (35) boot: chip revision: v0.2
E (39) boot_comm: Image requires chip rev >= v0.3, but chip is v0.2      <------ !!!!
ESP-ROM:esp32c3-20200918
Build:Sep 18 2020
rst:0x3 (RTC_SW_SYS_RST),boot:0xc (SPI_FAST_FLASH_BOOT)
Saved PC:0x400483a0
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fcd5820,len:0x1714
load:0x403cc710,len:0x968
load:0x403ce710,len:0x2f9c
entry 0x403cc710
I (34) boot: ESP-IDF v5.1.2-342-gbcf1645e44 2nd stage bootloader
I (35) boot: compile time Dec 12 2023 10:50:58
(...)

[brainstorm]

also even if it's hardcoded at 8000Hz, I find myself adjusting it on Audacity to 16000Hz for it to hear the original sound properly???... might point to a hardware initialization issue?

@bjoernQ that. After getting my setup a bit more robust (soldered, no loose wires, no resistor needed) and getting rid of the spurious digital garbage. I'm observing that the bits and samplerate are quite off? For instance, setting those:

        DataFormat::Data16Channel16,
        16000.Hz(),

Yields audio data when imported to audacity, but the 16000Hz sample rate is way off... only an Audacity setting of ~30000Hz sounds closest to the original audio (neither the standard 22050Hz (too slow) nor 44100Hz (too fast, sounds like Speedy Gonzales)).

Which brings me to one of your earlier comments:

I experimented a bit but only could get the INMP441 to produce useful signal on DIN with Data32Channel32 at 8KHz

So I'm starting to suspect that (perhaps!) some register config/initialization/clock/pll within the I2S HAL is off somehow? Why would there be such a discrepancy between the firmware settings and Audacity's sample rate setting otherwise? ... other than an Audacity bug, fair enough.

For instance, can you help me interpret which registers and fields are being set here, specially the ones marked with ?? in the line comments?:

esp-hal/esp-hal/src/i2s.rs

Lines 1039 to 1055 in 2fd1594

	.clear_bit()
	.tx_msb_shift()
	.set_bit() // ?
	.rx_msb_shift()
	.set_bit() // ?
	.tx_short_sync()
	.bit(false) //??
	.rx_short_sync()
	.bit(false) //??
	.tx_msb_right()
	.clear_bit()
	.rx_msb_right()
	.clear_bit()
	.tx_right_first()
	.clear_bit()
	.rx_right_first()
	.clear_bit()

Thanks in advance!

[bjoernQ]

I think those settings were initially set to what was found in esp-idf - the TRM ( https://www.espressif.com/sites/default/files/documentation/esp32-s3_technical_reference_manual_en.pdf ) should explain them

Probably you are more interested in set_clock and calculate_clock

BTW in your Audacity screenshots it's using "mono" - did you remove the right channel in Audacity manually? Just asking since importing the stereo stream as mono might explain a factor 2 off sample rate

[brainstorm]

BTW in your Audacity screenshots it's using "mono" - did you remove the right channel in Audacity manually? Just asking since importing the stereo stream as mono might explain a factor 2 off sample rate

Yesss!!! You are absolutely right about importing it as stereo, thanks! 🤦🏻‍♂️ Sorry for doubting your HAL/clock initialization bits, I'm just trying to dissect this and learn as fast as I can to not burden you more than needed.

Regarding my initial async/embassy snippet... Is it possible to tweak the executor to not chop the waveform like this? Upper waveforms are with your sync code, lower waveforms are with the embassy code:

I initially wanted to have this all working under embassy so that I can write individual tasks that get preempted well... is it possible/doable with an application like this (audio)? Perhaps defining separate tasks for I2S & UART would help instead of putting it all on the main thread or is this unlikely to work anyway w/ async? Here's the current embassy code I originally posted:

#![no_std]
#![no_main]

use embassy_executor::Spawner;
use embedded_io::Write;
use esp_backtrace as _;
use esp_hal::{
    clock::ClockControl,
    dma::{Dma, DmaPriority},
    dma_buffers, embassy,
    gpio::Io,
    i2s::{DataFormat, I2s, Standard, asynch::I2sReadDmaAsync},
    peripherals::Peripherals,
    prelude::*,
    system::SystemControl,
    timer::timg::TimerGroup,
    uart::{config::Config, TxRxPins, Uart},
};

// DMA buffer size
const I2S_BYTES: usize = 4092;

#[main]
async fn main(_spawner: Spawner) {
    // Prepare all the peripherals and clocks
    let peripherals = Peripherals::take();
    let system = SystemControl::new(peripherals.SYSTEM);
    let clocks = ClockControl::boot_defaults(system.clock_control).freeze();
    let timg0 = TimerGroup::new_async(peripherals.TIMG0, &clocks);
    let io = Io::new(peripherals.GPIO, peripherals.IO_MUX);
    let dma = Dma::new(peripherals.DMA);
    let dma_channel = dma.channel0;

    embassy::init(&clocks, timg0);

    // Set DMA buffers
    let (_, mut tx_descriptors, dma_rx_buffer, mut rx_descriptors) = dma_buffers!(0, I2S_BYTES * 4);

    // I2S settings
    let i2s = I2s::new(
        peripherals.I2S0,
        Standard::Philips,
        DataFormat::Data32Channel32,
        8000.Hz(),
        dma_channel.configure_for_async(
            false,
            &mut tx_descriptors,
            &mut rx_descriptors,
            DmaPriority::Priority0,
        ),
        &clocks,
    );

    let i2s_rx = i2s
        .i2s_rx
        .with_bclk(io.pins.gpio2)
        .with_ws(io.pins.gpio4)
        .with_din(io.pins.gpio5)
        .build();

    // UART settings, egress bytes as fast as possible
    let uart_config = Config::default().baudrate(921_600);
    // On Lolin S3 1.0.0, no ::default() for this
    let uart_pins = Some(TxRxPins::new_tx_rx(io.pins.gpio43, io.pins.gpio44));
    let uart0 = Uart::new_async_with_config(peripherals.UART0, uart_config, uart_pins, &clocks);
    let (mut uart_tx, _rx) = uart0.split();

    // I2S transactions to DMA buffers
    let mut i2s_data = [0u8; I2S_BYTES];
    let mut transaction = i2s_rx.read_dma_circular_async(dma_rx_buffer).unwrap();

    loop {
        let _i2s_bytes_read = transaction.pop(&mut i2s_data).await.unwrap();
        uart_tx.write_all(&i2s_data).unwrap();
    }
}

Regardless of my personal project ramblings here, I think this could turn into a good esp-hal async/embassy audio example once it works well and PR accordingly? Happy to keep pushing it to attain that and somehow repay your effort and attention to this.

[bjoernQ]

Sorry for doubting your HAL/clock initialization bits, I'm just trying to dissect this ...

No problem, really! There is always a chance to hit a bug.

What you could try is changing the loop to

    loop {
        let i2s_bytes_read = transaction.pop(&mut i2s_data).await.unwrap();
        uart_tx.write_async(&i2s_data[..i2s_bytes_read]).await.unwrap();
    }

Probably pop will get much less data than the buffer can hold. Writing the whole buffer would result in writing the actual data and a lot of zeros. Using the async-version of write probably doesn't help but would be the better option in general in async code

I haven't tried the code yet but at least that would explain how the screenshot looks like

[brainstorm]

Probably pop will get much less data than the buffer can hold. Writing the whole buffer would result in writing the actual data and a lot of zeros. Using the async-version of write probably doesn't help but would be the better option in general in async code

Yay, it works, thanks @bjoernQ! For the record, here's how a healthy waveform looks like (async2) vs a digital garbage one (async3)... seems to be just a shift in bytes positions, I have no idea why other than improving the quality of the sensor wiring and electrical conditioning... could be also related to timing on the device-to-host UART "go signal"... happy to hear other ideas, but otherwise this async example is finally working with async/embassy!

Next stop is getting rid of the 2nd stage bootloader bootlog messages, removing the need for host tooling, but I'll open another issue if I cannot figure that out, thanks heaps again!