Mastering ESP32 Arduino Queues: Dual-Core Sensor Data Pipeline

When you push an ESP32 beyond blinking LEDs and start reading high-frequency sensors while simultaneously streaming data over WiFi, the single-threaded loop() paradigm falls apart. You need multithreading, and the moment you have multiple threads touching the same data, you need Inter-Process Communication (IPC). This is where ESP32 Arduino queues—specifically FreeRTOS queues under the hood—become mandatory.

A FreeRTOS queue allows you to pass data safely between tasks running on different cores without race conditions. You allocate the queue with xQueueCreate, push structs into it with xQueueSend, and pop them out with xQueueReceive. Unlike global variables protected by clunky mutexes, queues copy data by value into a managed memory buffer, guaranteeing that the receiving task gets a pristine snapshot of the data exactly as it was when sent.

ESP32 FreeRTOS IPC: Why Queues Win for Sensor Data

Before writing code, you must choose the right IPC mechanism. The ESP32 Arduino core exposes the full FreeRTOS API, which includes several synchronization primitives. Picking the wrong one leads to wasted RAM or blocked tasks. Here is how queues compare to the alternatives when building sensor pipelines.

For a deep dive into the underlying kernel mechanics, the official FreeRTOS Queue documentation remains the definitive reference for understanding how the RTOS manages the internal ring buffers.

Hardware Build: Dual-Core BME280 Queue Pipeline

We are building a dual-core pipeline. Core 1 will handle the time-sensitive I2C polling of a BME280 environmental sensor. Core 0 will handle the blocking, lower-priority task of formatting and printing that data to the Serial monitor (which simulates sending it over WiFi/MQTT).

Parts List

• Microcontroller: ESP32-WROOM-32 DevKit V1 (30-pin variant). Note: The code targets the standard 30-pin mapping. If using the 38-pin ESP32-WROVER, adjust I2C pins accordingly.
• Sensor: Adafruit BME280 Breakout (STEMMA QT / Qwiic I2C version, Product ID 2652).
• Wiring: 4x silicone jumper wires (female-to-female).
• Power: USB-C cable for serial monitoring and 5V power.

Pin Mapping Table

According to the Espressif FreeRTOS API reference, pinning tasks to specific cores prevents the WiFi stack (which runs on Core 0 by default) from starving your sensor polling task. We will explicitly pin the sensor read to Core 1.

The Code: Thread-Safe Data Passing

Below is the complete, compilable Arduino IDE code. It requires the Adafruit BME280 Library and the Adafruit Unified Sensor library installed via the Library Manager. The code includes explicit error handling for queue creation and I2C initialization failures.

#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#include <freertos/queue.h>

// --- Pin Definitions ---
#define I2C_SDA 21
#define I2C_SCL 22
#define SEALEVELPRESSURE_HPA (1013.25)

// --- Global Objects ---
Adafruit_BME280 bme;
QueueHandle_t sensorQueue;

// --- Data Structure for Queue ---
struct SensorData {
  float temperature;
  float humidity;
  float pressure;
  uint32_t timestamp;
};

// --- Task 1: Sensor Reading (Pinned to Core 1) ---
void sensorReadTask(void *pvParameters) {
  SensorData reading;
  
  while (true) {
    reading.temperature = bme.readTemperature();
    reading.humidity = bme.readHumidity();
    reading.pressure = bme.readPressure() / 100.0F;
    reading.timestamp = millis();

    // Push to queue with a 100ms timeout
    if (xQueueSend(sensorQueue, &reading, pdMS_TO_TICKS(100)) != pdPASS) {
      Serial.println("[ERROR] Queue send failed: Queue full or timeout.");
    }
    
    // Yield to lower priority tasks and wait 2 seconds
    vTaskDelay(pdMS_TO_TICKS(2000));
  }
}

// --- Task 2: Data Processing (Pinned to Core 0) ---
void dataProcessTask(void *pvParameters) {
  SensorData receivedData;
  
  while (true) {
    // Block indefinitely until data arrives in the queue
    if (xQueueReceive(sensorQueue, &receivedData, portMAX_DELAY) == pdPASS) {
      Serial.printf("[T:%lu] Temp: %.2f C | Hum: %.1f %% | Pres: %.2f hPa\n",
                    receivedData.timestamp, 
                    receivedData.temperature,
                    receivedData.humidity, 
                    receivedData.pressure);
    }
  }
}

void setup() {
  Serial.begin(115200);
  delay(1000); // Allow serial monitor to connect
  
  Wire.begin(I2C_SDA, I2C_SCL);

  if (!bme.begin(0x77, &Wire)) {
    Serial.println("[FATAL] BME280 init failed. Check I2C wiring and address.");
    while (true) { delay(10); } // Halt execution
  }

  // Create queue: 10 slots deep, each slot holds one SensorData struct
  sensorQueue = xQueueCreate(10, sizeof(SensorData));
  if (sensorQueue == NULL) {
    Serial.println("[FATAL] Queue creation failed. Insufficient heap memory.");
    while (true) { delay(10); }
  }

  // Create tasks and pin to specific cores
  xTaskCreatePinnedToCore(sensorReadTask, "SensorRead", 4096, NULL, 1, NULL, 1);
  xTaskCreatePinnedToCore(dataProcessTask, "DataProcess", 4096, NULL, 1, NULL, 0);
}

void loop() {
  // The loop is empty. We yield to the FreeRTOS scheduler.
  vTaskDelay(pdMS_TO_TICKS(10000));
}

Debugging Queue Failures: Exact Errors and Fixes

When ESP32 Arduino queues fail, they rarely fail gracefully. Because you are manipulating memory at the RTOS level, mistakes result in hard kernel panics. Here are the exact error strings you will see in the serial monitor, ranked by frequency, and how to fix them.

1. Guru Meditation Error: Core 1 panic'ed (LoadProhibited)

The Symptom: The ESP32 reboots instantly when the queue send or receive function is called. The backtrace points to xQueueGenericSend or xQueueGenericReceive.

Ranked Causes:

1. Uninitialized Queue Handle: You called xQueueSend before xQueueCreate finished, or xQueueCreate failed (returned NULL) due to heap fragmentation, but you didn't check for NULL.
2. Dangling Pointers: You created a queue of pointers (QueueHandle_t q = xQueueCreate(5, sizeof(SensorData*))) and passed the address of a local variable inside a function. By the time the receiving task reads the pointer, the local variable is destroyed, and the ESP32 tries to read unmapped memory.

The Fix: Always pass structs by value (as shown in the code above) rather than by pointer, unless you are using dynamically allocated memory (malloc) that the receiving task explicitly free()s.

2. Silent Data Drops (Queue Full)

The Symptom: No crash, but the receiving task misses every 3rd or 4th reading. The serial monitor shows gaps in timestamps.

Ranked Causes:

1. Consumer Task Starvation: The processing task is blocked by a slow operation (like a DNS lookup or a blocking delay()) and isn't calling xQueueReceive fast enough. The 10-slot queue fills up.
2. Timeout Too Short: You used xQueueSend(queue, &data, 0). If the queue is full, it drops the data immediately instead of waiting.

The Fix: Increase the queue depth in xQueueCreate, or increase the timeout parameter in xQueueSend to pdMS_TO_TICKS(500) to allow the sender to wait for a slot to open.

3. assert failed: xQueueGenericSend queue.c

The Symptom: Hard crash with an assertion failure pointing directly to the FreeRTOS queue.c source file.

Ranked Causes:

1. ISR Context Violation: You are trying to send data to the queue from an Interrupt Service Routine (ISR) using the standard xQueueSend function.

The Fix: If you are inside an ISR (like a pin change interrupt or a hardware timer callback), you must use the FromISR variant: xQueueSendFromISR(sensorQueue, &reading, &xHigherPriorityTaskWoken).

Extending and Simplifying the Pipeline

Once your basic ESP32 Arduino queues implementation is stable, you will inevitably need to scale it. Here is how to adapt the architecture based on your production requirements.

How to Extend: Adding WiFi and MQTT

To turn this into an IoT node, do not add WiFi logic to the sensorReadTask. Network stacks are inherently blocking and jittery. Instead, keep the current architecture and modify the dataProcessTask. Add the PubSubClient library, connect to your MQTT broker in setup(), and inside the xQueueReceive success block, serialize the SensorData struct into a JSON string and publish it. Because Core 0 handles the WiFi stack natively, keeping your network transmissions on Core 0 prevents cross-core bus contention.

How to Simplify: Direct Task Notifications

If you realize you don't actually need to pass data between tasks, but just need to wake one task up when another finishes (e.g., telling a display task to redraw after a button press), drop the queue entirely. Queues consume heap memory. Instead, use FreeRTOS Task Notifications via xTaskNotifyGive() and ulTaskNotifyTake(). It uses zero extra RAM, operates roughly 40% faster than a binary semaphore, and is natively supported in the ESP32 Arduino core.

Mastering ESP32 Arduino queues is the bridge between writing simple Arduino sketches and engineering robust, multi-threaded embedded firmware. Respect the memory boundaries, pin your cores intentionally, and always check your return codes.