ESP32 HTTPClient readBytes: Stream Large Payloads Without OOM Crashes

The Heap-Killer: Why getString() Fails and readBytes() Saves Your ESP32

If you have ever tried to download a 2MB OTA firmware file, a high-resolution JPEG, or a massive JSON telemetry payload using http.getString() on an ESP32, you have likely encountered a sudden reboot or a Guru Meditation Error. The root cause is heap fragmentation. The ESP32-WROOM-32 has roughly 520KB of internal SRAM, but the largest contiguous free block is often less than 120KB. When getString() attempts to allocate a single contiguous String object for a large payload, the allocator fails, and your sketch crashes.

The direct, bulletproof solution is to use the Arduino Stream readBytes() method via the HTTPClient stream pointer. Instead of loading the entire payload into RAM, readBytes() pulls the data from the WiFi buffer in small, manageable chunks (typically 1KB) and writes them directly to a storage medium like an SD card or LittleFS. This keeps your heap footprint flat and eliminates out-of-memory (OOM) panics, regardless of whether the server sends a 10KB text file or a 16MB binary blob.

Target Board: This guide and the accompanying code are explicitly written and tested for the ESP32-WROOM-32 DevKit V1 (30-pin) running the Arduino ESP32 Core v2.0.14 or newer.

ESP32 HTTPClient Payload Methods: Memory & Performance Matrix

Choosing the right ingestion method dictates whether your ESP32 survives the download. The table below benchmarks the four primary HTTPClient payload extraction methods based on real-world heap behavior.

Hardware BOM and SD Card SPI Pin Mapping

To demonstrate readBytes() safely, we will stream a binary file directly to a microSD card. The ESP32's internal LittleFS is great, but frequent large writes degrade the SPI flash over time. An SD card is the correct choice for heavy logging or firmware staging.

Parts List

• MCU: ESP32-WROOM-32 DevKit V1 (30-pin)
• Storage: MicroSD Card Adapter Module (SPI interface, not SDIO)
• Passives: 4x 10kΩ pull-up resistors, 1x 100nF decoupling capacitor
• Wiring: 22 AWG silicone wire (keep SPI traces under 10cm)

VSPI Pin Mapping Table

The ESP32 has multiple SPI buses. We use the default VSPI bus for the SD card. Do not use the HSPI bus if you plan to add an SPI display later without careful CS pin management.

Complete Compilable Code: Streaming Binary Data via readBytes()

The following sketch connects to WiFi, mounts the SD card, and streams a 5MB dummy binary file from a public test server. It includes robust error handling for network drops and storage failures.

#include <WiFi.h>
#include <HTTPClient.h>
#include <SD.h>
#include <SPI.h>

// --- PIN DEFINITIONS ---
#define SD_CS_PIN   5
#define SD_MOSI_PIN 23
#define SD_MISO_PIN 19
#define SD_SCK_PIN  18

// --- NETWORK CREDENTIALS ---
const char* ssid = "YOUR_WIFI_SSID";
const char* password = "YOUR_WIFI_PASSWORD";

// --- TARGET URL ---
// A 5MB dummy binary file for testing
const char* payloadUrl = "http://speedtest.ftp.otenet.gr/files/test5Mb.db";

void setup() {
  Serial.begin(115200);
  delay(1000);
  Serial.println("\n[BOOT] ESP32 HTTPClient readBytes Streamer");

  // 1. Initialize WiFi
  WiFi.begin(ssid, password);
  Serial.print("[WIFI] Connecting");
  int retries = 0;
  while (WiFi.status() != WL_CONNECTED && retries < 40) {
    delay(500);
    Serial.print(".");
    retries++;
  }
  
  if (WiFi.status() != WL_CONNECTED) {
    Serial.println("\n[ERROR] WiFi connection failed. Rebooting.");
    ESP.restart();
  }
  Serial.printf("\n[WIFI] Connected. RSSI: %d dBm\n", WiFi.RSSI());

  // 2. Initialize SD Card
  SPI.begin(SD_SCK_PIN, SD_MISO_PIN, SD_MOSI_PIN, SD_CS_PIN);
  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("[ERROR] SD Card Mount Failed. Check wiring and MISO pull-up.");
    return; // Halt execution, do not reboot in a loop for hardware faults
  }
  Serial.println("[SD] Card mounted successfully.");

  // 3. Execute HTTP Stream
  downloadFile(payloadUrl, "/test5Mb.db");
}

void loop() {
  // Nothing to do in loop for this single-shot example
  delay(10000);
}

void downloadFile(const char* url, const char* localPath) {
  HTTPClient http;
  http.begin(url);
  http.setTimeout(15000); // 15 second timeout for slow servers

  Serial.printf("[HTTP] GET... %s\n", url);
  int httpCode = http.GET();

  if (httpCode != HTTP_CODE_OK) {
    Serial.printf("[ERROR] HTTP GET failed, error: %s\n", http.errorToString(httpCode).c_str());
    http.end();
    return;
  }

  // Check payload size (-1 means chunked encoding/unknown size)
  int len = http.getSize();
  Serial.printf("[HTTP] Payload size: %d bytes\n", len);

  // Open file for writing
  File file = SD.open(localPath, FILE_WRITE);
  if (!file) {
    Serial.println("[ERROR] Failed to open file for writing.");
    http.end();
    return;
  }

  // Get the WiFi stream pointer
  WiFiClient* stream = http.getStreamPtr();
  
  // Pre-allocate buffer in heap (1024 bytes is optimal for ESP32 WiFi buffers)
  uint8_t buff[1024];
  size_t totalWritten = 0;
  unsigned long startTime = millis();

  Serial.println("[STREAM] Downloading...");
  
  // THE CORE READBYTES LOOP
  while (http.connected() && (len > 0 || len == -1)) {
    size_t size = stream->available();
    if (size) {
      // Read up to sizeof(buff) bytes, or whatever is available
      int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
      
      size_t written = file.write(buff, c);
      if (written != c) {
        Serial.println("\n[ERROR] SD write failed. Disk full or unmounted.");
        break;
      }
      
      totalWritten += written;
      if (len > 0) {
        len -= c;
      }
      
      // Print progress dot every 32KB
      if (totalWritten % (32 * 1024) < sizeof(buff)) {
        Serial.print(".");
      }
    }
    // Yield to the watchdog timer and WiFi stack
    delay(1); 
  }

  unsigned long duration = millis() - startTime;
  file.close();
  http.end();

  Serial.printf("\n[SUCCESS] Downloaded %u bytes in %lu ms (%.2f KB/s)\n", 
                totalWritten, duration, (totalWritten / 1024.0) / (duration / 1000.0));
}

Debugging: First Three Checks and Exact Error Strings

When streaming fails, the ESP32 rarely gives you a polite error message. It either hangs, reboots, or throws a cryptic string. If your sketch fails, here are the first three things to check, followed by a breakdown of specific error strings.

The First Three Checks

1. Server Content-Length Header: If your server uses Transfer-Encoding: chunked (common with Node.js/Express or PHP without explicit headers), http.getSize() returns -1. The code above handles this via the len == -1 condition, but if you wrote your own loop relying strictly on len > 0, it will abort instantly. Always verify your server sends a fixed Content-Length.
2. WiFi RSSI and Chunk Timeouts: If the ESP32 is far from the router (RSSI worse than -80dBm), the WiFi buffer starves. The stream->available() check returns 0, and if you don't have a timeout or a http.connected() check, the sketch hangs in an infinite while loop.
3. SD SPI Wiring & Pull-ups: If the download starts but aborts at exactly 4KB or 8KB, your SD card is dropping off the SPI bus due to noise or missing MISO pull-ups. Check your physical wiring.

Ranked Causes for Exact Error Strings

Extending and Simplifying the Build

Depending on your final application, you may not need an external SD card, or you might need stricter security. Here is how to adapt the readBytes() architecture.

How to Simplify: Drop the SD Card for LittleFS

If your payload is strictly under 1.5MB (e.g., a small web UI bundle, a configuration JSON, or a low-res image), you can eliminate the SD card hardware entirely. Replace the SD.h includes with LittleFS.h. Change SD.begin() to LittleFS.begin(true) (the true formats on fail), and swap SD.open() for LittleFS.open(). This reduces your BOM cost by $3-$5 and removes all SPI wiring headaches, at the cost of using a small portion of the ESP32's internal 4MB flash.

How to Extend: Add Chunked SHA-256 Hashing

If you are downloading OTA firmware binaries or critical security certificates, you must verify the file wasn't corrupted or intercepted in transit. You can extend the readBytes() loop by feeding every 1KB chunk into a mbedtls_sha256_context object.

Instead of just calling file.write(buff, c), you simultaneously call mbedtls_sha256_update(&ctx, buff, c). Once the loop finishes, finalize the hash and compare it against a known-good hash provided by your server's API. This adds roughly 15% to the CPU overhead of the download loop but guarantees cryptographic integrity before you ever attempt to flash or execute the downloaded file.