The ESP32 ZIP Library Guide: Stream Extraction Without OOM Crashes

The ESP32 ZIP Library Landscape: Why Standard Wrappers Crash

If you are searching for a reliable esp32 zip library, you have likely already hit a wall. The direct answer to handling ZIP files on the ESP32 is to avoid standard C++ ZipLib wrappers entirely and instead use the C-based miniz library configured for streaming chunk extraction.

Standard Arduino ZIP libraries attempt to allocate contiguous RAM blocks to decompress file headers and payloads. On the classic ESP32-WROOM-32 DevKit V1 (30-pin), which has 520KB of SRAM but suffers from severe heap fragmentation, attempting to unzip a mere 2MB file will almost instantly trigger a Guru Meditation Error or a silent reboot. By targeting the ESP32-WROOM-32 and utilizing a streaming callback architecture, we can extract 50MB+ ZIP files using less than 32KB of RAM.

Memory Footprint: Standard ZipLib vs. Miniz Streaming

Before writing code, it is critical to understand why your previous attempts failed. The table below benchmarks the memory behavior of common ZIP handling methods on an ESP32-WROOM-32 with no PSRAM.

As the data shows, the miniz streaming callback is the only viable approach for standard .zip archives when dealing with files larger than the ESP32's available contiguous heap. For more on ESP32 heap capabilities, refer to the Espressif Memory Allocation Documentation.

Hardware BOM and SPI Pin Mapping

To demonstrate this, we will build a circuit that reads a payload.zip from a MicroSD card and extracts its contents to a separate directory on the same card.

Parts List

• MCU: ESP32-WROOM-32 DevKit V1 (30-pin)
• Storage: MicroSD Card Adapter Module (with 3.3V LDO and level shifters)
• Media: 16GB MicroSD Card (Must be formatted to FAT32 with 32KB allocation unit size)
• Passives: 10kΩ pull-up resistor (for MISO line if module lacks one)

SPI Pin Mapping Table

Step-by-Step: Crash-Free Streaming Extraction Code

The following code uses the miniz library. You must download miniz.c and miniz.h from the repository and place them directly into your Arduino sketch folder, or install a wrapper like miniz_cpp via the Library Manager.

#include <SD.h>
#include <SPI.h>
#include "miniz.h"

// --- PIN DEFINITIONS ---
#define SD_CS_PIN 5
#define SPI_MOSI 23
#define SPI_MISO 19
#define SPI_SCK 18

// --- GLOBALS ---
File extractFile;
const char* zipPath = "/payload.zip";
const char* extractDir = "/extracted/";

// --- STREAMING CALLBACK ---
// This function is called by miniz repeatedly with chunks of decompressed data.
size_t sd_write_callback(void *pOpaque, mz_uint64 file_ofs, const void *pBuf, size_t n) {
    File* f = (File*)pOpaque;
    if (!f) return 0;
    
    // Seek to the correct offset (crucial for fragmented zip payloads)
    if (!f->seek(file_ofs)) {
        Serial.println("ERROR: SD Seek Failed");
        return 0;
    }
    
    size_t written = f->write((const uint8_t*)pBuf, n);
    return written;
}

void setup() {
    Serial.begin(115200);
    delay(1000);
    Serial.println("ESP32 Miniz Streaming Zip Extractor");

    // Initialize SPI and SD Card at a conservative 4MHz to prevent signal integrity issues
    SPI.begin(SPI_SCK, SPI_MISO, SPI_MOSI, SD_CS_PIN);
    if (!SD.begin(SD_CS_PIN, SPI, 4000000)) {
        Serial.println("FATAL: SD Card Mount Failed. Check wiring and FAT32 format.");
        while(1) delay(1000);
    }

    // Create extraction directory if it doesn't exist
    if (!SD.exists(extractDir)) {
        SD.mkdir(extractDir);
    }

    // --- MINIZ EXTRACTION LOGIC ---
    mz_zip_archive zip_archive;
    memset(&zip_archive, 0, sizeof(zip_archive));

    if (!mz_zip_reader_init_file(&zip_archive, zipPath, 0)) {
        Serial.printf("ERROR: Failed to open %s. Error: %s\n", zipPath, mz_zip_get_error_string(mz_zip_get_last_error(&zip_archive)));
        return;
    }

    int num_files = mz_zip_reader_get_num_files(&zip_archive);
    Serial.printf("Found %d files in archive.\n", num_files);

    for (int i = 0; i < num_files; i++) {
        mz_zip_archive_file_stat file_stat;
        if (!mz_zip_reader_file_stat(&zip_archive, i, &file_stat)) {
            Serial.println("ERROR: Could not read file stat.");
            continue;
        }

        // Skip directories
        if (mz_zip_reader_is_file_a_directory(&zip_archive, i)) continue;

        char out_path[128];
        snprintf(out_path, sizeof(out_path), "%s%s", extractDir, file_stat.m_filename);
        
        Serial.printf("Extracting: %s (%llu bytes)\n", out_path, file_stat.m_uncomp_size);

        // Open target file on SD card
        extractFile = SD.open(out_path, FILE_WRITE);
        if (!extractFile) {
            Serial.printf("ERROR: Could not create %s\n", out_path);
            continue;
        }

        // Extract using the streaming callback
        mz_bool status = mz_zip_reader_extract_to_callback(
            &zip_archive, 
            i, 
            sd_write_callback, 
            &extractFile, 
            0
        );

        extractFile.close();

        if (!status) {
            Serial.printf("ERROR: Extraction failed for %s. Miniz Error: %s\n", 
                          file_stat.m_filename, 
                          mz_zip_get_error_string(mz_zip_get_last_error(&zip_archive)));
        } else {
            Serial.println(" -> Success.");
        }
    }

    mz_zip_reader_end(&zip_archive);
    Serial.println("Extraction Complete.");
}

void loop() {
    // Idle
    delay(10000);
}

Debugging: Exact Panic Strings and Ranked Causes

When working with file systems and dynamic memory on the ESP32, things will go wrong. Here are the exact error strings you will encounter, ranked by probability, and how to fix them.

1. Guru Meditation Error: Core 1 panic'ed (LoadProhibited)

Cause: A null pointer dereference, almost always caused by SD.open() failing silently and returning an invalid File object, which is then passed into the sd_write_callback.

Fix: Verify the target directory exists. The Arduino SD library does not automatically create nested directories. If your ZIP contains folder/subfolder/file.txt, you must programmatically create folder and subfolder on the SD card before calling the extraction callback.

2. mz_zip_reader_init_file failed: MZ_ZIP_FILE_OPEN_FAILED

Cause: The miniz library cannot read the ZIP archive's central directory. This is rarely a corrupted file; it is almost always an SPI bus timing or voltage issue.

Fix:

1. Drop the SPI clock speed from the default 16MHz down to 4MHz (as done in the code above).
2. Ensure your SD card is formatted to FAT32. The ESP32 SD library struggles with exFAT.

3. assertion "heap_caps_check_integrity_all(true)" failed

Cause: Heap corruption. You are likely using a standard ZipLib wrapper that allocates and frees memory rapidly inside a loop, fragmenting the ESP32 heap until a background task (like WiFi or RTOS idle) attempts to allocate memory and steps on a corrupted block.

Fix: Switch to the streaming callback method provided in this guide. By allocating zero dynamic memory during the extraction loop, you bypass heap fragmentation entirely.

Extending the Build: OTA and LittleFS Integration

Once you have the streaming callback working with an SD card, extending this architecture to other storage mediums or network sources is straightforward.

Simplifying: Switch to ESP32-targz

If you control the server or the pipeline generating the compressed files, abandon the .zip format entirely. The Arduino SD library and standard ZIP formats require reading the end of the file first to find the central directory. This means you cannot stream a ZIP file directly from an HTTP download without saving it to flash first. By switching to .tar.gz and using the ESP32-targz library, you can stream decompressed data sequentially from an HTTP client directly into LittleFS.

Extending: HTTP OTA Streaming

To adapt the miniz callback for Over-The-Air (OTA) updates without an SD card:

1. Replace the File* f pointer in pOpaque with a pointer to a LittleFS File object.
2. Instead of mz_zip_reader_init_file, use mz_zip_reader_init_mem if the ZIP is small (< 200KB), or write a custom mz_zip_reader_init read callback that pulls chunks directly from an HTTPClient stream.
3. This allows you to download a 10MB firmware asset bundle from AWS S3 and extract it directly into the ESP32's SPIFFS/LittleFS partition using only 32KB of RAM.

By respecting the ESP32's hardware limitations and leaning on streaming C-architecture rather than convenience C++ wrappers, you turn the esp32 zip library from a source of endless frustration into a bulletproof production feature.