How to Use Kuba Zip in Arduino IDE: ESP32 SD Card Compression

Why Kuba Zip Beats Standard Zlib on Microcontrollers

If you are trying to compress sensor logs or bundle configuration files on a microcontroller, standard zlib will quickly exhaust your SRAM and flash. The direct answer to how to use Kuba Zip in Arduino environments is to pair the lightweight kuba--/zip C library with a high-memory board like the ESP32, leveraging the ESP32's POSIX Virtual File System (VFS) to bridge the gap between C-standard file I/O and Arduino's SD library.

Kuba Zip is a single-file, portable C library that implements the ZIP format without the massive overhead of zlib. However, there is a critical trap: standard AVR Arduinos (like the Uno or Nano) lack the RAM (2KB) and the POSIX file system wrappers required to run it. Therefore, this guide explicitly targets the ESP32 DevKit V1 (ESP32-WROOM-32) running via the Arduino IDE. The ESP32 offers 520KB of SRAM and natively maps its SD card to a POSIX file path, allowing Kuba Zip's fopen() calls to work flawlessly.

Embedded ZIP Library Comparison

Before wiring up the hardware, it is vital to understand why Kuba Zip is the optimal choice for ESP32/STM32 data logging compared to alternatives. The table below benchmarks real-world footprints for embedded archiving.

Sources: Benchmarks derived from ESP32-WROOM-32 compilation via kuba--/zip GitHub repository and Espressif linker maps.

Hardware BOM and ESP32 SPI Pin Mapping

To compress files directly to an SD card, we need a reliable SPI connection. Do not use the cheap MicroSD adapters without level shifters; the ESP32 operates at 3.3V logic, and feeding 5V into the ESP32's GPIO pins will eventually brick the silicon.

Parts List

• MCU: ESP32 DevKit V1 (ESP32-WROOM-32 variant, 30-pin or 38-pin)
• Storage: MicroSD SPI Breakout Board (with 3.3V LDO and logic level shifters, e.g., Adafruit 254 or equivalent)
• Media: 8GB or 16GB MicroSDHC Card (FAT32 formatted)
• Wiring: 22 AWG solid core jumper wires

SPI Pin Mapping Table

The ESP32 has multiple SPI buses. We will use the default VSPI bus for the SD card to leave HSPI free for other peripherals like SPI displays.

Step-by-Step Implementation and Compilable Code

The biggest hurdle when using standard C libraries in the Arduino IDE is file I/O. Standard Arduino SD.open() returns a custom File object, but Kuba Zip expects standard C FILE* pointers via fopen().

1. Install the Library

Kuba Zip is not in the standard Arduino Library Manager. Download zip.h and zip.c from the official GitHub repository and place them directly into your Arduino sketch folder alongside your .ino file.

2. The Complete Code

This sketch initializes the SD card, creates a dummy sensor log, compresses it into a ZIP archive using Kuba Zip, and verifies the heap memory to ensure no leaks occurred.

#include <SD.h>
#include <SPI.h>
#include "zip.h" // Include the kuba--/zip header from your sketch folder

// --- Pin Definitions ---
#define SD_CS_PIN 5
#define SPI_MOSI 23
#define SPI_MISO 19
#define SPI_SCK 18

// --- File Paths (ESP32 VFS Mount Point) ---
const char* RAW_FILE_PATH = "/sd/sensor_log.txt";
const char* ZIP_FILE_PATH = "/sd/archive.zip";
const char* ZIP_ENTRY_NAME = "log_data.txt";

void setup() {
  Serial.begin(115200);
  while(!Serial) { delay(10); }
  
  Serial.println("--- ESP32 Kuba Zip Compression Demo ---");
  Serial.printf("Initial Free Heap: %d bytes\n", esp_get_free_heap_size());

  // 1. Initialize SPI and SD Card
  SPI.begin(SPI_SCK, SPI_MISO, SPI_MOSI, SD_CS_PIN);
  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("[FATAL] SD Card Mount Failed. Check wiring and FAT32 format.");
    while(1) { delay(1000); }
  }
  Serial.println("SD Card mounted successfully via VFS.");

  // 2. Create a raw data file using standard POSIX calls
  FILE* f = fopen(RAW_FILE_PATH, "w");
  if (f == NULL) {
    Serial.println("[ERROR] Failed to create raw file.");
    return;
  }
  fprintf(f, "Timestamp,Temp_C,Humidity_Pct\n");
  for(int i=0; i<100; i++) {
    fprintf(f, "%d,24.5,45.2\n", i);
  }
  fclose(f);
  Serial.println("Raw sensor log written to /sd/sensor_log.txt");

  // 3. Compress using Kuba Zip
  compressFile();

  // 4. Cleanup raw file
  remove(RAW_FILE_PATH);
  Serial.println("Raw file deleted. Archive ready for extraction.");
  Serial.printf("Final Free Heap: %d bytes\n", esp_get_free_heap_size());
}

void loop() {
  // Nothing to do in loop for this demo
  delay(10000);
}

void compressFile() {
  Serial.println("Opening ZIP archive...");
  
  // ZIP_DEFAULT_COMPRESSION_LEVEL is usually 6. Use 0 for store-only (faster).
  struct zip_t *zip = zip_open(ZIP_FILE_PATH, ZIP_DEFAULT_COMPRESSION_LEVEL, 'w');
  if (!zip) {
    Serial.println("[ERROR] zip_open returned NULL. VFS path issue or SD write error.");
    return;
  }

  // Open entry inside the zip
  if (zip_entry_open(zip, ZIP_ENTRY_NAME) < 0) {
    Serial.println("[ERROR] zip_entry_open failed. Check heap memory.");
    zip_close(zip);
    return;
  }

  // Read raw file and write to zip entry
  FILE* raw = fopen(RAW_FILE_PATH, "r");
  if (raw) {
    char buffer[512];
    size_t bytesRead;
    while ((bytesRead = fread(buffer, 1, sizeof(buffer), raw)) > 0) {
      if (zip_entry_write(zip, buffer, bytesRead) < 0) {
        Serial.println("[ERROR] zip_entry_write failed mid-stream.");
        break;
      }
    }
    fclose(raw);
  }

  zip_entry_close(zip);
  zip_close(zip);
  Serial.println("Compression complete. File saved to /sd/archive.zip");
}

Debugging: Exact Error Strings and the First 3 Checks

When working at the intersection of C libraries and Arduino wrappers, things will break. If your serial monitor halts or throws errors, follow this decision path.

The First 3 Things to Check When It Fails

1. Verify the VFS Mount Point: The ESP32 Arduino core mounts the SD card to /sd/. If you are using a custom ESP-IDF implementation or an older core version, it might mount to /sdcard/. If zip_open fails immediately, check your mount path.
2. Check SPI Pin Strapping: GPIO 12 (MTDI) is a strapping pin on the ESP32. If your SD breakout board pulls GPIO 12 high during boot, the ESP32 will fail to boot or mount the SD card. Ensure your breakout board's MISO is strictly on GPIO 19.
3. Heap Fragmentation: Zipping requires contiguous memory blocks. If your sketch has been running for days allocating and freeing Strings, zip_open may fail due to fragmentation. Always use esp_get_free_heap_size() to monitor heap health.

Ranked Causes for Specific Error Strings

Scaling the Build: Extending and Simplifying

Once you have the basic archive generating, you will likely need to adapt the code for your specific production environment. Here is how to scale the implementation up or down.

How to Simplify (Store-Only Mode)

If you are logging high-frequency sensor data and the ESP32's CPU is bottlenecking on the deflate compression algorithm, you can bypass compression entirely while still bundling files into a standard ZIP container. Change the compression level parameter in zip_open:

// 0 = Store only (no compression). Drastically reduces CPU load and write latency.
struct zip_t *zip = zip_open(ZIP_FILE_PATH, 0, 'w');

This is highly recommended for battery-powered ESP32 deployments where CPU cycles directly correlate to power draw.

How to Extend (Streaming Large Sensor Logs)

The example code reads the raw file in 512-byte chunks. If you are logging audio or high-speed accelerometer data, reading a massive raw file from the SD card just to write it back to the SD card as a ZIP is inefficient and wears out the flash memory.

To extend this, use Kuba Zip's streaming capabilities to write directly to the ZIP entry as the sensor data arrives in RAM, completely bypassing the intermediate raw file:

// Inside your sensor ISR or polling loop:
zip_entry_open(zip, "streamed_data.bin");

while (recording) {
    read_sensor_into_buffer(buffer, 1024);
    zip_entry_write(zip, buffer, 1024); // Append directly to ZIP stream
}

zip_entry_close(zip);
zip_close(zip);

Note: When streaming directly, ensure your power supply is stable. A brownout during zip_close() will corrupt the ZIP central directory, rendering the archive unreadable. Always implement a watchdog timer and a supercapacitor brownout-bridge for critical data logging.