MQTT with Arduino: Physical Layer Wiring, Broker Mechanics, and ESP32 Code

The Physical Reality of MQTT: It’s Not a Hardware Bus

A common point of confusion when moving from local sensor wiring to IoT is treating MQTT like I2C, SPI, or CAN. MQTT is a Layer 7 (Application) publish/subscribe protocol. It does not define wires, voltage levels, or pull-up resistors. It relies entirely on a TCP/IP network stack to move bytes. Therefore, when we talk about implementing MQTT with Arduino or ESP32, we are actually talking about two distinct layers: the physical network transport (Wi-Fi or Ethernet) and the MQTT application layer running on top of it.

If you are designing a system, you must first choose the physical transport based on distance, device count, and environment. The table below contrasts local hardware buses with the network transports that carry MQTT, giving you the exact mechanics of how your data actually reaches the wire.

Transport & Bus Mechanics: How MQTT Reaches the Wire

Protocol / Transport	Physical Medium & Wires	Max Distance	Speed	Addressing	Arduino/ESP32 Interface
Wi-Fi (802.11n 2.4GHz) (Carries MQTT)	RF Spectrum (Antenna)	~50m indoors	Up to 150 Mbps	MAC / IP Address	Internal ESP32 Radio
Ethernet (10/100Base-T) (Carries MQTT)	Cat5e/6 (4 twisted pairs)	100m per segment	100 Mbps	MAC / IP Address	SPI (via W5500/ENC28J60)
RS485 (Modbus RTU) (Local Bus)	Twisted Pair (A, B, GND)	1200m	115.2 kbps max	Node ID (1-247)	UART + MAX485 Transceiver
CAN 2.0B (Local Bus)	Twisted Pair (CAN_H, CAN_L)	40m (at 1Mbps)	1 Mbps	Arbitration ID	CAN Transceiver (MCP2515)
I2C (Local Bus)	SDA, SCL, GND (Requires 4.7kΩ pull-ups)	~1m (capacitance limited)	400 kHz (Fast)	7-bit / 10-bit Hex	Hardware I2C Pins

Decision Framework: Use local buses (RS485/CAN) to gather sensor data into a central gateway, then use Wi-Fi or Ethernet to bridge that gateway to an MQTT broker over TCP/IP.

Wiring the Network Transport: SPI Ethernet vs. Wi-Fi

While an ESP32-WROOM-32 has an internal Wi-Fi radio requiring zero physical data wiring (just stable 3.3V power and RF clearance), industrial or high-reliability setups often demand hardwired Ethernet. To add Ethernet to a standard Arduino Uno/Mega or an ESP32, you use an SPI-to-Ethernet bridge like the Wiznet W5500.

Physical Wiring & Pull-Up Requirements

Unlike I2C, SPI does not require pull-up resistors on the data lines. However, the Chip Select (CS) line must be carefully managed. If you have multiple SPI devices on the same bus, the CS line for the W5500 must be held HIGH (inactive) by the microcontroller whenever you are communicating with an SD card or SPI sensor. Floating CS lines will cause bus collisions and corrupt your TCP/IP stack.

For Wi-Fi deployments, the physical layer constraint is RF environment and power delivery. The ESP32 draws up to 500mA during Wi-Fi transmission bursts. If your 3.3V rail sags below 2.9V, the TCP socket will drop silently, and the MQTT client will hang.

Broker Mechanics and the "Classic Failures"

MQTT operates on a publish/subscribe model mediated by a broker (like Mosquitto, HiveMQ, or AWS IoT). Clients publish to Topics (e.g., home/livingroom/temp) and subscribe to topics to receive messages. But because MQTT sits on top of TCP/IP, the most common failures aren't MQTT errors—they are network state errors.

The Classic Failures (And How to Fix Them)

1. Client ID Clashes (The Infinite Reconnect Loop)
Every MQTT client must present a unique Client ID to the broker. If two Arduino/ESP32 nodes connect with clientId = "sensor_node", the broker accepts the second connection and forcefully disconnects the first. The first node's code detects the drop and reconnects, kicking the second node. This creates an endless loop of connect/disconnect packets that can crash low-end brokers.
Fix: Generate the Client ID dynamically using the chip's MAC address or a hardcoded unique string per device.

2. NAT Table Timeouts (The Silent Drop)
If your device is behind a router performing Network Address Translation (NAT), the router maintains a mapping table for outbound TCP connections. If no data flows for a few minutes, the router drops the mapping to save memory. The MQTT broker thinks the client is still connected; the client thinks the broker is still there. But the TCP pipe is dead.
Fix: Set the MQTT KeepAlive interval (e.g., 60 seconds) to be strictly less than your router's NAT timeout (usually 300+ seconds). The PubSubClient library handles sending PINGREQ packets automatically if configured correctly.

3. Wi-Fi Modem Sleep Dropping Sockets
Espressif's Wi-Fi Power Save modes can turn off the RF modem to save battery. If a message arrives at the broker while the ESP32 is sleeping, the TCP ACK is missed, and the socket breaks.
Fix: Disable Wi-Fi modem sleep in your setup code if you require real-time MQTT subscriptions, or use MQTT QoS 1 with persistent sessions (Clean Session = false).

How to Sniff and Debug the Bus

Because MQTT is just TCP payload, you cannot use a logic analyzer on a wire. You must debug at the network or broker level:

• Broker Level: Run mosquitto_sub -v -h localhost -t '#' -F '%t %p' on your broker machine. This subscribes to all topics and prints the exact topic and payload, confirming if your Arduino is actually publishing.
• Network Level: Open Wireshark, filter by tcp.port == 1883, and look for the CONNECT, CONNACK, and PUBLISH MQTT control packets. If you see TCP Retransmissions, your physical Wi-Fi/Ethernet link is dropping packets.

Minimal Working Exchange: ESP32 to Mosquitto Broker

Below is a robust, copy-pasteable implementation for the ESP32 using the ubiquitous PubSubClient library. It includes MAC-based Client ID generation to prevent clashes, and a non-blocking reconnect loop that won't stall your sensor readings.

Prerequisites: Install the PubSubClient library via the Arduino IDE Library Manager. Ensure your broker IP is reachable from the ESP32's Wi-Fi network.

#include <WiFi.h>
#include <PubSubClient.h>

// Network & Broker Configuration
const char* ssid = "YourNetworkSSID";
const char* password = "YourNetworkPassword";
const char* mqtt_server = "192.168.1.100"; // Local Mosquitto Broker IP
const int mqtt_port = 1883;

WiFiClient espClient;
PubSubClient client(espClient);

// Timing variables for non-blocking execution
unsigned long lastMsg = 0;
const long interval = 5000; // Publish every 5 seconds

void setup_wifi() {
  delay(10);
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  // Disable Wi-Fi power save to prevent TCP socket drops
  WiFi.setSleep(false); 
  
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWiFi connected. IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void reconnect() {
  // Loop until reconnected
  while (!client.connected()) {
    // Generate unique Client ID from MAC address to prevent clashes
    String clientId = "ESP32-";
    clientId += String((uint32_t)ESP.getEfuseMac(), HEX);
    
    Serial.print("Attempting MQTT connection as ");
    Serial.print(clientId);
    Serial.print("...");
    
    // Connect with 60s KeepAlive to beat NAT timeouts
    if (client.connect(clientId.c_str(), NULL, NULL, "status", 1, true, "offline")) {
      Serial.println("connected");
      // Publish online status and subscribe to command topic
      client.publish("status", "online", true);
      client.subscribe("home/livingroom/cmd");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" retrying in 5 seconds");
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  setup_wifi();
  client.setServer(mqtt_server, mqtt_port);
  client.setCallback(callback);
  // Set buffer size if expecting large JSON payloads (default is 256 bytes)
  client.setBufferSize(512); 
}

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop(); // Must be called frequently to process incoming messages and keepalives

  unsigned long now = millis();
  if (now - lastMsg > interval) {
    lastMsg = now;
    
    // Simulate sensor reading
    float temp = 22.5 + random(-10, 10) / 10.0;
    char payload[50];
    snprintf(payload, sizeof(payload), "{\"temp\": %.2f}", temp);
    
    Serial.print("Publishing: ");
    Serial.println(payload);
    client.publish("home/livingroom/temp", payload);
  }
}

Verification Step

After flashing, open the Serial Monitor at 115200 baud. You should see the Wi-Fi connect, followed by Attempting MQTT connection... connected. On your broker machine, run mosquitto_sub -h 192.168.1.100 -t "home/#" -v. You must see the JSON payload arriving every 5 seconds. If the serial monitor shows rc=-2, the ESP32 cannot reach the broker's IP (check subnets). If it shows rc=-4, the broker is dropping the connection (check broker logs for authentication or memory limits).

For the official protocol specification and QoS mechanics, refer to the OASIS MQTT v5.0 Standard Documentation. For hardware-specific SPI constraints when using Ethernet shields, consult the Arduino SPI Reference.