Rust on ESP32 - How to send (and receive) data using the MQTT protocol to AWS IoT (Core)?

Question

First off: I know running Rust on an ESP32 isn't a very common practice yet, and some (quite a bit of) trouble is to be expected. But I seem to have hit a roadblock.

What works:

• flashing and running the code on an ESP32
• passing along the certificates in the src/certificates directory
• WiFi connection (simple WPA Personal, nothing fancy like WPA Enterprise)
• publishing and suscribing to topics using MQTT

What doesn't work:

• publising and subscribing to AWS IoT (Core) using MQTT. This needs certificates, and as far as I'm aware I'm handling this properly (see code below).

Some additional info (see code below):

• server.cert.crt is renamed from the AWS provided root-CA.crt
• client.cert.pem is renamed from the AWS provided my-thing-rev1.cert.pem
• client.private.key is renamed from the AWS provided my-thing-rev1.private.key
• I also received my-thing-rev1.public.key and my-thing-rev1-Policy, but I don't think I need these...?
• I know this is not the proper way of implementing this (I should not provide the certificates directly, instead use a service to get them, but this is a very basic POC)
• the code works fine if I don't want to connect to AWS, but instead use my own broker or broker.emqx.io for testing (even with the certificates included)

This is the code I'm currently using (heavily based on Rust on ESP32 STD demo app):

use embedded_svc::httpd::Result;
use embedded_svc::mqtt::client::{Connection, MessageImpl, QoS};
use esp_idf_svc::mqtt::client::{EspMqttClient, MqttClientConfiguration};
use esp_idf_svc::tls::X509;
use esp_idf_sys::EspError;
// other needed imports (not relevant here)

extern crate dotenv_codegen;
extern crate core;

const AWS_IOT_ENDPOINT: &str = dotenv!("AWS_IOT_ENDPOINT");
const AWS_IOT_CLIENT_ID: &str = dotenv!("AWS_IOT_CLIENT_ID");
const AWS_IOT_TOPIC: &str = dotenv!("AWS_IOT_TOPIC");

fn main() -> Result<()> {
    esp_idf_sys::link_patches();

    // other code

    let mqtt_client: EspMqttClient<ConnState<MessageImpl, EspError>> = test_mqtt_client()?;

    // more code

    Ok(())
}

fn convert_certificate(mut certificate_bytes: Vec<u8>) -> X509<'static> {
    // append NUL
    certificate_bytes.push(0);

    // convert the certificate
    let certificate_slice: &[u8] = unsafe {
        let ptr: *const u8 = certificate_bytes.as_ptr();
        let len: usize = certificate_bytes.len();
        mem::forget(certificate_bytes);

        slice::from_raw_parts(ptr, len)
    };

    // return the certificate file in the correct format
    X509::pem_until_nul(certificate_slice)
}

fn test_mqtt_client() -> Result<EspMqttClient<ConnState<MessageImpl, EspError>>> {
    info!("About to start MQTT client");

    let server_cert_bytes: Vec<u8> = include_bytes!("certificates/server.cert.crt").to_vec();
    let client_cert_bytes: Vec<u8> = include_bytes!("certificates/client.cert.pem").to_vec();
    let private_key_bytes: Vec<u8> = include_bytes!("certificates/client.private.key").to_vec();

    let server_cert: X509 = convert_certificate(server_cert_bytes);
    let client_cert: X509 = convert_certificate(client_cert_bytes);
    let private_key: X509 = convert_certificate(private_key_bytes);

    // TODO: fix the following error: `E (16903) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x7280`
    let conf = MqttClientConfiguration {
        client_id: Some(AWS_IOT_CLIENT_ID),
        crt_bundle_attach: Some(esp_idf_sys::esp_crt_bundle_attach),
        server_certificate: Some(server_cert),
        client_certificate: Some(client_cert),
        private_key: Some(private_key),
        ..Default::default()
    };
    let (mut client, mut connection) =
        EspMqttClient::new_with_conn(AWS_IOT_ENDPOINT, &conf)?;

    info!("MQTT client started");

    // Need to immediately start pumping the connection for messages, or else subscribe() and publish() below will not work
    // Note that when using the alternative constructor - `EspMqttClient::new` - you don't need to
    // spawn a new thread, as the messages will be pumped with a backpressure into the callback you provide.
    // Yet, you still need to efficiently process each message in the callback without blocking for too long.
    //
    // Note also that if you go to http://tools.emqx.io/ and then connect and send a message to the specified topic,
    // the client configured here should receive it.
    thread::spawn(move || {
        info!("MQTT Listening for messages");

        while let Some(msg) = connection.next() {
            match msg {
                Err(e) => info!("MQTT Message ERROR: {}", e),
                Ok(msg) => info!("MQTT Message: {:?}", msg),
            }
        }

        info!("MQTT connection loop exit");
    });

    client.subscribe(AWS_IOT_TOPIC, QoS::AtMostOnce)?;

    info!("Subscribed to all topics ({})", AWS_IOT_TOPIC);

    client.publish(
        AWS_IOT_TOPIC,
        QoS::AtMostOnce,
        false,
        format!("Hello from {}!", AWS_IOT_TOPIC).as_bytes(),
    )?;

    info!("Published a hello message to topic \"{}\".", AWS_IOT_TOPIC);

    Ok(client)
}

Here are the final lines of output when I try to run this on the device (it's setup to compile and flash to the device and monitor (debug mode) when running cargo run):

I (16913) esp32_aws_iot_with_std: About to start MQTT client
I (16923) esp32_aws_iot_with_std: MQTT client started
I (16923) esp32_aws_iot_with_std: MQTT Listening for messages
I (16933) esp32_aws_iot_with_std: MQTT Message: BeforeConnect
I (17473) esp-x509-crt-bundle: Certificate validated
E (19403) MQTT_CLIENT: mqtt_message_receive: transport_read() error: errno=119  # <- This is the actual error
E (19403) MQTT_CLIENT: esp_mqtt_connect: mqtt_message_receive() returned -1
E (19413) MQTT_CLIENT: MQTT connect failed
I (19413) esp32_aws_iot_with_std: MQTT Message ERROR: ESP_FAIL
I (19423) esp32_aws_iot_with_std: MQTT Message: Disconnected
E (19433) MQTT_CLIENT: Client has not connected
I (19433) esp32_aws_iot_with_std: MQTT connection loop exit
I (24423) esp_idf_svc::eventloop: Dropped
I (24423) esp_idf_svc::wifi: Stop requested
I (24423) wifi:state: run -> init (0)
I (24423) wifi:pm stop, total sleep time: 10737262 us / 14862601 us
W (24423) wifi:<ba-del>idx
I (24433) wifi:new:<1,0>, old:<1,1>, ap:<1,1>, sta:<1,0>, prof:1
W (24443) wifi:hmac tx: ifx0 stop, discard
I (24473) wifi:flush txq
I (24473) wifi:stop sw txq
I (24473) wifi:lmac stop hw txq
I (24473) esp_idf_svc::wifi: Stopping
I (24473) esp_idf_svc::wifi: Disconnect requested
I (24473) esp_idf_svc::wifi: Stop requested
I (24483) esp_idf_svc::wifi: Stopping
I (24483) wifi:Deinit lldesc rx mblock:10
I (24503) esp_idf_svc::wifi: Driver deinitialized
I (24503) esp_idf_svc::wifi: Dropped
I (24503) esp_idf_svc::eventloop: Dropped
Error: ESP_FAIL

This error seems to indicate the buffer holding the incoming data is full and can't hold any more data, but I'm not sure. And I definately don't know how to fix it.

(I assume the actual certificate handling is done properly)

When I run the following command, I do get the message in AWS IoT (MQTT test client): mosquitto_pub -h my.amazonawsIoT.com --cafile server.cert.crt --cert client.cert.pem --key client.private.key -i basicPubSub -t my/topic -m 'test'

Does anyone have some more experience with this who can point me in the right direction? Is this actually a buffer error, and if so: how do I mitigate this error? Do I need to increase the buffer size somehow (it is running on a basic ESP32 revision 1, ESP32_Devkitc_v4, if that helps). As far as I can tell this version has a 4MB flash size, so that might explain the buffer overlow, although I think this should be enough. The total memory used is under 35% of the total storage (App/part. size: 1347344/4128768 bytes, 32.63%)

UPDATE 1: I have been made aware that this data is stored in RAM, not in flash memory (didn't cross my mind at the time), but I'm not entirely sure on how large the RAM on my specific device is (ESP32 revision 1, ESP32_Devkitc_v4). My best guess is 320KB, but I'm not sure.

UPDATE 2: I've tried changing the buffer size like so:

    let conf = MqttClientConfiguration {
        client_id: Some(AWS_IOT_CLIENT_ID),
        crt_bundle_attach: Some(esp_idf_sys::esp_crt_bundle_attach),
        server_certificate: Some(server_cert),
        client_certificate: Some(client_cert),
        private_key: Some(private_key),
        buffer_size: 50,      // added this (tried various sizes)
        out_buffer_size: 50,  // added this (tried various sizes)
        ..Default::default()
    };

I've tried various combinations, but this doesn't seem to change much: either I get the exact same error, or this one (when choosing smaller numbers, for example 10):

E (18303) MQTT_CLIENT: Connect message cannot be created
E (18303) MQTT_CLIENT: MQTT connect failed
E (18313) MQTT_CLIENT: Client has not connected

I'm not sure how big this buffer size should be (when sending simple timestamps to AWS IoT), and can't find any documentation on what this number represents: is it in Bit, KiloBit, ... No idea.

Answer 1

I had the same issue (certificate being validated and the connection dropping immediately after with the error code 119). From what I managed to understand, that code refers to the servers sending a FIN, thus causing the connection to be closed. In my case it was caused by the policy attached to the certificate, which didn't allow the use of the topic I was doing my tests on (https://github.com/eclipse/paho.mqtt.rust/issues/119#issuecomment-825745402 this is what made me figure it out). You can try to edit your policy in the AWS console and temporarily allowing all the IoT operations () on all the resources (). Be sure to set the new version as active with the checkbox below the permissions section. If this solves your problem, you can then proceed to properly limit the permission in your policy to the required IoT operations + topics.