What does mbed-tls error code -0x3B00 mean

If you see an error message like the following one on your microcontroller (such as ESP32):

E (41544) esp-tls-mbedtls: mbedtls_ssl_handshake returned -0x3B00

this means MBEDTLS_ERR_PK_INVALID_PUBKEY.

As of the version of mbed TLS used in esp-idf v4.4.3, only RSA & (certain types of) Elliptic Curve keys are supported. In my tests, X25519/EC256 keys didn’t work and there were indications that P-384 keys also didn’t work. Generally, using RSA keys is a safe bet when working with mbed-tls.