Use the sample pkcs8KeyStore.der file in the $ESP_HOME/adapters/esp_open/lib/security directory to generate self-signed RSA keys using OpenSSL.
openssl pkcs8 -nocrypt -in server.key -out pkcs8KeyStore.der -outform DER -topk8