Technical Reference

A Little OpenSSL Guide

Practical cookbook for certificate verification, CMS/S/MIME decryption, and P12 handling. No theory - just the commands that work.

Comparing if a Public Certificate Matches the Private Key

Get hash of certificate's public key

$ openssl x509 -noout -modulus -in cert.pem | openssl md5

Get hash of private key

$ openssl rsa -noout -modulus -in private_key.pem | openssl md5
The meats'n'potatoes: The RSA modulus (n) lives in both the certificate's public key and the private key. Extract it from each file and compare the MD5 fingerprints. If they match, the files are a pair. If they don't, decryption or signing will fail.
The meats'n'potatoes (for EC keys): If you use elliptic curve keys instead of RSA, -modulus won't work. Use this instead:

openssl x509 -noout -pubkey -in cert.pem | openssl md5
openssl pkey -pubout -in privkey.pem | openssl md5

Both should still match if certificate and key belong together.

Show Infos of a CMS-Encrypted File

$ openssl cms -cmsout -print -inform DER -in encrypted_file.enc $ openssl x509 -noout -serial -in pubcert.pem
The meats'n'potatoes: CMS (Cryptographic Message Syntax) wraps the symmetric encryption key with each recipient's public key. -cmsout -print dumps the full ASN.1 structure of the CMS envelope as readable text. The subjectKeyIdentifier in the recipient info tells you which certificate was used for encryption. Cross-reference the serialNumber from openssl x509 -serial - if it matches your cert, you can decrypt. If not, you can't.
The meats'n'potatoes (DER vs PEM): The -inform DER flag means the CMS envelope is in binary DER format. If your file is base64-encoded (starts with -----BEGIN CMS-----), use -inform PEM instead. Most tools output DER by default, but some email clients produce PEM.

Unpacking from P12: PEM Cert + PEM Privkey

$ openssl pkcs12 -in user.p12 -nodes -out pubcertprivkey.pem
⚠ Warning: The private key is saved as it is in the P12 - encrypted or decrypted! If the P12 was exported with a key password, the output PEM will contain an encrypted private key. If no password was set, the private key is in plaintext.
The meats'n'potatoes: PKCS#12 (.p12/.pfx) bundles a certificate, private key, and optionally intermediate CA certs into one password-protected file. -nodes tells OpenSSL not to encrypt the private key on output, but if the key was already encrypted inside the P12, it stays encrypted. To extract into separate files instead of one combined file:

openssl pkcs12 -in user.p12 -nokeys -out pubcert.pem
openssl pkcs12 -in user.p12 -nocerts -nodes -out privkey.pem
The meats'n'potatoes (newer OpenSSL): OpenSSL 3.x deprecated some older PKCS#12 algorithms. If you get MAC verify error or pkcs12 routines errors, try adding -legacy or -provider default -provider legacy to work with older P12 files.

Decrypting CMS-Encrypted Files

With PEM cert + PEM privkey (binary mode)

$ openssl cms -decrypt -inform DER -binary -in encrypted_file.enc \ -out decrypted_file.txt \ -inkey privkey.pem \ -recip pubcert.pem

With P12 containing cert + privkey (binary mode)

$ openssl cms -decrypt -inform DER -binary -in encrypted_file.enc \ -out decrypted_file.txt \ -inkey user.p12 \ -recip user.p12
The meats'n'potatoes: -decrypt needs two things: the private key (-inkey) to unwrap the symmetric content-encryption key, and the certificate (-recip) to pick the right recipient slot in the CMS envelope. When both point to the same P12 file, OpenSSL extracts the cert and key automatically. -binary stops OpenSSL from treating the input as base64 - important when the file is raw DER.
The meats'n'potatoes (PEM-encoded CMS): If your encrypted file has PEM headers (-----BEGIN CMS-----), drop -inform DER -binary or use -inform PEM explicitly. The default input format depends on your OpenSSL build.

Decrypting S/MIME-Encrypted Files

As DER Binary

$ openssl smime -decrypt -inform DER -in encrypted_file.enc \ -out decrypted_file.txt \ -inkey privkey.pem \ -recip pubcert.pem

As Base64/S/MIME Text

$ openssl smime -decrypt -in encrypted_file.enc \ -out decrypted_file.txt \ -inkey privkey.pem \ -recip pubcert.pem
The meats'n'potatoes: S/MIME is the email-oriented wrapper around CMS. openssl smime and openssl cms handle the same crypto, but smime defaults to PEM (base64) and auto-detects DER vs PEM, while cms defaults to DER. If a mail client gave you the encrypted blob, try smime first - it's more forgiving with mixed formats.
The meats'n'potatoes (CMS vs S/MIME): openssl cms is the more modern tool. openssl smime exists for backward compatibility with email workflows. If both work, use cms - it supports more algorithms and gives clearer error messages.

Quick Troubleshooting

Common errors and what they mean

The meats'n'potatoes: OpenSSL error messages are cryptic. The error:XXXXX: codes are hex-encoded internal reason codes - you almost never need to decode them. Read the human-readable part after the last colon. Pay attention to whether it's a "routine" (internal logic) or "system library" (OS/permissions) error. 90% of issues come down to: wrong format, wrong password, or wrong key.
← Back to Schabernack