Why won't some SSH SFTP clients connect after updating Cerberus to 12.11?

Cerberus FTP Server 12.11 and newer migrated to OpenSSL 3.0 to introduce support for TLS v1.3 and to maintain compliance with the latest FIPS 140-2 standard.   As part of the new compliance requirements for FIPS 140-2, some SSH key exchange parameter types are no longer compliant.  The most important one from a compatibility perspective with older clients is the ssh-rsa host key type that most older SSH clients support.  

If you are operating Cerberus FTP Server 12.11 or higher in FIPS 140-2 mode and using an RSA server key pair, you'll need to make sure your SSH SFTP clients support one or both of the following, more secure host key types:

• rsa-sha2-256
• rsa-sha2-512 

Diagnosing this Issue

The rsa-sha2-256 and rsa-sha2-512 host key parameters are sent during the initial SSH connection's key exchange by the server to advertise to the client the secure methods the server can use to transmit the server's host key. If your older SSH SFTP client does not support one of these RSA host key transmission methods then you will see an error message in the Cerberus log similar to the following:

[2022-11-08 14:07:39]: ERROR [ 12] - [127.0.0.1]: - No common host key: [S: rsa-sha2-256,rsa-sha2-512] [C: ssh-rsa,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss]
[2022-11-08 14:07:39]: ERROR [ 12] - [127.0.0.1]: - Key exchange failed: Could not agree on key exchange parameters

Solutions

Upgrade to a newer version of the SSH SFTP client that supports the secure and FIP 140-2 compliant rsa-sha2-256 or rsa-sha2-512 host key exchange parameter.

Cerberus 13.1.0 or above now supports using multiple SSH host key pairs simultaneously. If the client software having issues supports other algorithms, such as ECDSA (Elliptical Curves), you can try creating an additional SSH Host Key Pair in another format and activating it for use in Cerberus. Please note that client software will need to be re-keyed if you create a new key pair. See Multiple SSH Host Keys.

Add an Elliptical Curves SSH Server Host Key (Cerberus 13.1.0 and higher only) Cerberus 13.1.0 and higher now support having multiple SSH Server Host Keys. Many of the older clients that do not support rsa-sha2-256 or rha-sha2-512 keys, do support Elliptical Curves ciphers for host keys. You can consult this document: Multiple SSH Host Keys to generate an Elliptical Curve SSH Host Key and make it active alongside your current host key. This should allow most older clients to connect once they cache the new Elliptical Curve host key. If the administrators of the client software attempting the connections need to manually cache your server's Elliptical Curve host key, you can follow the instructions in the document to export the server's public key to send to the user.

Turn off FIPS 140-2 mode (not recommended): To disable FIPS 140-2 see 'Enable FIPS 140-2' in this article: Security Settings. 

Known SFTP clients not supporting FIPS-compliant Host Key Parameters

WinSCP releases below 5.20.0 (December 2021) only support ssh-rsa as the host key algorithm for SSH authentication. 

As such, if a user attempts to connect to SFTP using WinSCP releases below 5.20.0 and you have FIPS 140-2 mode turned on at 'Server Manager' > 'General', the authentication request will fail.

You can confirm this is the issue in the Cerberus logs. Look for an error entry similar to this:

[2022-11-08 14:07:39]: INFO [ 12] - [127.0.0.1]: - Client Identification: SSH-2.0-WinSCP_release_5.19.6
[2022-11-08 14:07:39]: ERROR [ 12] - [127.0.0.1]: - No common host key: [S: rsa-sha2-256,rsa-sha2-512] [C: ssh-rsa,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss]
[2022-11-08 14:07:39]: ERROR [ 12] - [127.0.0.1]: - Key exchange failed: Could not agree on key exchange parameters
[2022-11-08 14:07:39]:CONNECT [ 12] - [127.0.0.1]: - The client closed the connection
[2022-11-08 14:07:39]:CONNECT [ 12] - [127.0.0.1]: - Connection terminated

The resolution is for the WinSCP user to update to the latest WinSCP release (https://winscp.net/eng/download.php)