Configure FIPS mode in the Server

Before you proceed, make sure of the following:

• You have already configured the SM Server's JRE for FIPS mode according to the JRE that the SM Server uses (IBM JDK/JRE or Oracle JRE). For details, see the Server section in Configure Java for FIPS mode.

• You have already generated the following certificate files for the SM Server. For details, see Generate FIPS validated certificates for the SM Server and other components.

  • The CA certificates file: \certs\smcacerts.p12
  • The Server's trusted clients keystore file: \certs\trustedclients.p12

  • The Server keystore file: \key\sun-server-smserver.mycompany.net.p12

To enable FIPS mode in the SM Server, complete the following tasks.

After enabling FIPS mode, you must upgrade your database's encryption algorithm from DES to AES by running the sm -upgradeencralg command.

1. Make sure your server and applications have upgraded to version 9.32 or later.

2. Make a backup of your database.

   Caution FIPS mode requires the database to upgrade to the 256-bit AES encryption algorithm. Because the upgrade is irreversible, you need to make a backup of your database.

3. Check the length of each encrypted field in your database, and increase the length if needed.

   Tip Use the following formula to determine a safe length for an AES encrypted field (in bytes):

   Safe_Length=32 + 2 * source_string_length

   Where: source_string_length is the length of the source string (for example, if you enter an 8-character password, the source string length is 8 bytes).

4. Upgrade your applications to the AES data encryption algorithm.

   FIPS mode requires the use of the 256-bit AES data encryption algorithm for the database. By default, the Service Manager applications uses the 64-bit Data Encryption Standard (DES) encryption algorithm. You must upgrade the database from the DES encryption algorithm to AES by running the sm -upgradeencralg command.

   Caution This algorithm upgrade is required only when you want to enable FIPS mode.

   1. Make sure you can connect to the server successfully.
   2. Make sure all Service Manager server processes are stopped.
   3. Check that the server's configuration file (sm.ini) does not include the fipsmode parameter.

   4. Run the following command:

      sm -upgradeencralg                        

      Note If you have many tables with encrypted fields, running this command may take quite a while; if you have already run this command on the database, running it again will have no effect and do no harm to the database.

      Tip You can specify a 32-character alphanumeric value for this parameter, as the new encryption key; if you do not specify any value, a default 32-character (256-bit) encryption key is used. You can also change your encryption key later using the changeencrkey parameter.

   5. Check the server log (sm.log) for any errors.

5. In the server configuration file (sm.ini), set the following parameter.

    fipsmode:2

   Note In versions earlier than 9.41p3, only data encryption was supported for FIPS mode and "fipsmode:1" was used. Starting with version 9.41p3, full FIPS mode is supported and "fipsmode:2" is used instead. For backward compatibility, if you want to implement only AES-256 data encryption, you can still use "fipsmode:1" to run Service Manager in partial FIPS mode.

6. Start the Server.

7. If your server fails to start with an error message in the Server log that indicates the system failed to load libeay32.dll, do the following:

   1. Delete libeay32.dll, which is located in the Server's RUN folder.
   2. Rename libeay32_rba.dll in the same folder to libeay32.dll.
   3. Restart the Server.

   Note The libeay32_fba.dll file is a copy of libeay32.dll for backup purposes. The libeay32_rba.dll file is a variant of libeay32.dll, with a relocatable base address. If the system fails to load the default DLL (libeay32.dll), use this file to overwrite libeay32.dll.

Follow these steps:

1. Copy the certificate files listed in the previous step to the SM Server's RUN directory:

   • The CA certificates file: smcacerts.p12
   • The Server's trusted clients keystore file: trustedclients.p12

   • The Server keystore file: sun-server-smserver.mycompany.net.p12

   See the following figure for an example.

   

2. Add the following parameters to the sm.ini file in the Server's RUN folder.

   Caution The example certificate file names and passwords used in the following lines are the default ones specified in the certificate generation batch files provided by HPE. For details, see Generate FIPS validated certificates for the SM Server and other components. If you uses different file names or passwords when generating your certificates, be sure to replace them with your own values.

   Tip The ciphers and sslEnabledProtocols parameters are specified here with their default values. Since Service Manager will use their default values when the two parameters are not present in the sm.ini file, you can omit them safely. These parameters are provided just in case you do not want to use their default values.

   fipsmode:2
   ssl:1
   sslConnector:1
   ssl_reqClientAuth:2
   keystoreFile:sun-server-smserver.mycompany.net.p12
   keystorePass:serverkeystore
   ssl_trustedClientsJKS:trustedclients.p12
   ssl_trustedClientsPwd:trustedclients
   truststoreFile:smcacerts.p12
   truststorePass:changeit
   ciphers:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA
   sslEnabledProtocols:TLSv1,TLSv1.1,TLSv1.2

   Note When running under JDK 8, the JDK’s SunJSSE TLS implementation is unable to negotiate a TLS handshake when using an ECDHE cipher suite.

3. Restart the SM Server.

   Next, you need to enable FIPS mode in the Windows Client to verify that your FIPS mode is successfully enabled on the Server side.