Configure FIPS mode in the Server

Prerequisites

Before you proceed, make sure of the following:

Search for	Example	Results
A single word	`cat`	Topics that contain the word "cat". You will also find its grammatical variations, such as "cats".
A phrase. You can specify that the search results contain a specific phrase.	`"cat food"` (quotation marks)	Topics that contain the literal phrase "cat food" and all its grammatical variations. Without the quotation marks, the query is equivalent to specifying an OR operator, which finds topics with one of the individual words instead of the phrase.

Search for	Operator	Example
Two or more words in the same topic	`AND` `and` `+` (plus symbol) `&` (ampersand)	`cat AND dog` `"cat food"+milk` `"cat food"&"dog food"`
Either word in a topic	`OR` `or` `\|` (pipe)	`cat OR dog` `cat \| dog`
Topics that do not contain a specific word or phrase	`NOT` `not` `!` (exclamation point)	`NOT cat` `! dog`
Topics that contain one string and do not contain another	`^` (caret)	`cat ^ mouse`
A combination of search types	`( )` parentheses	`cat + (dog \| mouse)` `cat \| dog + (! mouse)`

You have already configured the SM Server's JRE for FIPS mode according to the JRE that the SM Server uses (IBM JDK/JRE or Oracle JRE). For details, see the Server section in Configure Java for FIPS mode.

You have already generated the following certificate files for the SM Server. For details, see Generate FIPS validated certificates for the SM Server and other components.

The CA certificates file: \certs\smcacerts.p12
The Server's trusted clients keystore file: \certs\trustedclients.p12
The Server keystore file: \key\sun-server-smserver.mycompany.net.p12

To enable FIPS mode in the SM Server, complete the following tasks.

Caution To enable FIPS mode in a scaling environment, you must configure FIPS mode in each of the SM server nodes.

Steps to enable FIPS mode for data encryption

After enabling FIPS mode, you must upgrade your database's encryption algorithm from DES to AES by running the sm -upgradeencralg command.

Make sure your server and applications have upgraded to version 9.32 or later.

Make a backup of your database.

Caution FIPS mode requires the database to upgrade to the 256-bit AES encryption algorithm. Because the upgrade is irreversible, you need to make a backup of your database.

Check the length of each encrypted field in your database, and increase the length if needed.

Tip Use the following formula to determine a safe length for an AES encrypted field (in bytes):

Safe_Length=32 + 2 * source_string_length

Where: source_string_length is the length of the source string (for example, if you enter an 8-character password, the source string length is 8 bytes).

Upgrade your applications to the AES data encryption algorithm.

FIPS mode requires the use of the 256-bit AES data encryption algorithm for the database. By default, the Service Manager applications uses the 64-bit Data Encryption Standard (DES) encryption algorithm. You must upgrade the database from the DES encryption algorithm to AES by running the sm -upgradeencralg command.

Caution This algorithm upgrade is required only when you want to enable FIPS mode.

Make sure you can connect to the server successfully.
Make sure all Service Manager server processes are stopped.
Check that the server's configuration file (sm.ini) does not include the fipsmode parameter.
Run the following command:
```
sm -upgradeencralg                        
```
Note If you have many tables with encrypted fields, running this command may take quite a while; if you have already run this command on the database, running it again will have no effect and do no harm to the database.

Tip You can specify a 32-character alphanumeric value for this parameter, as the new encryption key; if you do not specify any value, a default 32-character (256-bit) encryption key is used. You can also change your encryption key later using the changeencrkey parameter.
Check the server log (sm.log) for any errors.

In the server configuration file (sm.ini), set the following parameter.

 fipsmode:2

Note In versions earlier than 9.41p3, only data encryption was supported for FIPS mode and "fipsmode:1" was used. Starting with version 9.41p3, full FIPS mode is supported and "fipsmode:2" is used instead. For backward compatibility, if you want to implement only AES-256 data encryption, you can still use "fipsmode:1" to run Service Manager in partial FIPS mode.

Start the Server.

If your server fails to start with an error message in the Server log that indicates the system failed to load libeay32.dll, do the following:

Delete libeay32.dll, which is located in the Server's RUN folder.
Rename libeay32_rba.dll in the same folder to libeay32.dll.
Restart the Server.

Note The libeay32_fba.dll file is a copy of libeay32.dll for backup purposes. The libeay32_rba.dll file is a variant of libeay32.dll, with a relocatable base address. If the system fails to load the default DLL (libeay32.dll), use this file to overwrite libeay32.dll.

Steps to enable FIPS mode for TLS connections

Follow these steps:

Copy the certificate files listed in the previous step to the SM Server's RUN directory:

The CA certificates file: smcacerts.p12
The Server's trusted clients keystore file: trustedclients.p12
The Server keystore file: sun-server-smserver.mycompany.net.p12

See the following figure for an example.

Add the following parameters to the sm.ini file in the Server's RUN folder.

Caution The example certificate file names and passwords used in the following lines are the default ones specified in the certificate generation batch files provided by Micro Focus. For details, see Generate FIPS validated certificates for the SM Server and other components. If you uses different file names or passwords when generating your certificates, be sure to replace them with your own values.

Tip The ciphers and sslEnabledProtocols parameters are specified here with their default values. Since Service Manager will use their default values when the two parameters are not present in the sm.ini file, you can omit them safely. These parameters are provided just in case you do not want to use their default values.

fipsmode:2
ssl:1
sslConnector:1
ssl_reqClientAuth:2
keystoreFile:sun-server-smserver.mycompany.net.p12
keystorePass:serverkeystore
ssl_trustedClientsJKS:trustedclients.p12
ssl_trustedClientsPwd:trustedclients
truststoreFile:smcacerts.p12
truststorePass:changeit
ciphers:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA
sslEnabledProtocols:TLSv1,TLSv1.1,TLSv1.2

Note When running under JDK 8, the JDK’s SunJSSE TLS implementation is unable to negotiate a TLS handshake when using an ECDHE cipher suite.

Restart the SM Server.

Next, you need to enable FIPS mode in the Windows Client to verify that your FIPS mode is successfully enabled on the Server side.

Next step: