Securing SSL communication between Rational Team Concert and Urbancode Deploy

The Business Scenario

We recently had a challenge where a customer needed full end-to-end SSL/TLS encryption for communcation between IBM Team Concert, and Urbancode Deploy. They are a z/OS Mainframe customer implementing these tools for continuous Integration / continuous deployment operations.  Team Concert in this case, handles dependency build (compilation) of z/OS source code (COBOL, Assembler, etc). Once compiled, it packages the compiled outputs (load modules, CICS modules, DBRM, listings, and some non-compiled outputs such as REXX and JCL), and transfers this to Urbancode Deploy for deployment and configuration activities (deployment of datasets, DB2 BINDS, CICS new copies). The customer has RTC and Urbancode running on z/Linux instances on their z/OS, and communicating to the z/OS LPAR’s via hypersockets. While one may think this alone is secure enough, the are a situations where an employee may be able to view protected source code in transit. It is also an audit requirement that this be encrypted all the way through.

The Challenge

In this instance, the customer has their own certifacte authority (CA), to which all desktops and servers have the CA key in their browsers, and other trust stores. As part of our effort to automate as much as we can, we developed a plugin for Urbancode to kick off RTC promotions between streams when deploying to a production environment (this ensures that source and compiled output changes are kept in sync).  When we set up communication between the Urbancode Agent, we could not get a successful SSL/TLS handshake to the RTC server.

RTC and UCD communciations (z/OS)

The Solution

Something that is not documented well in the IBM Knowledge centers for RTC or UCD, is that when setting up SSL communication, the IBM JDK uses a separate trust store than its keystore. This means that the certificates you put into the configured keystore are ignored, and the JDK only looks at its default trust store. Even when attempting to override the keystores, it would not recognize the included CA certificate. The solution is found in the Knowledge Center for the IBM JDK, and this applies to the JDK for ALL platforms (Linux, Windows, and yes, even z/OS).  The IBM JDK looks for a javax.net.ssl.trustStore system property. If this is not set, it looks for the default keystore located at <java-home>/lib/security/cacerts. Thus unless you are directly overriding the system property, you must add your company CA root and intermediary certs to the <java-home>/lib/security/cacerts file.

Recommendations

1) Create your own Trust stores

All the default SSL/TLS keystores and trust stores use a default password. Thus even if you get the handshake working correctly, you still have a  gaping hole in your security matrix as anyone with the default password can manipuate your keystores. Managing access to the keystores and truststores is certainly a must, but we recommend using a separate, newly created keystore for each instance. This table below describes the default location for the keystore and trust store, as well as the default password for each (see, I told you it was not secure!)

First, create a new Truststore and put it on each VM or server where you will need access to it. This can be done using ikeyman, a graphical interface found in the IBM JDK bin directory. Put a password on it that is not the default password! Add your CA trust certs (root and all intermediary) as well as any other trust certs you wish to add. Note that for optimal security, add only your company CA trust certs to ensure that only the servers you trust to communicate with it can do so. You do not have to import all known CA certs (i.e. Verisign, Thawte, Go-Daddy, Google, etc)!

2) Create certs for your servers and agents

Create the CA certified key for each product (you can use the same key for all keystores on the same z/OS lpar). If you are using hypersockets, note that you may have two IP addresses, and two DNS names for the same LPAR or external system. You will need to add the additional DNS names into this as well.

3) Add the certs to your trust stores

Copy the certificate to the trust stores listed above (or create your own).

If you choose to use the default JDK truststore (please don’t), it is found in the IBM JDK at /usr/lpp/java/J8.0_64/lib/security/cacerts on z/OS. This trust store password is “changeit”. At the very least, you should not keep this password: you should …. change it!

4) Enable SSL communications

The Rational Build Agent (BLZBFA) runs unsecured by default. You will need to configure the SSL certificate in the bfagent.conf file, and restart the agent.

# Note: If using SSL, create or obtain PEM keystores from engine and set the proper key password.
#ssl_key_location /usr/local/bin/buildForgeKey.pem  <-- create a new one and put it where you want
#ssl_key_password <password> <-- This gets encrypted
#ssl_cert_location /usr/local/bin/buildForgeCert.pem <-- create a new one and put it where you want
#ssl_ca_location /usr/local/bin/buildForgeCA.pem <-- create a new one and put it where you want
# Note: The ssl_protocol can be TLSv1,TLSv1.1 or TLSv1.2. You select only one.
#ssl_protocol TLSv1
#ssl_protocol TLSv1.1
#ssl_protocol TLSv1.2  <--Use this !
#ssl_cipher_group ALL
#ssl_cipher_override <cipher_list>
#ssl_client_authentication true
#ssl_client_auth_sufficient true
#fips_enabled true
#gsk_ssl_kdb_password <password>  <-- Use bfagent -e <plaintext> to create the encrypted password from plain text.
#gsk_keyring_label <certificate label>
#gsk_ssl_protocol ALL
#gsk_ssl_cipher_v2 631
#gsk_ssl_cipher_v3 090201
#gsk_ssl_client_authentication true
#gsk_password_encrypt true 

Then, you will need to edit the build engine configuration in RTC. You should be using TLS 1.2 (which addresses issues such as the heartbleed bug identified a couple of years ago).

RTC Build Engine Configuration for the build agent.

5) Test communications

• Test communciaton between the RTC server and the build agent. (click the test button on the Build Agent tab)
• Test communciation between UCD and the UCD agent (visibile in the Agents tab, just click to test the agent)
• Ensure RTC communicates to UCD (attempt to create a package that gets shipped to UCD)
• Ensure the Urbancode agent can communicate to the RTC server – this is where you need to test your plugin, or groovy script, however this was created.

**If you are interested in our UCD plugin that automates RTC Enterprise Extension source code promotions, please contact us for details and pricing.

Lastly, I’ll leave you with the following references if you need more detailed steps for any configuration (or more background information):

References:

Customizing the Default Key and Trust Stores, Store Types, and Store Passwords (IBM JDK): 

https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/customizingstores.html

Installing a security certificate (RTC with Liberty Server):

https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.6/com.ibm.jazz.install.doc/topics/t_install_server_certificates.html

SSL configuration (for IBM Urbancode Deploy)

https://www.ibm.com/support/knowledgecenter/en/SS4GSP_6.2.7/com.ibm.udeploy.doc/topics/SSLinstall.html

bfagent.conf file for the Rational Build Agent on z/OS systems

https://www.ibm.com/support/knowledgecenter/SSYMRC_6.0.4/com.ibm.jazz.install.doc/topics/RTCz_agent_configure_bfagent_conf.html