The Rational Integration Tester HTTP Proxy: Recording HTTPS

This is the third in my series of blog posts on the Rational Integration Tester HTTP proxy.  If you need to catch up, see

I sketched out a plan for this series several weeks ago.  The topic of this third blog was to be Recording HTTPS.  In the meantime, John Chewter (one of the smartest guys I know, by the way), has written two articles on the IBM developerWorks site on Using SSL with Rational Integration Tester:

JC has crammed a bunch of really valuable guidance into these articles.  He reveals the secrets behind a lot of the magic of SSL and HTTPS communications and how Rational Integration Tester works with it.  These two articles are absolute must-reads before you go any further.  Since there is little value in me just repeating what JC already documented, I decided to try hitting this from a little different angle that builds on JC’s articles.  Armed with the knowledge he provides, let’s walk through a brief example applying these concepts to a reasonably realistic scenario.

Let’s say we have a client web application running on Tomcat that consumes a hotel provider web service.  In order to record the secure conversation and to later properly route requests to a stub, we will insert the RIT HTTP Proxy between the client and server. In this example, the client will require authentication from the service but the service will accept requests from anyone and will not require mutual authentication.

Simple Architecture configured for Proxy

Simple Architecture configured for Proxy

The proxy configuration

The Rational Integration Tester proxy configuration is controlled by the registration.xml file in the proxy’s installation directory (on Windows, the default is C:\Program Files\IBM\RIT-Platform\httptcp).  The area of the registration.xml file we want to focus on is the https-proxy section.

Portion of the registration.xml file

Portion of the registration.xml file

The outboundKeyStoreFile is used in mutual authentication schemes where the proxy must authenticate with a server.  Our example is a single-authentication architecture so outboundKeyStoreFile is not relevant in our case.

The keyStoreFile is what the proxy will use when it is acting as a server.  This is the primary role it will play in a typical server authentication scheme.  RIT ships with a preconfigured greenhat.jks keystore which contains a preconfigured key called mykey.  In other words, when the client application requests the proxy to provide its credentials for authentication, the proxy will send it the “mykey” public key.

As John mentioned in his article, you can certainly use this key for testing.  The only thing to consider is that your client app will need to be configured to trust it.  The other option is to use a key that is already trusted by the client – or more precisely, a key that is signed by a certificate authority that the client trusts.  Since John walked you through the latter, I will stick with the greenhat keystore and the mykey key.

Configuring the application client

We know we will need to add the greenhat certificate to the client’s trust store.  This will tell the client application that it can trust the proxy.  but first,  we need to know where the client’s trust store is.  Therefore we will verify and update the configuration of the client application first.

What you need to do will depend on what client technology you are using.  In this example, the client is a web application running in Tomcat.  Tomcat’s configuration settings are generally in the setenv.bat (or setenv.sh) file used to launch it.  We add two JVM arguments to the launch configuration to cause Tomcat to route HTTPS communications through the proxy.  So if the proxy is running on a host at 192.168.100.100, we add the following lines:

JVM arguments to enable proxy

JVM arguments to enable proxy

Note that unlike in the last blog, the prefix of the directives is https, not http.  Also, the default port the proxy uses for HTTPS traffic is 3129, although this can be changed in the registration.xml file.

Within the setenv.sh file, we see existing truststore JVM arguments:

Tomcat client truststore configuration

Tomcat client truststore configuration

These directives tell Tomcat where to find the trust store and how to open it.  The trust store contains certificates for servers that can be trusted or certificates for certificate authorities that the client application can consider reputable enough to vouch for a server.  This is where we will need to add the greenhat certificate which will enable the client to trust the proxy.

Exploring the keystore

System administrators will commonly use tools like the Java keytool program to manipulate keystores.  But RIT has a tool built into the UI that enables you to manage your configuration from within the project.  To use it, open the Physical view of the Architecture School perspective. Under the General toolbar, select Identity Store.  Use the Select browser to locate the wwtravelTrustStore.jks file.  You will be prompted for a keystore password.  This is the password in the setenv.bat file, above.

You will find at least one existing certificate in the trust store: wwtravel.  This is the certificate for the hotels server host which tells the client it can trust the hotel service.  We use the Import Key/Certificate(s) button to import the greenhat.cer file provided with RIT.  This certificate corresponds to the “mykey” public key that the proxy will send when the client attempts to open up a conversation with it.

identity store for wwtraveltruststore.jks

identity store for wwtraveltruststore.jks

Bounce the Tomcat server to get the changes to take effect.

Now you may be looking back at the simple architecture diagram at the beginning of this post and wondering why we need to go through all this work for the connection on the left side but not the connection on the right site.  Don’t we need to add the server’s certificate to the proxy so that the proxy will trust it?  To avoid putting you through all that, the RIT proxy is designed to “trust anybody”.  In other words, when the server sends its public key to the proxy, the proxy will accept it, no questions asked.

While we are in the Physical View, we will create an Identity Store for the greenhat.jks keystore as well.  We will need to reference that when we create the RIT model.  The keystore password is “passphrase”.

Identity Store for greenhat.jks

Identity Store for greenhat.jks

Verify the configuration

With the proxy settings configured on the client, the proxy configured to supply an SSL key when requested and the client’s trust store configured so that it will accept that key, everything should be functional.  The client should be able to make requests of the live service and get responses with the messages being passed through the proxy.  You should run two tests to verify your setup at this time:

  • With the proxy stopped, verify that the client is unable to make a request of the service.  This proves that you have all the plumbing configured correctly.  If the client can get a response from the service with the proxy stopped, it is bypassing the proxy.
  • With the proxy running, verify that the client can make a request and get a response from the service.  This proves that you have all the authentication set up right.

Configuring the RIT model

Next, we will configure the model in Rational Integration Tester for recording.  This is done just like it was for HTTP with one additional step.  On the SSL tab of the Web Server physical resource, you much check “Use SSL”.  This tells RIT tests to invoke the SSL handshake when they connect to the service.

There are two groups of SSL settings on the Web Server SSL tab that affect how RIT operates in an SSL environment.  The Provided Certificates area defines how RIT should behave when it is acting as a Server.  In other words, if you will run stubs on this endpoint, these settings will be used to define which key in which keystore RIT will send to the client when asked to identify itself.  Check Specify Provided Certificate; select the greenhat.jks keystore and the mykey key.

HTTPS Web Server physical resource

HTTPS Web Server physical resource

The Trusted Certificates group defines how RIT should operate when acting as a client.  RIT’s default behavior for tests is again to “trust anybody”, but you can override that behavior by specifying trusted certificates or verifying the certificate chain.

Recording HTTPS

With the keystores and model settings in place, recording HTTPS is really no different than recording HTTP.  Create an event monitor by right-clicking on the element in the Logical View and start recording in the Recording Studio perspective.  Because the proxy is acting as a “man in the middle”, it sees and records the messages in clear text.  You can modify tests and stubs based on those recordings just as you would plain text artifacts.

So working with SSL shouldn’t be scary.  It will require some understanding of the components of SSL (keystores, certificates, keys, etc.) and some understanding of the handshake process, but that should be expected.  If you are performing component-level integration tests, you will need to know how those components interact with each other – at least long enough to configure RIT.  Once the plumbing is in place, RIT makes it easy to work with data in plain text.

The Rational Integration Tester HTTP Proxy: Recording HTTP

This blog post is the second in a series about the Rational Integration Tester HTTP Proxy.  If you are just now joining us, I suggest you drop back and first read

The Rational Integration Tester HTTP Proxy: What is it?

Now that we have covered what the proxy is, what it is used for and the basics of installing it, we will look deeper into using the proxy for its first application – recording.  There probably is no better way to describe how the proxy works than “man in the middle”.  That term strikes fear into most people.  You think of malicious attacks on web sites bent on malfeasance.  In such attacks, a 3rd party finds a way to convince a client application to allow it to listen in as that client talks to a service it uses.  In an attack like that, the goal is to enable the 3rd party to harvest information in those messages.  Although the Rational Integration Tester HTTP proxy uses the same general approach, you are the 3rd party and the purpose of harvesting information in those messages is ultimately to create either tests that replicate the client or stubs that emulate the service.

Yes, there are security implications here.  And that’s exactly why you really need to understand what is going on and how to ensure you are only giving the Rational Integration Tester HTTP Proxy access to your messages.  The next post in this series will dig deeper into security, but we need to get a foundational understanding of how to set up an unsecured system first and then build on that.

Overview

The overall process you will follow consists of the following steps:

  1. Understand the System Under Test (SUT)
  2. Configure the client to route requests through the proxy.
  3. Verify the SUT works correctly with the proxy in place.
  4. Model the endpoint in Rational Integration Tester and verify you can access the service.
  5. Use RIT to record as you send messages to the service by exercising the client.

The Process

Understanding the System Under Test

This probably sounds obvious.  Of course you need to understand the system you are testing.  But the system you want to test is probably a complex mishmash of subsystems and sub-subsystems sending messages through all sorts of protocols and message formats.  What do you need to know about the system and what is just extra information that doesn’t help?  Ultimately any given communication path you want to record boils down to a client sending messages and receiving responses from a service. Let’s consider a simple example where a web client application running in Tomcat consumes a service that provides hotel search and booking:

Simple Communication Architecture

Simple Communication Architecture

The key details you need to know are the hostname of the server hosting the service and the port number on which the service listens.  In the example, the hotel service runs on a fictitious server called myHotelServer.com on port 8089.  Once we start considering authentication and security, there will be other things you will need to know, but we will get to that later.

Configuring the client

Recording messages with the proxy requires that you reconfigure your System Under Test (SUT) a bit.  This might sound like invasive testing.  The thought that you must tweak your system in order to enable testing might rub you the wrong way.  But then again, think about it this way: should your SUT allow just anyone to tap into the wire and record messages?  Rest assured that Rational Integration Tester – or any other tool – can’t somehow sneak into a properly configured and secured system to record data.  You must deliberately configure your system’s client to allow this recording to occur.  This is a necessary thing – and a good thing.

The specific steps required to configure your client will depend on the container your client runs in.  Some specific technologies are addressed in the InfoCenter.  In the case of most Java-based containers, you will be able to add two Java startup switches to the command line that launches the container and then restart the container.  For our example where the client is running in Apache Tomcat, we would add the following switches to the script that launches Tomcat:

-Dhttp.proxyHost=myHotelServer.com –Dhttp.proxyPort=3128

Note that the InfoCenter recommends editing catalina.bat or catalina.sh.  I prefer to keep environment settings like this localized to the setenv.bat or setenv.sh scripts.

Simple Architecture configured for Proxy

Simple Architecture configured for Proxy

As long as your client application correctly uses the standard Java HTTP client libraries, this will cause it to route messages through the proxy.  There are many corner cases where simply setting these two switches will not be sufficient.  This all depends on how the client is implemented.  For now, let’s assume this works for our example.  Troubleshooting these corner cases may be a good topic for a future blog article.

Verifying the System Under Test

Before you go any further, it is very important that you verify that A) you haven’t broken your SUT and B) messages are actually routing through the proxy.  It is easy to make typos when modifying textual startup scripts so double-check your edits.

Start the Rational Integration Tester HTTP proxy.  As we discussed in the previous blog article, you normally want to have the proxy installed as a Windows service and have it start automatically upon boot.  Now is a good time to NOT do that.  Go to the Windows Services and stop the IBM RIT HTTP Proxy service.  Now navigate to your proxy install location (C:\Program Files\IBM\RIT-Platform\httptcp by default) in a command prompt and launch startup.bat.  Many lines of logging information will scroll by.  This may be worth referencing later if you have problems.

If you haven’t done so already, restart your client application.  You will need to restart for the startup directives to take effect.  It is worth checking log files in your client that might indicate whether or not the client picked up your startup directives.

Now try invoking the service from the client.  The first thing to verify is that the client gets the response you would expect and does not get errors.  If your client isn’t able to send the request or get a response back, then you broke something.  One thing to verify is that you started the proxy.  If the proxy isn’t running, no messages will get through.

Secondly, check the command window where you launched the proxy.  You should see a number of new logging messages.  In particular, look for the string “proxy rule found: Proxy on port 3128”.  This indicates that the request made it from the client to the proxy.  Further down in the messages you should see something like “Sending request message to endpoint”.  A little further down you should see “Reading response message from endpoint”.

Proxy messages indicating a successful route

Proxy messages indicating a successful route

Don’t bother proceeding if this isn’t happening for you.  There are a number of possible reasons that messages aren’t getting through.  Firewalls are common blockers.  You will need to correct the situation because nothing beyond this point will work.

Configuring the RIT model

With the proxy in place and verified, we are ready to model the system in Rational Integration Tester in preparation for recording.  I wrote some weeks ago about the model in Rational Integration Tester.  Since this post is about the proxy, I’ll stick to the bare minimum modeling and focus on what we need to get a recording.

At a minimum, you will need a logical HTTP Connection and a physical Web Server in your model.  Create the HTTP Connection in the logical view.  Create a physical Web Server.  On the Config > Settings tab, enter the host and port of the hotels server.

Physical resource for the Web Server

Physical resource for the Web Server

Note – you should NOT be entering any information about the proxy here.  Rational Integration Tester handles all the proxy stuff behind the scenes with the help of Rational Test Control Panel.  All you need to do in the model is describe the SUT.

Also, switch the Recording Mode on the Recording tab to External Proxy Server.  This is not the default.  If you forget to switch the recoding mode to use the proxy, you will get no errors or messages, but you will get no recording.  If it is any comfort, you won’t be the first to forget this.  It happens all the time.

Recording Mode setting on the physical Web Server

Recording Mode setting on the physical Web Server

Recording events

Start up recording in RIT on the HTTP Connection.  You should see a couple more messages appear in the proxy command window such as “Added recording rule…”.  This appears when RIT registers a rule in the proxy which tells the proxy to copy all the messages it sees and send them to RIT where they will appear as events in the Recording Studio perspective.

Proxy message indicating a recording rule

Proxy message indicating a recording rule

In the events view, you should now see the messages that were recorded.  In our example, the client sent a request consisting of an HTTP POST to the getHotels operation on the Hotel Server.  The service replied with a list of hotels in an XML message.

Recorded events in RIT

Recorded events in RIT

These events can then be used to create operations, requirements, tests or stubs.

With the recorder engaged, here is what you really have in place:

Recording Architecture

Recording Architecture

The request comes from the client destined for the service, but routed through the proxy.  The recording rule that has been registered with the proxy causes it to send a copy of the request to RIT at the same time that it forwards on the request to the service.  When the service replies, the proxy once again sends a copy of the reply to RIT while forwarding on the reply to the client.

I have left out some important details such as the role of the Rational Test Control Panel but I wanted to keep things fairly simple for now and focus on the role of the proxy.  Hopefully this article has helped clear up any confusion you may have had around recording basic HTTP with the proxy.  In my next post, I will look at recording HTTPS which adds another dimension to things.