Domains, Environment and Projects – Administration

This is a follow-up to my previous post Domains, Environments and Projects – making sense of it all.

Now that we understand the purpose of Domains and their related artifacts in the Rational Test Control Panel, let’s take a look at how we can make use of them.

Visibility

Again, as described in the KnowledgeCenter, A domain represents a logical grouping of related systems that are part of a real business project and it is the basic unit of management within IBM Rational Test Virtualization Server.  In the last post, I mapped each organizational division to a domain.  This will enable JKE to organize which stubs are published and visible in which domains.  That will help us make only relevant stubs visible to the various teams.

Security Access

Domain-level security is disabled by default.  As an administrator, go into the Administration > Security tab of Rational Test Control Panel and enable it.

Enable Domain-level security

Enable Domain-level security

Once enabled, any user that is not specifically added to a domain will not have access to anything in that domain.

No domains for unauthorized user

No domains for unauthorized user

Users are added to domains in the Administration > Domains and environments area by selecting the domain and then the Users sub-menu and clicking the green add icon.  Users can be given three levels of access: Domain administrator, User and API User.

Add a user to a domain

Add a user to a domain

Now that user will see any available domains in the domain selection box at the upper-right of RTCP.

Domains available to a user appear in the domain selection list

Domains available to a user appear in the domain selection list

With this in place, you not only provide visibility constraints, but also access constraints.  But, access constraints apply not only to users, but to proxies and agents as well.  Since we have not yet configured any proxies or agents for domain-level security, they cannot register to domains or environments within those domains.  If our domain user looks at the Environments tab (was VIE tab prior to v8.6), he will see this message:

No agents registered for this domain

No agents registered for this environment

This brings us to our third use for domains – controlling access to services.

Proxy/Agent access control

Proxy and Agent access control is implemented with security tokens.  An administrator creates a security token which is used by the proxy or agent to authenticate to RTCP.  The security token’s permissions determine which domains the proxy or agent can register with.  The security token is associated to a user when it is created.  The proxy or agent therefore gets the access of the user who corresponds to the security token being presented.

As an administrator user, create a token in the Administration > Security tab.  Select the user to which the security token will be bound and enter a description for the token.

Create user security token

Create user security token

Now provide the security token to the proxy or agent configuration settings.  For proxies, this is done by pasting the security token string into the security-token attribute of the server element of the registration.xml file for the proxy.

Adding the security token to the proxy registration.xml file

Adding the security token to the proxy registration.xml file

For agents, this is done by pasting the security token string into the security-token attribute of the rtcpURL element of the Agent.config file for the agent.

Adding the security token to the Agent.config file

Adding the security token to the Agent.config file

Restart any proxies or agents you have modified.  The proxies and agents started using this security token will be visible in all domains and environments, but they can only be used in domains where the user associated with the security key has permissions to work.

Proxy/Agent visibility

A proxy or agent’s visibility can be restricted as well.  This is done by the domains element in the proxy’s registration.xml or the agent’s Agent.config file.  For example, to cause my proxy and agent to only be visible to the Mobile Banking domain, I would add the following:

Restricting proxies or stubs to a specific domain

Restricting proxies or stubs to a specific domain

You can further restrict a proxy or agent’s access to only given environments within a domain.

Restricting proxies or stubs to a specific environment within a domain

Restricting proxies or stubs to a specific environment within a domain

Wrapping it all up

Now this becomes really powerful.  So consider again our multiple JKE teams all trying to build client applications that leverage the same payment calculator service.  Each team wants to use a stub for the payment calculator service, but they each want their own variant of that service.  The domain architecture of Rational Test Control Panel and the
ability to control which domains the proxies and agents can register into would enable each team to have their own personal proxy/agent pair visible only in their domain.  The members of each team would have permissions to their own domain.  Each team would publish and run their own Payments Calculator stub to their own domain.  Now, without changing the endpoints of any of the client applications, each team’s proxy will route requests from their client to their stub.  There are all sorts of options where multiple domains could share proxies and agents, thereby enabling them to use a common stub.  This could be part of a strategy to use incremental integration testing at an enterprise level.

Domains with separate and shared proxies

Domains with separate and shared proxies

Advertisements

Domains, Environments and Projects – making sense of it all

I was recently supporting an opportunity that involved multiple test teams supporting multiple products that all needed to virtualize common dependent services. It’s not that difficult to visualize how to create, publish and use virtual services in a small, well-contained project, but when you start looking at this from an enterprise perspective, it can turn into a large hairball quickly.

A fictional example

The Auto Loan application in the Personal Banking division of JKE Bank had a need for computing loan payments. It’s a pretty simple operation – you provide the loan amount, term of the loan and interest rate. In return, you get a value for the monthly payment amount. So the Personal Banking Auto Loan team encapsulated that capability in a web service, published it to the JKE world and built the Auto Loan application to leverage it.

Pretty soon, the Personal Banking division software architects decided it didn’t make sense to duplicate this functionality across all the applications in their portfolio, so they decreed that all Personal Banking apps that need to calculate payments must use the common Payment Calculator service. The two applications delivered by the Personal Banking team leveraged a common, consistent payment calculator.

At an executive golf outing, the VP of Personal Banking was bragging to the other VPs how efficient and cost effective his software team was as demonstrated by the common payments calculator service. Not to be outdone, the VP of Corporate Banking wanted in on this action and demanded that his peer share the payment calculator with his group. Just after another round of beers and just before a brawl broke out on the 12th hole, they agreed to allow the VP of Core Services to take ownership of the Payment Calculator service and deliver it as a common service across JKE. By the end of the round, The VPs of Billing, On-Line Banking and Mobile Banking were also all committed to leveraging this new, awesome common service.

Back in the office the following week, everyone realized the Payment Calculator was designed for auto loans with a maximum of five years so enhancements needed to be added to support longer loan terms. The Auto Loan team had to redo a bunch of integration testing to verify the changes didn’t break the Auto Loan application – which they did – and had to coordinate getting the regressions fixed – which they did. The Corporate Banking team required the addition of provisions for semi-annual payments instead of monthly payments. The Billing team wanted the service to return the total amount of interest that would be paid across the term of the loan. The On-Line team wanted the ability to deduct down payment values. The Mobile team wanted all values to be converted to local currencies. Each of the respective test teams realized they each had Dev, Test, Performance and UAT environments for each application. They just shook their heads asking “How are we ever going to test our applications while all these changes are going into the service we depend on?”

Projects and Environments dependent on a common service

Projects and Environments dependent on a common service

Hopefully you now see the problem. Today’s software systems are interconnected webs of loosely coupled applications; yet stable versions are required to complete integration testing. We have established how stubs or virtualized services can help, but how do you coordinate all this across so many teams and applications?

 Rational Integration Tester and Rational Test Control Panel Concepts

Let’s take a quick look at several Rational Integration Tester concepts that help us manage this challenge.

The Project

I can’t think of any other word in IT that is more overloaded and overused that “Project”, but nonetheless, it is a fundamental concept of Rational Integration Tester. The project is a physical disk location that contains files. From the Rational Integration Tester Workbench user’s perspective, the project is the container that holds all of your development artifacts including tests, stubs, datasets and other things. Let’s leave it at that – the Rational Integration Tester project is the disk location where you store the development artifacts related to the tests and stubs you are creating with the Workbench.

There is no Project concept in Rational Test Control Panel. You typically have a small number (maybe one) of Rational Test Control Panel instances, even in a large deployment where you may have many RIT projects. So the relationship between RIT project and RTCP instance is a many-to-one relationship.

The Environment

Every software project – err – “undertaking” must go through multiple stages in its journey to production delivery such as Dev, Test, Performance, UAT and Prod. There is hardware and software devoted to testing in each of these stages which is generally called an Environment. So although we are talking about one system, the implementation may vary considerably between, say, a Dev environment on very simple systems like Tomcat to HA WAS deployment in UAT or Prod. Rational Integration Tester captures the physical details in its own Environment artifact in the Rational Integration Tester project. The reason for this is that we need to know the implementation details of each environment to know how to apply tests and/or stubs to those environments.

The RIT environment maps directly to the environment in RTCP. Stubs, for example, can be created once in RIT and then applied to any or all environments. You can think of it this way: When a stub is deployed into RTCP, it carries along with it the environment details such as hostnames, IP addresses and port numbers. Stubs that have been deployed into RTCP are specific to the environment(s) into which it is deployed.

The Domain

The domain is a bit more of an obscure concept. A domain is an Rational Test Control Panel artifact that represents a logical grouping of related systems that are part of a real business project. It may be served by one or more IBM Rational Integration Tester projects, depending on how large or small the domain is, and how many Rational Integration Tester users are working on it. A domain can contain one or more environments.

The domain is much more loosely coupled to Rational Integration Tester. There is no domain artifact in RIT – you simply choose the domain into which you want to publish your stubs when you publish to RTCP.

Fictional example meets Rational Integration Tester

So let’s look at how our fictional JKE organization might apply RIT and RTCP to support their efforts to virtualize their common Payment Calculator. Keep in mind this is not the only way to implement a solution but one way.

Each of these application teams wants to create a stub that will represent the “to be” status of the Payment Calculator service that implements the new functionality they require. The Personal Banking Home Loans application team needs to virtualize the service supporting longer term loans. The Corporate Banking teams need to virtualize a service calculating semi-annual payments. The Billing team wants to virtualize the Total Interest Paid response value and so on.

So each team creates their own Rational Integration Tester project and implements their own stubs within their separate project. They each define their individual Environments within their projects.

Each of the application teams is mapped to a Domain in Rational Test Control Panel. This enables logical separation between the teams. The structure in the Environments page of RTCP will look something like this:

Domains and Environments in RTCP

Domains and Environments in RTCP

There is one area we could debate heavily on this proposal and that is the decision to have each team maintain their own separate stub in their own separate RIT projects. You may believe it would be better to have a single stub within a single project. I can definitely see why that would appear to be desirable. Rational Integration Tester does provide SCM integrations for version control. However, my perception is that these teams are all working on very different tasks in parallel. SCM is good for keeping track of sequential versions of artifacts like stubs but it isn’t so good at keeping track of parallel development efforts. RIT’s SCM integration definitely has no capabilities for merging changes together later. So my reasoning for keeping separate projects is based on my perception that these fictional teams want to keep their distance from each other and will for the foreseeable future. Again, this point could be debated and there are many other factors to consider.

So we have a scenario to build upon. In my next blog, I will look at how you can control permissions and access to domains and environments for both people and services such as Agents and Proxies.

What’s New?

I’m finally recovering enough from last week’s IBM Innovate conference to jot down a few notes on exciting stuff I saw in the testing space at the show.

Rational Test Workbench Mobile Test Edition V8.5.1

Support for testing mobile applications – native, hybrid and web – has been in Rational Test Workbench for a while now.  For those organizations that are looking for a complete package that includes mobile UI test automation, performance testing, service virtualization, integration testing and even UI test automation for desktop applications, Rational Test Workbench provides it all in one package.  RTW has been a compelling offering for enterprise clients for that reason.

However, if you only need a hammer, you don’t want to buy the whole tool chest. So as of now, you can buy just the mobile UI test automation capability in a lower cost offering.  This not only gives customers what they want, but it also makes the Rational offering more price competitive for mobile test automation.

Rational Test Workbench V8.6

There are some really exciting updates to the service virtualization and integration testing capabilities in Rational Test Workbench as well.  I would categorize the improvements in the following:

  • SAP support improvements
  • Support for MQ Telemetry Transport (MQTT)
  • Topology Discovery
  • Mainframe support enhancments
  • Other stuff

SAP support improvements

SAP support has been enhanced to include support for SAP XI 3.0 and SAP PI 7.31 and above for both integration testing and service virtualization.

Support for MQ Telemetry Transport (MQTT)

MQTT is at the heart of the Internet of Things.  It is a lightweight TCP protocol that is quite prevalent in smart appliances, vehicles and, well, “things”.  The Dev team had a pretty clever demo in the Expo Center where they could inject MQTT messages into a a traffic simulation and verify the expected behavior of other vehicles.  For example, send a message from one vehicle indicating it had stalled, blocking a street and verify other vehicles rerouted to avoid the incident.  Smart vehicles and self-driving cars are not just futuristic fiction anymore.  Testing these systems to ensure safety and reliability will require advanced testing techniques and tools.

Topology Discovery

Rational Integration Tester (RIT) can be a very powerful tool in the hands of a tester.  But what if the tester is just handed a complex system and told “test this”?  It is quite common when working with a client on a proof of concept that we end up discovering component relationships that the tester didn’t even know existed.  That “discovery” process can be extremely time consuming.  With Topology Discovery, you let Rational Integration Tester do the work for you.  In discovery mode, you tell RIT to monitor the system to determine which components are accessed by your component under test.  RIT will monitor the interactions and build a visual model for you, including capturing the message schemas and formats.  This can really help get you started much faster when faced with testing a new or unknown application.

Mainframe support enhancements

Ok, I will be the first to admit I am not a mainframe guy.  A lot of the new stuff I have seen is alphabet soup to me.  But for those of you that can make sense of it, RIT now supports executing tests against CICS programs directly using IPIC without the need for the CICS transaction gateway.  There have also been enhancements to System Z synchronization of WAS to create the IMS and CICS models.

Other stuff

  • The stub editor and message editors have seen some really nice UI enhancements making it easier to edit and manage stubs and test messages.
  • Enhanced TIBCO support.
  • Enhanced ability to publish and start stubs through automation in DevOps scenarios.

I also expect to hear about more advances in the near future as well and will pass them along when I do.

 

The Rational Integration Tester HTTP Proxy: Recording HTTPS

This is the third in my series of blog posts on the Rational Integration Tester HTTP proxy.  If you need to catch up, see

I sketched out a plan for this series several weeks ago.  The topic of this third blog was to be Recording HTTPS.  In the meantime, John Chewter (one of the smartest guys I know, by the way), has written two articles on the IBM developerWorks site on Using SSL with Rational Integration Tester:

JC has crammed a bunch of really valuable guidance into these articles.  He reveals the secrets behind a lot of the magic of SSL and HTTPS communications and how Rational Integration Tester works with it.  These two articles are absolute must-reads before you go any further.  Since there is little value in me just repeating what JC already documented, I decided to try hitting this from a little different angle that builds on JC’s articles.  Armed with the knowledge he provides, let’s walk through a brief example applying these concepts to a reasonably realistic scenario.

Let’s say we have a client web application running on Tomcat that consumes a hotel provider web service.  In order to record the secure conversation and to later properly route requests to a stub, we will insert the RIT HTTP Proxy between the client and server. In this example, the client will require authentication from the service but the service will accept requests from anyone and will not require mutual authentication.

Simple Architecture configured for Proxy

Simple Architecture configured for Proxy

The proxy configuration

The Rational Integration Tester proxy configuration is controlled by the registration.xml file in the proxy’s installation directory (on Windows, the default is C:\Program Files\IBM\RIT-Platform\httptcp).  The area of the registration.xml file we want to focus on is the https-proxy section.

Portion of the registration.xml file

Portion of the registration.xml file

The outboundKeyStoreFile is used in mutual authentication schemes where the proxy must authenticate with a server.  Our example is a single-authentication architecture so outboundKeyStoreFile is not relevant in our case.

The keyStoreFile is what the proxy will use when it is acting as a server.  This is the primary role it will play in a typical server authentication scheme.  RIT ships with a preconfigured greenhat.jks keystore which contains a preconfigured key called mykey.  In other words, when the client application requests the proxy to provide its credentials for authentication, the proxy will send it the “mykey” public key.

As John mentioned in his article, you can certainly use this key for testing.  The only thing to consider is that your client app will need to be configured to trust it.  The other option is to use a key that is already trusted by the client – or more precisely, a key that is signed by a certificate authority that the client trusts.  Since John walked you through the latter, I will stick with the greenhat keystore and the mykey key.

Configuring the application client

We know we will need to add the greenhat certificate to the client’s trust store.  This will tell the client application that it can trust the proxy.  but first,  we need to know where the client’s trust store is.  Therefore we will verify and update the configuration of the client application first.

What you need to do will depend on what client technology you are using.  In this example, the client is a web application running in Tomcat.  Tomcat’s configuration settings are generally in the setenv.bat (or setenv.sh) file used to launch it.  We add two JVM arguments to the launch configuration to cause Tomcat to route HTTPS communications through the proxy.  So if the proxy is running on a host at 192.168.100.100, we add the following lines:

JVM arguments to enable proxy

JVM arguments to enable proxy

Note that unlike in the last blog, the prefix of the directives is https, not http.  Also, the default port the proxy uses for HTTPS traffic is 3129, although this can be changed in the registration.xml file.

Within the setenv.sh file, we see existing truststore JVM arguments:

Tomcat client truststore configuration

Tomcat client truststore configuration

These directives tell Tomcat where to find the trust store and how to open it.  The trust store contains certificates for servers that can be trusted or certificates for certificate authorities that the client application can consider reputable enough to vouch for a server.  This is where we will need to add the greenhat certificate which will enable the client to trust the proxy.

Exploring the keystore

System administrators will commonly use tools like the Java keytool program to manipulate keystores.  But RIT has a tool built into the UI that enables you to manage your configuration from within the project.  To use it, open the Physical view of the Architecture School perspective. Under the General toolbar, select Identity Store.  Use the Select browser to locate the wwtravelTrustStore.jks file.  You will be prompted for a keystore password.  This is the password in the setenv.bat file, above.

You will find at least one existing certificate in the trust store: wwtravel.  This is the certificate for the hotels server host which tells the client it can trust the hotel service.  We use the Import Key/Certificate(s) button to import the greenhat.cer file provided with RIT.  This certificate corresponds to the “mykey” public key that the proxy will send when the client attempts to open up a conversation with it.

identity store for wwtraveltruststore.jks

identity store for wwtraveltruststore.jks

Bounce the Tomcat server to get the changes to take effect.

Now you may be looking back at the simple architecture diagram at the beginning of this post and wondering why we need to go through all this work for the connection on the left side but not the connection on the right site.  Don’t we need to add the server’s certificate to the proxy so that the proxy will trust it?  To avoid putting you through all that, the RIT proxy is designed to “trust anybody”.  In other words, when the server sends its public key to the proxy, the proxy will accept it, no questions asked.

While we are in the Physical View, we will create an Identity Store for the greenhat.jks keystore as well.  We will need to reference that when we create the RIT model.  The keystore password is “passphrase”.

Identity Store for greenhat.jks

Identity Store for greenhat.jks

Verify the configuration

With the proxy settings configured on the client, the proxy configured to supply an SSL key when requested and the client’s trust store configured so that it will accept that key, everything should be functional.  The client should be able to make requests of the live service and get responses with the messages being passed through the proxy.  You should run two tests to verify your setup at this time:

  • With the proxy stopped, verify that the client is unable to make a request of the service.  This proves that you have all the plumbing configured correctly.  If the client can get a response from the service with the proxy stopped, it is bypassing the proxy.
  • With the proxy running, verify that the client can make a request and get a response from the service.  This proves that you have all the authentication set up right.

Configuring the RIT model

Next, we will configure the model in Rational Integration Tester for recording.  This is done just like it was for HTTP with one additional step.  On the SSL tab of the Web Server physical resource, you much check “Use SSL”.  This tells RIT tests to invoke the SSL handshake when they connect to the service.

There are two groups of SSL settings on the Web Server SSL tab that affect how RIT operates in an SSL environment.  The Provided Certificates area defines how RIT should behave when it is acting as a Server.  In other words, if you will run stubs on this endpoint, these settings will be used to define which key in which keystore RIT will send to the client when asked to identify itself.  Check Specify Provided Certificate; select the greenhat.jks keystore and the mykey key.

HTTPS Web Server physical resource

HTTPS Web Server physical resource

The Trusted Certificates group defines how RIT should operate when acting as a client.  RIT’s default behavior for tests is again to “trust anybody”, but you can override that behavior by specifying trusted certificates or verifying the certificate chain.

Recording HTTPS

With the keystores and model settings in place, recording HTTPS is really no different than recording HTTP.  Create an event monitor by right-clicking on the element in the Logical View and start recording in the Recording Studio perspective.  Because the proxy is acting as a “man in the middle”, it sees and records the messages in clear text.  You can modify tests and stubs based on those recordings just as you would plain text artifacts.

So working with SSL shouldn’t be scary.  It will require some understanding of the components of SSL (keystores, certificates, keys, etc.) and some understanding of the handshake process, but that should be expected.  If you are performing component-level integration tests, you will need to know how those components interact with each other – at least long enough to configure RIT.  Once the plumbing is in place, RIT makes it easy to work with data in plain text.

The Rational Integration Tester HTTP Proxy: Recording HTTP

This blog post is the second in a series about the Rational Integration Tester HTTP Proxy.  If you are just now joining us, I suggest you drop back and first read

The Rational Integration Tester HTTP Proxy: What is it?

Now that we have covered what the proxy is, what it is used for and the basics of installing it, we will look deeper into using the proxy for its first application – recording.  There probably is no better way to describe how the proxy works than “man in the middle”.  That term strikes fear into most people.  You think of malicious attacks on web sites bent on malfeasance.  In such attacks, a 3rd party finds a way to convince a client application to allow it to listen in as that client talks to a service it uses.  In an attack like that, the goal is to enable the 3rd party to harvest information in those messages.  Although the Rational Integration Tester HTTP proxy uses the same general approach, you are the 3rd party and the purpose of harvesting information in those messages is ultimately to create either tests that replicate the client or stubs that emulate the service.

Yes, there are security implications here.  And that’s exactly why you really need to understand what is going on and how to ensure you are only giving the Rational Integration Tester HTTP Proxy access to your messages.  The next post in this series will dig deeper into security, but we need to get a foundational understanding of how to set up an unsecured system first and then build on that.

Overview

The overall process you will follow consists of the following steps:

  1. Understand the System Under Test (SUT)
  2. Configure the client to route requests through the proxy.
  3. Verify the SUT works correctly with the proxy in place.
  4. Model the endpoint in Rational Integration Tester and verify you can access the service.
  5. Use RIT to record as you send messages to the service by exercising the client.

The Process

Understanding the System Under Test

This probably sounds obvious.  Of course you need to understand the system you are testing.  But the system you want to test is probably a complex mishmash of subsystems and sub-subsystems sending messages through all sorts of protocols and message formats.  What do you need to know about the system and what is just extra information that doesn’t help?  Ultimately any given communication path you want to record boils down to a client sending messages and receiving responses from a service. Let’s consider a simple example where a web client application running in Tomcat consumes a service that provides hotel search and booking:

Simple Communication Architecture

Simple Communication Architecture

The key details you need to know are the hostname of the server hosting the service and the port number on which the service listens.  In the example, the hotel service runs on a fictitious server called myHotelServer.com on port 8089.  Once we start considering authentication and security, there will be other things you will need to know, but we will get to that later.

Configuring the client

Recording messages with the proxy requires that you reconfigure your System Under Test (SUT) a bit.  This might sound like invasive testing.  The thought that you must tweak your system in order to enable testing might rub you the wrong way.  But then again, think about it this way: should your SUT allow just anyone to tap into the wire and record messages?  Rest assured that Rational Integration Tester – or any other tool – can’t somehow sneak into a properly configured and secured system to record data.  You must deliberately configure your system’s client to allow this recording to occur.  This is a necessary thing – and a good thing.

The specific steps required to configure your client will depend on the container your client runs in.  Some specific technologies are addressed in the InfoCenter.  In the case of most Java-based containers, you will be able to add two Java startup switches to the command line that launches the container and then restart the container.  For our example where the client is running in Apache Tomcat, we would add the following switches to the script that launches Tomcat:

-Dhttp.proxyHost=myHotelServer.com –Dhttp.proxyPort=3128

Note that the InfoCenter recommends editing catalina.bat or catalina.sh.  I prefer to keep environment settings like this localized to the setenv.bat or setenv.sh scripts.

Simple Architecture configured for Proxy

Simple Architecture configured for Proxy

As long as your client application correctly uses the standard Java HTTP client libraries, this will cause it to route messages through the proxy.  There are many corner cases where simply setting these two switches will not be sufficient.  This all depends on how the client is implemented.  For now, let’s assume this works for our example.  Troubleshooting these corner cases may be a good topic for a future blog article.

Verifying the System Under Test

Before you go any further, it is very important that you verify that A) you haven’t broken your SUT and B) messages are actually routing through the proxy.  It is easy to make typos when modifying textual startup scripts so double-check your edits.

Start the Rational Integration Tester HTTP proxy.  As we discussed in the previous blog article, you normally want to have the proxy installed as a Windows service and have it start automatically upon boot.  Now is a good time to NOT do that.  Go to the Windows Services and stop the IBM RIT HTTP Proxy service.  Now navigate to your proxy install location (C:\Program Files\IBM\RIT-Platform\httptcp by default) in a command prompt and launch startup.bat.  Many lines of logging information will scroll by.  This may be worth referencing later if you have problems.

If you haven’t done so already, restart your client application.  You will need to restart for the startup directives to take effect.  It is worth checking log files in your client that might indicate whether or not the client picked up your startup directives.

Now try invoking the service from the client.  The first thing to verify is that the client gets the response you would expect and does not get errors.  If your client isn’t able to send the request or get a response back, then you broke something.  One thing to verify is that you started the proxy.  If the proxy isn’t running, no messages will get through.

Secondly, check the command window where you launched the proxy.  You should see a number of new logging messages.  In particular, look for the string “proxy rule found: Proxy on port 3128”.  This indicates that the request made it from the client to the proxy.  Further down in the messages you should see something like “Sending request message to endpoint”.  A little further down you should see “Reading response message from endpoint”.

Proxy messages indicating a successful route

Proxy messages indicating a successful route

Don’t bother proceeding if this isn’t happening for you.  There are a number of possible reasons that messages aren’t getting through.  Firewalls are common blockers.  You will need to correct the situation because nothing beyond this point will work.

Configuring the RIT model

With the proxy in place and verified, we are ready to model the system in Rational Integration Tester in preparation for recording.  I wrote some weeks ago about the model in Rational Integration Tester.  Since this post is about the proxy, I’ll stick to the bare minimum modeling and focus on what we need to get a recording.

At a minimum, you will need a logical HTTP Connection and a physical Web Server in your model.  Create the HTTP Connection in the logical view.  Create a physical Web Server.  On the Config > Settings tab, enter the host and port of the hotels server.

Physical resource for the Web Server

Physical resource for the Web Server

Note – you should NOT be entering any information about the proxy here.  Rational Integration Tester handles all the proxy stuff behind the scenes with the help of Rational Test Control Panel.  All you need to do in the model is describe the SUT.

Also, switch the Recording Mode on the Recording tab to External Proxy Server.  This is not the default.  If you forget to switch the recoding mode to use the proxy, you will get no errors or messages, but you will get no recording.  If it is any comfort, you won’t be the first to forget this.  It happens all the time.

Recording Mode setting on the physical Web Server

Recording Mode setting on the physical Web Server

Recording events

Start up recording in RIT on the HTTP Connection.  You should see a couple more messages appear in the proxy command window such as “Added recording rule…”.  This appears when RIT registers a rule in the proxy which tells the proxy to copy all the messages it sees and send them to RIT where they will appear as events in the Recording Studio perspective.

Proxy message indicating a recording rule

Proxy message indicating a recording rule

In the events view, you should now see the messages that were recorded.  In our example, the client sent a request consisting of an HTTP POST to the getHotels operation on the Hotel Server.  The service replied with a list of hotels in an XML message.

Recorded events in RIT

Recorded events in RIT

These events can then be used to create operations, requirements, tests or stubs.

With the recorder engaged, here is what you really have in place:

Recording Architecture

Recording Architecture

The request comes from the client destined for the service, but routed through the proxy.  The recording rule that has been registered with the proxy causes it to send a copy of the request to RIT at the same time that it forwards on the request to the service.  When the service replies, the proxy once again sends a copy of the reply to RIT while forwarding on the reply to the client.

I have left out some important details such as the role of the Rational Test Control Panel but I wanted to keep things fairly simple for now and focus on the role of the proxy.  Hopefully this article has helped clear up any confusion you may have had around recording basic HTTP with the proxy.  In my next post, I will look at recording HTTPS which adds another dimension to things.

The Rational Integration Tester HTTP Proxy: What is it?

Over the next few weeks, I will be devoting a series of blog postings to the concept of a message “interceptor” and in particular to the Rational Integration Tester HTTP Proxy.  In order to effectively use Rational Integration Tester to create and run integration tests or stubs, it is critical that you understand the role of the HTTP proxy, how to install it, how to configure your environment and how to troubleshoot issues deploying it.  This series of blogs will start with the basic concepts but hopefully even seasoned Rational Integration Tester users will find value as we move along.

The Interceptor Concept

The proxy is really just one specific instance of a general concept called the Interceptor.  In general, the interceptor has two basic functions:

  1. Listen in on conversations between a client and a service to record message traffic that can later by transformed into automated tests or stubs.
  2. Act as a router to cause messages to be redirected from a client to a stub rather than the live service.

The details of how an interceptor works and the roles it fulfills will vary depending on the messaging technology being used.  For example, the interceptor for JDBC traffic is implemented as a JDBC library.  For IBM WebSphere MQ, the interceptor is a set of user exit libraries that are hooked into the MQ Queue Manager.  The collection of all these interceptors is known as the Rational Integration Tester Platform Pack.  I’m sure I will have opportunities in the future to dive deeper into those interceptors as well as others, but for this series, I’ll focus in on the HTTP Proxy used for HTTP, HTTPS and TCP messaging.

What is the HTTP/TCP Proxy?

I will just refer to it as “the proxy” from here on out, but the full name is really the HTTP/TCP Proxy.  As mentioned above, the proxy is the interceptor for HTTP, HTTPS and TCP messages.  Web proxies are pretty much a mainstay of networking these days.  Wikipedia has a pretty decent discussion on proxies (http://en.wikipedia.org/wiki/Proxy_server).  This diagram represents a proxy in its simplest form.

The Proxy Concept

The Rational Integration Tester proxy sits between a service which uses the HTTP, HTTPS or TCP protocols such as a SOAP web service, REST services, etc. and the client that consumes the service.

Proxy Architecture

Proxy Architecture

Once the proxy is in place and running, the client and the service are unaware of its presence for the most part.  I say “for the most part” because there are some security considerations in an HTTPS environment that we will examine in an upcoming blog post.

The first function of the proxy is to listen in on the conversation between the client and the live service so that messages can be recorded.  The proxy doesn’t do this all by itself.  The proxy is really just the workhorse for the Rational Integration Tester workbench.  What basically happens is that an instance of Rational Integration Tester workbench tells the proxy that it should make a copy of the request coming from the client destined for the service.  The proxy sends that copy to the Rational Integration Tester workbench but it also sends the original request – unmodified – to the live service.  So the service thinks the request came from the client and responds.  The proxy makes a copy of the reply, sends the copy to RIT and sends the reply on to the client.

Proxy used for Recording

Proxy used for Recording

The second function of the proxy is to route request messages from the client to the live service or to a stub, depending on what messages your stub is supposed to handle.  Again, this is managed by the Rational Integration Tester workbench inserting “rules” into the proxy.  As a request message comes into the proxy, it evaluates it against the rules that have been registered with the proxy and acts accordingly.  For example, a rule might say that requests to the operation getHotels on myHotelServer:8089 should be routed to a stub instead of the live service.

ProxyStubbing

Proxy used for Stubbing

Those of you with more Rational Integration Tester experience have probably already noticed that proxy rules are not normally set directly by Rational Integration Tester.  To be precise, Rational Integration Tester tells the Rational Test Control Panel which in turn sets the rule in the proxy.  That distinction will become more important later, but for now, let’s leave RTCP out of the picture.  We will dive into the details in a later post.

Installation

As was mentioned earlier, the HTTP/TCP Proxy is part of the IBM Rational Integration Tester Platform Pack.  It is installed using the IBM Installation Manager.

Installing the Proxy

Installing the Proxy

Running as a service

There are two options available when installing on Windows: Install Service and Start on Boot, which are both selected by default.  Install Service installs a Windows Service that will enable you to run the proxy as a background service without a visible command shell.  This is recommended for most all situations.

The Windows Service

The Windows Service

If you deselect the Start on Boot option, the Windows service will be installed but set to a Manual startup type.  In most all situations, I would recommend that you select Start on Boot.  Remember that if a System Under Test client has been configured to use a proxy and that proxy is not running, the SUT client will not be able to communicate to the service.  It is therefore advisable to do what you can to ensure the proxy is running at all times.  By installing the service and having the service start automatically, the proxy will fire up immediately upon reboot, which is generally a good thing.

Running from the command line

It is possible to run the proxy from the command line.  This can actually be advantageous when you are debugging message routing issues since the proxy will scroll logging messages in the command window where it was launched.  The same logging information is captured in log files when run as a service, but scrolling back through the command window is sometimes easier than repeatedly opening log files.

If you are not running the Windows service, open a command prompt, navigate to the proxy installation directory (by default, C:\Program Files\IBM\RIT-Platform\httptcp) and run startup.bat (or startup.sh on Linux).

Network Considerations

Keep in mind that any non-trivial Rational Integration Tester installation will involve components installed across multiple machines.  It is quite possible your proxy is installed on a different machine from your SUT client, the live service or the Rational Integration Tester workbench.  You must have network connections between all of these machines and ports used to communicate between them must be open.  Depending on your company policies, you may need to adjust firewalls to enable such communications.

The best way to figure out what holes you need in the firewalls on the various machines is to trace the flow of messages through a block diagram like those above.  Keep in mind that firewalls can control message flow both out of and into a host.  So let’s look at this example.  The ports necessary to allow the client to communicate with the live service are already open.  We need the following additional paths to be opened to allow the proxy to relay the request and reply back and forth:

  • CLIENT_HOST OUTPUT New request to destination port 3128
  • RITPP_HOST INPUT New request on destination port 3128
  • RITPP_HOST OUTPUT New request to myHotelServer.com destination port 8089
  • RITPP_HOST INPUT response on Existing connection from source port 8089
  • RITPP_HOST OUTPUT response on Existing connection from source port 3128

There may be additional ports that need to be opened to allow communication between the Rational Integration Tester workbench and the proxy as well.

In the next blog post in this series (The Rational Integration Tester HTTP Proxy: Recording HTTP), I look in detail at using the proxy to record HTTP messages.  After that, we will take a look at what additional considerations there are for recording secure communications using HTTPS.  I’ll also blog about using the proxy to route messages to stubs.  That’s my short list for now.  It’s likely that additional topics will come to mind as I work through these so don’t be surprised if that short list gets a little longer.

Working with REST interfaces in Rational Integration Tester

REST (Representational State Transfer) is a simple, stateless architectural style typically running over HTTP which is common among web service implementations.  REST depends on the use of standard methods such as GET, POST, PUT and DELETE. Until recently, there wasn’t an accepted standard for how REST services were defined. There was no service specification that Rational Integration Tester could synchronized to as an external resource. Web Application Description Language (WADL) is a specification recently ratified by the W3C for defining REST services. WADL models the resources provided by a service and the relationships between them.  We will look at WADL in a bit, but it is important to understand how to define Rational Integration Tester REST schemas manually for a few reasons. First of all, you really need to understand what’s happening under the hood to understand how RIT uses the schema later. Secondly, WADL is new enough that many existing REST services do not have WADL specifications. And finally, there are some limitations in Rational Integration Tester’s support for WADL that may mean you need to manually adjust your model after synchronizing.

Creating REST schemas manually

The Login example

Forget about WADL for a moment. Let’s look at what a simple REST implementation might look like as an example. Let’s say the fictitious JKE Bank has a REST system that authenticates users of its online system. The REST URI for authentication looks like this:

http://services.jkebank.net/user/<username>?password=<password>

Yes, I realize passing the password as a query parameter is highly unsecure, but play along with me, please. It illustrates my point well. Julie Brown is an account holder at JKE and she uses her pet cat’s name, Snuggles, as her password. The request to authenticate the user jbrown might look like this:

http://services.jkebank.net/user/jbrown?password=snuggles

If you simply record a request message to this URL with Rational Integration Tester, how should the elements of the URL be interpreted? What constitutes the “operation” of this system? If you want to create tests for different users, you will want to create data-driven tests and stubs. How can you get RIT to substitute values from a test data file for the “jbrown” element of the URL? The answer lies in a REST schema definition in the Schema view of the Architecture School. REST schemas enable you to define which elements of the URL should be parametrized and which ones should be considered fixed values. To make the process of defining the schema easier, RIT provides the ability to import a URL template.

Once again, let’s use the JKE example to illustrate. Let’s say you have defined an HTTP connection in the logical view and associated it to a physical resource representing your web server. You record the request and you see something like this:

Recorded events for login of user jbrown

Recorded events for login of user jbrown

If you select those two events and attempt to save them as unit test with a simple data set, you will see where the problem comes in.

First Attempt at a Data Driven Test

First Attempt at a Data Driven Test

You would want the username to be a column in your data file, but only the password has been identified as a parameter.  So if you go ahead with this test as it is, this test can only be used for this one username – jbrown.  Furthermore, notice that the operation has been identified as jbrown.  In REST terms, jbrown is the unique ID for the resource.  The operation should really be “user” – the element of the URI just before the unique ID.

This illustrates why Rational Integration Tester needs a little coaching in order to properly identify the fixed versus parameterized elements of the URI.  That coaching comes in the form of a REST schema which is created in the Schema Library view of the Architecture School perspective.  The simplest way that I have found to do this is to follow the steps below.

Enhancing the REST schema

Switch to the Schema Library view of the Architecture School perspective and select the REST schema type on the left. Here you see that RIT did create a schema when you attempted to create a test from the recorded events, but it needs some human intelligence applied to really make it useful.

Initial REST Schema for JKE User

Initial REST Schema for JKE User

First of all, the schema name is extracted from the hostname of the physical resource that was recorded.  That’s not entirely useful, so we will change that to “JKE Services”.

Next, let’s edit the template for the URL by selecting it and clicking the edit icon.

Initial template for the JKE User URL

Initial template for the JKE User URL

On the left side, you see the URL path segments.  On the right, you see any query segments.  The password part is pretty easy – it’s a pretty fair bet that anything that appears as a query parameter value is something you will want to parameterize.  That’s why you see the curly braces around the password value.

But since RIT has only seen one URL, it has to make some guesses on what the URL represents.  But the best guess RIT can make is that all elements of the URL are fixed. You can edit the schema a bit to make jbrown a parameter rather than a fixed element. You can also rename that parameter to something more descriptive of what it really represents, which is “username”.

Username parameter

Username parameter

The last thing you probably want to do is to change the template name from jbrown to something more appropriate like “user”.

Saving the test – Redux

With this schema in place, when you save the recorded events as a data-driven test, both the username and password are recognized as parameters and the operation name is identified as user.

Updated Data Driven Test

Updated Data Driven Test

jbrown will be placed into the test data file as a parameter along with the password.

Test Data Preview

Test Data Preview

The retrieve accounts example

Next, let’s look at the REST request that retrieves Julie’s account information:

http://services.jkebank.net/user/jbrown/accounts

If you record the request and its response then attempt to save the events as a data-driven test, you will find again that the username element of the URL will not be identified as a parametrized value.  I will use a slightly different procedure to create this template to demonstrate how you can import a URL template.

Creating the REST template for accounts

Copy the URL of the operation from the Header section of the Events View in Recording studio. In the Schema Library, use the Import URL Template tool and paste your URL into the dialog and click OK.

Initial Accounts Template

Initial Accounts Template

You can see here that RIT has done a rather curious thing.  In the absence of a query parameter, RIT makes the assumption that every element of the URL must be a parameter. That’s not what you wanted either but again, you can edit the schema such that the username is the only parameter.

Updated Accounts Template

Updated Accounts Template

Now what about that WADL…

Now, all that is good to know, but I’m one who likes to avoid extra work if I can. So let’s get back to WADL…

As I mentioned earlier, WADL is relatively new and I have found that, although there is a W3C Specification for WADL, people interpret and apply the spec in various ways.  Let’s take a look at ways we could represent the JKE interfaces in WADL.

Here’s how I think these services would be represented in WADL:

JKE Banking Fully Structured WADL

JKE Banking Fully Structured WADL

I mentioned some limitations in Rational Integration Tester’s WADL support at the beginning of this post.  One of the limitations is that RIT doesn’t support parameters at the resource level yet – that’s something we expect in a future release.  So I’ve reorganized the resource hierarchy a little in the following iteration of the WADL file:

Restructured JKE WADL

Restructured JKE WADL

Now go to the Synchronization view of the Architecture School perspective in RIT and add a WADL reference to the external WADL file.  This can be done with the Web > WADL toolbar action or the Add a new item to the view toolbar action.  When you synchronize, you will see the following elements.

Synchronized WADL

Synchronized WADL

Let’s browse the model artifacts to see what was created by going to the Schema Library.

Schema Library for WADL

Schema Library for WADL

RIT left the schema untitled and uses a default name for the schema template of “url name”.  This is something you should probably change to be more human friendly so name the schema JKE, the first template user and the second accounts.

Updated Schema Library for WADL

Updated Schema Library for WADL

In the Logical View, the endpoint and two operations were created.

Synchronized elements in the Logical View

Synchronized elements in the Logical View

That first operation name looks odd.  This is another limitation of the RIT support today.  If the last element of the resource URL is a parameter, RIT doesn’t really know what to do about a name for the operation.  You can rename the operation to something more appropriate like user.

Now open the user operation properties.

User operation properties

User operation properties

Several attributes such as the operation pattern and method were imported as well as the request schema.

One more limitation – My WADL had specified a mediaType of application/json but the schema for the Reply has been set as Text.  This should be changed by browsing to JSON > Object.

Summary

Rational Integration Tester has some pretty powerful support for REST testing and virtualization.  The nature of the beast of REST typically means you will need to do some coaching in the schema library to get RIT to understand the meaning of your application resources.  RIT 8.5.1 has introduced support for WADL file synchronization.  Although this support is evolving, there is a lot of value in using WADL to jump-start your model creation.