Using Cucumber to Integrate Distributed Systems and Test Messaging

Cucumber is a fantastic tool that can be used in many ways to accomplish different goals. One interesting use of Cucumber is to facilitate integration and communication between different systems. At my last job we had several distributed systems that communicated via a messaging broker. It was very important that the messages sent between the different systems be kept in sync and handled appropriately. For example, System A needed to know the exact message format and queue that System B was going to be using, and vice versa. This type of integration between systems is very error prone and when something goes awry the problems can be very hard to track down. In order to ensure both systems were on the same page we used the exact same Cucumber feature in both systems but had the step definitions verify different things on the respective systems. In this post I’ll walk through a quick example illustrating the tools and techniques we used to do this.

Developers often struggle when writing features that entail systems or are purely technical in nature. I highly recommend Liz Keogh’s post about this subject as it greatly helped me in learning how to phrase such technical stories. With that said, lets look at the example feature:

  In order to enable accurate reporting
  As an analyst I want System A
  to keep it's widget data in sync with System B

  Scenario: widget creation
    Given I am logged in to System A
    When I create a widget
    Then the same widget should exist in System B

As you can see there is not much to the actual feature itself. In real situations more context (more Givens) is usually needed to set up needed data, but outside of that the above is a good representation of what we found to be a good scenario format to stick to. The key is to be simple and state declaratively the behaviour of the two systems. This allows you keep the details of the implementations in the step definitions and not clutter the overall intent of the scenario with technical noise.

I said above that both projects use the exact same plain-text feature, but implement the steps differently. While this is true, we actually ended up having the exact same step definition files as well. Blocks were then used to distinguish the code for either system within each step. Continuing the example from above the pertinent step definitions would be:

When "I create a widget" do
  @expected_message = {"name" => "Foo", "color" => "Red"} 

  system_a do
    create_widget(@expected_message)  # this call should then make system_a publish a message
  end

  system_b do
    # when system_b runs the feature it needs to simulate a message from system_a
    publish_message(@expected_message, :to => :widget_queue)
  end

end

Then "the same widget should exist in System B" do
  system_a do
   # to ensure system_a is working correctly we need to make sure that the message was published
   @expected_message.should be_published_to(:widget_queue)
  end

  system_b do
    consume_once_with WidgetConsumer # have the consumer process the expected message from above
    Widget.count(:conditions => @expected_message).should == 1
  end

end

end

We gravitated to this solution because by keeping the files the same it was easy to keep the two systems in sync. If one system changed the expected message or queue the step definitions would be updated and then both systems would make the feature pass again.

To be clear, these step definitions do not run both systems simultaneously. Each system runs the feature independent of the other. That is why System B in our example simulates what System A is publishing. At this point we are trusting that the messaging system will do it’s job and deliver the message. If the systems are publishing and subscribing to the correct queues with an agreed upon message format then the messaging system should take care of the rest. We did investigate running both systems from the same feature but in our situation the hassle and cost was not justified by the little value we saw that adding.

In order to accommodate this the following helpers were needed:

System A:

module MessagingHelpers
  def system_a
    yield
  end

  def system_b
    #no-op
  end

end

World(MessagingHelpers)

For System B the opposite was needed:

module MessagingHelpers
  def system_a
    # no-op
  end

  def system_b
    yield
  end

end

World(MessagingHelpers)

Of course you will need the actual messaging helpers too…

The above step definitions have some nice messaging helpers (i.e. publish_message, should be_published_to, etc…). These are not just imaginary helpers for use in the example, but are helpers in a library Chris Wyckoff and I recently released called Rosetta Queue. Rosetta Queue is named such because it has an adapter layer built-in that allows you to easily swap out which messaging system you are using. For example, you could start out by using the stomp adapter with ActiveMQ but then later switch to use the AMQP adapter for use with RabbitMQ. We used a real messaging system in our Cucumber features but for our RSpec code examples (unit tests) we disconnected the messaging system and used null and fake adapters provided by Rosetta Queue. You can read more about how to use Rosetta Queue on github.

As a disclaimer, I should say that Rosetta Queue is still relatively immature. It is being used in production systems, but we aren’t happy with the API for publishing and receiving messages. We will most likely be moving to an API that looks like:

queue(:widgets) << "some message"
message = queue(:widgets).pop

The fake adapter is also really a dummy adapter that I want to replace with an actual in-memory fake adapter. But overall, it is a nice library that aids in testing asynchronous messaging on the acceptance and unit level.

The pattern I’ve outlined above could be used with any messaging library, not just Rosetta Queue. We found that using Cucumber in this fashion greatly eased the work required to integrate our systems and reduced our error rate. By hitting the real messaging systems in our features we gained confidence in our system and helped prevent against regressions. This also allowed us to disconnect our messaging systems from our unit tests so we could test our objects in isolation and keep them lightning fast. (Yep, our unit tests were actual unit tests!)