Hello Beer Camel Quality Control

In our previous blog post we saw our Camels smuggling along their craft beer contraband to our thirsty customers. We can expect them to expand our craft business quite rapidly in the near future and open up new black and white markets (hopefully this will keep our shareholders happy and quiet for the time being!). For all this expansion to succeed however, we need to get our quality control in order pronto! The last thing we want is for dromedaries disguised as camels to deliver imitation crafts to our customers and thereby compromise our highly profitable trade routes. So, high time we put some unit testing in place and make our implementation a little more flexible and maintainable.

In this blog post we’ll unit test our Spring REST controller and our Camel route. We also get rid of those hardcoded endpoints and replace them with proper environment-specific properties. So buckle up, grab a beer and let’s get started!

Oh and as always, final code can be viewed online.

Unit testing the controller

Though technically this has nothing to do with Camel, it’s good practice to unit test all important classes, so let’s first tackle and unit test our Spring Boot REST controller.

We only need the basic Spring Boot Starter Test dependency for this guy:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

Let’s test the most interesting part of the controller, i.e. the saveOrder method.

@RunWith(SpringRunner.class)
@WebMvcTest(OrderController.class)
public class OrderControllerTest {

    @Autowired
    private MockMvc mockMvc;

    @MockBean
    private OrderRepository orderRepositoryMock;

    @Test
    public void saveOrder() throws Exception {
        OrderItem orderItem1 = new OrderItemBuilder().setInventoryItemId(1L).setQuantity(100L).build();
        OrderItem orderItem2 = new OrderItemBuilder().setInventoryItemId(2L).setQuantity(50L).build();
        Order order = new OrderBuilder().setCustomerId(1L).addOrderItems(orderItem1, orderItem2).build();

        OrderItem addedItem1 = new OrderItemBuilder().setId(2L).setInventoryItemId(1L).setQuantity(100L).build();
        OrderItem addedItem2 = new OrderItemBuilder().setId(3L).setInventoryItemId(2L).setQuantity(50L).build();
        Order added = new OrderBuilder().setId(1L).setCustomerId(1L).addOrderItems(addedItem1, addedItem2).build();

        when(orderRepositoryMock.save(any(Order.class))).thenReturn(added);

        mockMvc.perform(post("/hello-camel/1.0/order")
            .contentType(TestUtil.APPLICATION_JSON_UTF8)
            .content(TestUtil.convertObjectToJsonBytes(order)))
            .andExpect(status().isOk())
            .andExpect(content().contentType(TestUtil.APPLICATION_JSON_UTF8))
            .andExpect(jsonPath("$.id", is(1)))
            .andExpect(jsonPath("$.customerId", is(1)))
            .andExpect(jsonPath("$.orderItems[0].id", is(2)))
            .andExpect(jsonPath("$.orderItems[0].inventoryItemId", is(1)))
            .andExpect(jsonPath("$.orderItems[0].quantity", is(100)))
            .andExpect(jsonPath("$.orderItems[1].id", is(3)))
            .andExpect(jsonPath("$.orderItems[1].inventoryItemId", is(2)))
            .andExpect(jsonPath("$.orderItems[1].quantity", is(50)));

        ArgumentCaptor<Order> orderCaptor = ArgumentCaptor.forClass(Order.class);
        verify(orderRepositoryMock, times(1)).save(orderCaptor.capture());
        verifyNoMoreInteractions(orderRepositoryMock);

        Order orderArgument = orderCaptor.getValue();
        assertNull(orderArgument.getId());
        assertThat(orderArgument.getCustomerId(), is(1L));
        assertEquals(orderArgument.getOrderItems().size(), 2);
    }
}

Hopefully most of this code speaks for itself. Here are some pointers:

  • The WebMvcTest(OrderController.class) annotation ensures that you can test the OrderController in isolation. With this guy you can autowire a MockMvc instance that basically has all you need to unit test a controller;
  • The controller has a dependency on the OrderRepository, which we will mock in this unit test using the @MockBean annotation;
  • We first use some helper builder classes to fluently build our test Order instances;
  • Next we configure our mock repository to return a full fledged Order object when the save method is called with an Order argument;
  • Now we can actually POST an Order object to our controller and test the JSON being returned;
  • Next check is whether the mock repository was called and ensure that is was called only once;
  • Finally we check the Order POJO that was sent to our mock repository.

Running the test will show us we build a high quality controller here. There’s also a unit test available for the GET method. You can view it on GitHub. The GET method is a lot easier to unit test, so let’s skip it to keep this blog post from getting too verbose.

Testing the Camel route

Now for the most interesting part. We want to test the Camel route we built in our previous blog post. Let’s first revisit it again:

from("ftp://localhost/hello-beer?username=anonymous&move=.done&moveFailed=.error")
    .log("${body}")
    .unmarshal().jacksonxml(Order.class)
    .marshal(jacksonDataFormat)
    .log("${body}")
    .setHeader(Exchange.HTTP_METHOD, constant("POST"))
    .to("http://localhost:8080/hello-camel/1.0/order");

There’s a lot going on in this route. Ideally I would like to perform two tests:

  • One to check if the XML consumed from the ftp endpoint is being properly unmarshalled to an Order POJO;
  • One to check the quality of the subsequent marshalling of said POJO to JSON and also to check if it’s being sent to our REST controller.

So let’s split our route into two routes to reflect this:

from("ftp://localhost/hello-beer?username=anonymous&move=.done&moveFailed=.error")
    .routeId("ftp-to-order")
    .log("${body}")
    .unmarshal().jacksonxml(Order.class)
    .to("direct:new-order").id("new-order");

from("direct:new-order")
    .routeId("order-to-order-controller")
    .marshal(jacksonDataFormat)
    .log("${body}")
    .setHeader(Exchange.HTTP_METHOD, constant("POST"))
    .to("http://localhost:8080/hello-camel/1.0/order").id("new-order-controller");

Note that we added ids to our routes as well as our producer endpoints. You’ll see later on – when we’re gonna replace the producer endpoints with mock endpoints – why we need these. Also note that we’ve set up direct endpoints in the middle of our original route. This will allow us to split the route in two.

Testing camel routes requires one additional dependency:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-test-spring</artifactId>
    <version>${camel.version}</version>
    <scope>test</scope>
</dependency>

Alright now let’s get straight down to business and unit test those routes:

@RunWith(CamelSpringBootRunner.class)
@SpringBootTest
public class FtpOrderToOrderControllerTest {

    private static boolean adviced = false;
    @Autowired
    private CamelContext camelContext;
    @EndpointInject(uri = "direct:input")
    private ProducerTemplate ftpEndpoint;
    @EndpointInject(uri = "direct:new-order")
    private ProducerTemplate orderEndpoint;
    @EndpointInject(uri = "mock:new-order")
    private MockEndpoint mockNewOrder;
    @EndpointInject(uri = "mock:new-order-controller")
    private MockEndpoint mockNewOrderController;

    @Before
    public void setUp() throws Exception {
        if (!adviced) {
            camelContext.getRouteDefinition("ftp-to-order")
                .adviceWith(camelContext, new AdviceWithRouteBuilder() {
                    @Override
                    public void configure() {
                        replaceFromWith(ftpEndpoint.getDefaultEndpoint());
                        weaveById("new-order").replace().to(mockNewOrder.getEndpointUri());
                    }
                });

            camelContext.getRouteDefinition("order-to-order-controller")
                .adviceWith(camelContext, new AdviceWithRouteBuilder() {
                    @Override
                    public void configure() {
                         weaveById("new-order-controller").replace().to(mockNewOrderController.getEndpointUri());
                    }
                });

            adviced = true;
        }
    }

    @Test
    public void ftpToOrder() throws Exception {
        String requestPayload = TestUtil.inputStreamToString(getClass().getResourceAsStream("/data/inbox/newOrder.xml"));
        ftpEndpoint.sendBody(requestPayload);

        Order order = mockNewOrder.getExchanges().get(0).getIn().getBody(Order.class);
        assertNull(order.getId());
        assertThat(order.getCustomerId(), is(1L));
        assertNull(order.getOrderItems().get(0).getId());
        assertThat(order.getOrderItems().get(0).getInventoryItemId(), is(1L));
        assertThat(order.getOrderItems().get(0).getQuantity(), is(100L));
        assertNull(order.getOrderItems().get(1).getId());
        assertThat(order.getCustomerId(), is(1L));
        assertThat(order.getOrderItems().get(1).getInventoryItemId(), is(2L));
        assertThat(order.getOrderItems().get(1).getQuantity(), is(50L));
    }

    @Test
    public void orderToController() {
        OrderItem orderItem1 = new OrderItemBuilder().setInventoryItemId(1L).setQuantity(100L).build();
        OrderItem orderItem2 = new OrderItemBuilder().setInventoryItemId(2L).setQuantity(50L).build();
        Order order = new OrderBuilder().setCustomerId(1L).addOrderItems(orderItem1, orderItem2).build();
        orderEndpoint.sendBody(order);

        String jsonOrder = mockNewOrderController.getExchanges().get(0).getIn().getBody(String.class);
        assertThat(jsonOrder, hasNoJsonPath("$.id"));
        assertThat(jsonOrder, hasJsonPath("$.customerId", is(1)));
        assertThat(jsonOrder, hasNoJsonPath("$.orderItems[0].id"));
        assertThat(jsonOrder, hasJsonPath("$.orderItems[0].inventoryItemId", is(1)));
        assertThat(jsonOrder, hasJsonPath("$.orderItems[0].quantity", is(100)));
        assertThat(jsonOrder, hasNoJsonPath("$.orderItems[1].id"));
        assertThat(jsonOrder, hasJsonPath("$.orderItems[1].inventoryItemId", is(2)));
        assertThat(jsonOrder, hasJsonPath("$.orderItems[1].quantity", is(50)));
        assertThat(jsonOrder, hasNoJsonPath("$.orderItems[1].id"));
    }
}

Again a few pointers to the code above:

  • We’re using the recommended CamelSpringBootRunner here;
  • We autowire an instance of the CamelContext. This context is needed in order to alter the route later on;
  • Next we inject the Consumer and Producer endpoints we’re gonna use in our unit tests;
  • The Setup is the most important part of the puzzle. It is here we replace our endpoints with mocks (and our ftp consumer endpoint with a direct endpoint). It is also here we will use the ids we placed in our routes. They let us point to the endpoints (and the routes they’re in) we wish to replace;
  • Ideally we would have annotated this setUp code with the @BeforeClass annotation to let it run only once. Unfortunately that guy can only be placed on a static method. And static methods don’t play well with our autowired camelContext instance variable. So we use a static boolean to run this code only once (you can’t run it twice because the second time it’ll try to replace stuff that isn’t there anymore);
  • In the ftpToOrder unit test we shove an Order xml into the first route (using the direct endpoint) and check our mockNewOrder endpoint to see if a proper Order POJO has arrived there;
  • In the orderToController unit test we shove an Order POJO in the second route (again using a direct endpoint) and check our mockNewOrderController endpoint to see if a proper Order JSON String has arrived there.

Please note that the json assertion code in the OrderToController Test has a dependency on the json-path-assert library:

<dependency>
    <groupId>com.jayway.jsonpath</groupId>
    <artifactId>json-path-assert</artifactId>
    <version>2.4.0</version>
    <scope>test</scope>
</dependency>

This library is not really necessary. As an alternative you could write expressions like:

assertThat(JsonPath.read(jsonOrder,"$.customerId"), is("1"));

I think the json-path-assert notation is a bit more readable, but that’s just a matter of taste, I guess.

You can run the tests now (mvn clean test) and you will see that all tests are passing.

Externalizing properties

Alright we’re almost there. Only one last set of changes left to make the route a bit more flexible. Let’s introduce Camel properties to replace those hardcoded URIs in the endpoints. Camel and Spring Boot play along quite nicely here and Camel properties work out-of-the-box without further configuration.

So let’s introduce a property file (application-dev.properties) for the development environment and put those two endpoint URIs in it:

endpoint.order.ftp = ftp://localhost/hello-beer?username=anonymous&move=.done&moveFailed=.error
endpoint.order.http = http://localhost:8080/hello-camel/1.0/order

Add one line to the application.properties file to set development as the default Spring profile.

spring.profiles.active=dev

And here’s the final route after putting those endpoint properties in place:

from("{{endpoint.order.ftp}}")
    .routeId("ftp-to-order")
    .log("${body}")
    .unmarshal().jacksonxml(Order.class)
    .to("direct:new-order").id("new-order");

from("direct:new-order")
    .routeId("order-to-order-controller")
    .marshal(jacksonDataFormat)
    .log("${body}")
    .setHeader(Exchange.HTTP_METHOD, constant("POST"))
    .to("{{endpoint.order.http}}").id("new-order-controller");

And that’s it. You can run the application again to see that everything works like before.

Summary

This blog post was all about quality. We showed you how to setup testing in a Spring Boot Camel application and we built a couple of unit tests, one to test our Spring Boot REST controller and one to test our Camel route. As a small bonus we also externalized the endpoint URIs in our Camel route with the help of Camel properties.

Now all that’s left is to grab a beer and think about our next blog post.

References

Advertisements

HelloBeer’s first Camel ride

HelloBeerTM recently got some complaints from the Alcoholics Anonymous community. As it turns out, it’s very difficult to order a fine collection of craft beers online without ones wife finding out about it. Browser histories were scanned and some particularly resourceful spouses even installed HTTP sniffers to confront their husbands with their drinking problem. So in order to keep on top of the beer selling game, HelloBeer needs an obscure backdoor where orders can be placed lest it risks losing an important part of its clientele.

One of HelloBeer’s founding fathers has an old server residing in the attic of his spacious condo. He suggested to use that guy to serve as an old school FTPS server where customers can upload their orders to without their wives finding out about it.

In this blogpost we’re gonna build the integration between an FTPS server and our OrderService REST API (implemented in Spring Boot). To build the integration we’ll be relying on Apache Camel. It’s a great way for embedding Enterprise Integration Patterns in a Java based application, it’s lightweight and it’s very easy to use. Camel also plays nicely with Spring Boot as this blogpost will show.

To keep our non-hipster customers on board (and to make this blogpost a little more interesting), the order files placed on the FTP server, will be in plain old XML and hence have to be transformed to JSON. Now that we have a plan, let’s get to work!

Oh and as always, the finished code has been published on GitHub here.

Installing FTP

I’m gonna build the whole contraption on my Ubuntu-based laptop and I’m gonna use vsftpd to acts as an FTPS server. As a first prototype I’m gonna make the setup as simple as possible and allow anonymous users to connect and do everything they shouldn’t be able to do in any serious production environment.

These are the settings I had to tweak in the vsftpd.conf file after default installation:

# Enable any form of FTP write command.
write_enable=YES
# Allow anonymous FTP? (Disabled by default).
anonymous_enable=YES
# Allow the anonymous FTP user to upload files.
anon_upload_enable=YES
# Files PUT by anonymous users will be GETable
anon_umask=022
# Allow the anonymous FTP user to move files
anon_other_write_enable=YES

Also make sure the permissions on the directory where the orders will be PUT are non-restrictive enough:

Contents of /srv directory:

Contents of /srv/ftp directory:

Contents of /srv/ftp/hello-beer directory:

The .done and .error directories are where the files will be moved to after processing.

Spring Booting the OrderService

The OrderService implementation is just plain old Spring Boot. For a good tutorial, check one of my previous blog posts here. The REST controller exposes a GET method for retrieving a list of orders and a POST method for adding a new order:

@RestController
@RequestMapping("/hello-camel/1.0")
public class OrderController {

    private final OrderRepository orderRepository;

    @Autowired
    public OrderController(OrderRepository orderRepository) {
        this.orderRepository = orderRepository;
    }

    @RequestMapping(value = "/order", method = RequestMethod.POST, produces = "application/json")
    public Order saveOrder(@RequestBody Order order) {
        return orderRepository.save(order);
    }

    @RequestMapping(value = "/orders", method = RequestMethod.GET, produces = "application/json")
    public List<Order> getAllOrders() {
        return orderRepository.findAll();
    }
}

Most of the heavy lifting is done in the domain classes. I wanted the Order to be one coherent entity including its Order Items, so I’m using a bidirectional OneToMany relationship here. To get this guy to play nicely along with the REST controller and the generated Swagger APIs by springfox-swagger2 plugin I had to annotate the living daylights out of the entities. I consulted a lot of tutorials to finally get the configuration right. Please check the references section for some background material. These are the finalized classes that worked for me (please note that I’ve omitted the getters and the setters for brevity):

The Order class:

@Entity
@Table(name = "hb_order")
public class Order {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @ApiModelProperty(readOnly = true)
    @JsonProperty(access = JsonProperty.Access.READ_ONLY)
    private Long id;

    @NotNull
    private Long customerId;

    @OneToMany(
        mappedBy = "order",
        cascade = CascadeType.ALL,
        orphanRemoval = true)
    @JsonManagedReference
    private List<OrderItem> orderItems;
}

The ApiModelProperty is used by the generated Swagger definitions and takes care that the id field only pops up in the response messages of the GET and POST methods, not in the POST request message (since the id is generated). The JsonProperty takes care that no id fields sent to the API aren’t unmarshalled from the JSON message to the entity POJO instance. In the OneToMany annotation the mappedBy attribute is crucial for the bidirectional setup to work properly (again: check the references!). The JsonManagedReference annotation is needed to avoid circular reference errors. It goes hand in hand with the JsonBackReference annotation on the Order Item (stay tuned!).

The OrderItem class:

@Entity
@Table(name = "hb_order_item")
public class OrderItem {
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    @JsonProperty(access = JsonProperty.Access.READ_ONLY)
    @ApiModelProperty(readOnly = true)
    private Long id;

    @ManyToOne(fetch = FetchType.LAZY, cascade = CascadeType.ALL)
    @JoinColumn(name = "order_id")
    @JsonBackReference
    private Order order;

    @NotNull
    private Long inventoryItemId;

    @NotNull
    private Long quantity;
}

Again here the id field is made read-only for the API and for the Swagger definition. The ManyToOne and JoinColumn annotations are key to properly implement the bidirectional OneToMany relationship between the Order and OrderItem. And equally key is the JsonBackReference annotation on the Order field. Without this guy (and its corresponding JsonManagedReference annotation on the Order.orderItems field) you get errors when trying to POST a new Order (one last time: check the references!).

The rest of the code is available on the aforementioned GitHub location. If you give it a spin, you can check out the API on the Swagger page (http://localhost:8080/swagger-ui.html) and test it a bit. You should be able to POST and GET orders to and from the in-memory database.

Camelling out the integration

Now that we have a working OrderService running, let’s see if we can build a flow from the FTP server to the OrderService using Camel.

First step is adding the necessary dependencies to our pom.xml:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-spring-boot-starter</artifactId>
    <version>${camel.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-core-starter</artifactId>
    <version>${camel.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-ftp-starter</artifactId>
    <version>${camel.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-jacksonxml-starter</artifactId>
    <version>${camel.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-jackson-starter</artifactId>
    <version>${camel.version}</version>
</dependency>
<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-http-starter</artifactId>
    <version>${camel.version}</version>
</dependency>

The camel-spring-boot-starter is needed when you’re gonna work with camel in a Spring Boot application. For the other dependencies. It’s not that different from building a non-Spring Boot Camel application. For every camel component you need, you have to add the necessary dependency. The big difference is that you use the corresponding dependency with the -starter suffix.

Alright so what are all these dependencies needed for:

  • camel-core-starter: used for core functionality, you basically always need this guy;
  • camel-ftp-starter: contains the ftp component;
  • camel-jacksonxml-starter: used to unmarshal the XML in the Order to our Order POJO;
  • camel-jackson-starter: used to marshal the Order POJO to JSON;
  • camel-http-starter: used to issue a POST request to the OrderController REST API.

Believe it or not, now the only thing we have to take care of is to build our small Camel integration component utilizing all these dependencies:

@Component
public class FtpOrderToOrderController extends RouteBuilder {

    @Override
    public void configure() throws Exception {

    JacksonDataFormat jacksonDataFormat = new JacksonDataFormat();
    jacksonDataFormat.setInclude("NON_NULL");
    jacksonDataFormat.setPrettyPrint(true);

    from("ftp://localhost/hello-beer?username=anonymous&move=.done&moveFailed=.error")
        .log("${body}")
        .unmarshal().jacksonxml(Order.class)
        .marshal(jacksonDataFormat)
        .log("${body}")
        .setHeader(Exchange.HTTP_METHOD, constant("POST"))
        .to("http://localhost:8080/hello-camel/1.0/order");
    }
}

Some pointers to the above code:

  • The .done and .error directories are where successfully and unsuccessfully processed Orders end up. If you don’t take care of moving the orders, they will be processed again and again;
  • The NON_NULL clause added to the JacksonDataFormat, filters out the id fields when marshalling the POJO to JSON;
  • The XML and JSON will be logged so you can verify that the transformations are working as expected.

The rest of the route imho is self-explanatory.

Oh and one more thing. I like my XML elements to be capitalized. So our Order XML element contains a CustomerId element, not a customerId element. This only works is you give the jacksonxml mapper some hint in the form of annotations on the Order (and OrderItem) POJO (note that I’ve omitted the other annotations in the code below):

public class Order {
    
    private Long id;

    @JacksonXmlProperty(localName="CustomerId")
    private Long customerId;

    @JacksonXmlProperty(localName="OrderItems")
    private List<OrderItem> orderItems;
}

The same applies for the OrderItem, see Github for the definitive code.

Testing the beasty

Now as always the proof is in the tasting of the beer. Time to fire up the Spring Boot application and place our first Order on the FTP server.

I’ve created a small newOrder.xml file and put it in a local directory. It looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<Order>
    <CustomerId>1</CustomerId>
    <OrderItems>
        <OrderItem>
            <InventoryItemId>1</InventoryItemId>
            <Quantity>100</Quantity>
        </OrderItem>
        <OrderItem>
            <InventoryItemId>2</InventoryItemId>
            <Quantity>50</Quantity>
        </OrderItem>
    </OrderItems>
</Order>

Now when i connect to my local FTP server, change to the hello-beer directory and issue a PUT of that local newOrder.xml file, I can see the logging of the Camel component appearing in my IntelliJ IDE:

As you can see the first log statement has been executed and the XML content of the file is displayed. The second log statement has been executed as well and nicely displays the message body after it has been transformed into JSON.

You will also notice that the file has been moved to the .done directory. You can also do this test with an invalid xml file and notice that it ends up in the .error directory.

One last test needed. Let’s issue a GET against the hello-camel/1.0/orders endpoint with the help of the Swagger UI. And lo and behold the response:

Great, so our newOrder.xml that arrived on our FTP server has been nicely stored in our database. Our first prototype is working. Our AA customers will be delighted to hear this.

Summary

In this blog post we’ve seen how easy it is to integrate with Apache Camel in a Spring Boot application. We coded an FTP-to-REST integration flow in no time and even put some XML-to-JSON transformation into the mix. I like the fact that we can keep the integration code nice and clean and separated from the rest of the application.

Testing is still a bit of trial and error though. Let’s see if we can put some proper unit tests in place in the next blog post. For now: happy drinking!

References