Advanced Apache Camel Parallel Processing Tutorial

In the previous advanced Apache Camel tutorial we looked at sequential multicasting .  In this tutorial, I will discuss multitasking with parallel processing. To multitask, you need to define an executor service first. This can be done via applicationContext.xml file as shown below.

  
        <bean id="executor" class="java.util.concurrent.Executors"
  factory-method="newFixedThreadPool">
  <constructor-arg index="0" value="16" />
 </bean>

Then in the adv_file_route.xml file reference the "executor" along with other attributes as shown below

   
        <route id="advFileConversion">
   <from
    uri="file://{{adv.in.dir}}?include={{adv.in.file.pattern}}&delay={{poll.delay}}&initialDelay={{initial.delay}}&move=archive/" />

   <multicast stopOnException="true" parallelProcessing="true" executorServiceRef="executor">
    <to uri="direct:email" />
    <to uri="direct:transfer" />
   </multicast>
  </route>

When using File components to consume files, you can define your own thread pool as shown below. This option allows you to share a thread pool among multiple file consumers.

 
<camel:threadPool id="scheduledExecutorService" 
  poolSize="${scheduledExecutorService.pool.size}" 
  maxPoolSize="${scheduledExecutorService.pool.size.max}" 
  scheduled="true" 
  threadName="scheduledExecutorService" />

 
<route id="advFileConversion">
  <from uri="file://{{adv.in.dir}}?include={{adv.in.file.pattern}}&delay={{poll.delay}}&initialDelay={{initial.delay}}
        &move=archive/&scheduledExecutorService=#scheduledExecutorService" />

  <multicast stopOnException="true" parallelProcessing="true" executorServiceRef="executor">
   <to uri="direct:email" />
   <to uri="direct:transfer" />
  </multicast>
</route>

As shown above, the file consumer uses the threadpool with the following attribute name and value scheduledExecutorService=#scheduledExecutorService".

You can send messages to a number of Camel Components to achieve parallel processing and load balancing with components such as

1. SEDA for in-JVM load balancing across a thread pool. The SEDA component provides asynchronous SEDA behavior, so that messages are exchanged on a BlockingQueue and consumers are invoked in a separate thread from the producer. Note that queues are only visible within a single CamelContext. If you want to communicate across CamelContext instances (for example, communicating between Web applications), see the VM component.
2. The VM component provides asynchronous SEDA behavior, but differs from the SEDA component in that VM supports communication across CamelContext instances - so you can use this mechanism to communicate across web applications
3. JMS or ActiveMQ for distributed load balancing and parallel processing. The JMS component allows messages to be sent to (or consumed from) a JMS Queue or Topic. The implementation of the JMS Component uses Spring's JMS support for declarative transactions, using Spring's JmsTemplate for sending and a MessageListenerContainer for consuming.
4. The Splitter from the EIP (Enterprise Integration Patterns) patterns allows you split a message into a number of pieces and process them individually.  The split component takes the attribute "executorServiceRef". This attribute refers to a custom Thread Pool to be used for parallel processing, and notice that if you set this option, then parallel processing is automatically implied, and you do not have to enable that option as well. 

Note: As soon as you send multiple messages to different threads or processes you will end up with an unknown ordering across the entire message stream as each thread is going to process messages concurrently. For many use cases the order of messages is not too important. However for some applications this can be crucial, and you need to preserve the order.


Q. What is a SEDA?
A. SEDA stands for Staged Event Driven Architecture. This architecture decomposes a complex, event-driven application into a set of stages connected by queues. The most fundamental aspect of SEDA architecture is the programming model that supports stage-level backpressure and load management. Stage is analogous to "Event", to simplify the idea, think SEDA as a series of events sending messages between them.

Q. What does Apache Camel "event" component does?
A. The event component provides access to the Spring ApplicationEvent objects. This allows you to publish ApplicationEvent objects to a Spring ApplicationContext or to consume them. You can then use Enterprise Integration Patterns to process them such as Message Filter.