Understanding service and socket timeouts in Java enterprise applications?

Q. Why is it important to set proper timeout values in your applications?
A. Security and performance.

Security reason: it is often necessary to control how long a Web Service client or other valuable clients invoking valuable resources like database connections, transaction management, etc waits for a response after sending a request. Not setting proper timeouts can be a potential candidate for DoS attacks, especially via web service calls. This is why the application servers provide various timeout configurations to prevent DoS attacks. For example, in application servers

• The RequestReadTimeout allows to limit the time a client may take to send the request.
• The TimeOut directive should be lowered on sites that are subject to DoS attacks. Setting this to as low as a few seconds may be appropriate. 
• The KeepAliveTimeout directive may be also lowered on sites that are subject to DoS attacks. Some sites even turn off the keepalives completely, which has of course other drawbacks on performance.

Performance reason: A common cause of bad Java EE performance for highly distributed systems is due to not properly implementing timeouts causing client threads to get stuck, which can lead to full domino effect. So, it is imperative to test for non-happy path and implement proper service, transaction, and socket timeouts.

Q. Can you give some examples as to how you can set service timeouts?

A. 

1. If you are using Apache HttpClient via Spring, you can set it via the application context file. The timeouts you have set are associated with the time it takes to establish a (socket) connection and the maximum time between establishing a connection and receiving data from the connection. 

<http:conduit name="*.http-conduit">
     <http:tlsClientParameters secureSocketProtocol="TLS" disableCNCheck="true" useHttpsURLConnectionDefaultSslSocketFactory="false">
              <sec:keyManagers keyPassword="${myapp.password}">
                 <sec:keyStore type="JKS" resource="${myapp.keystore}"/>
              </sec:keyManagers>
       </http:tlsClientParameters>
      <http:client ConnectionTimeout="30000" ReceiveTimeout="30000"/>
</http:conduit>

Make the timeout value configurable via a properties file. Don't hard code it.

2. If you want to set it from you Java code

Future<String> future = executor.submit(new Callable<String>() {
    public String call() {
        return webService.executeServiceMethod();
    }
});

try {
    String result = future.get(5, TimeUnit.SECONDS);
    processResult(result);
} catch (TimeoutException ex) {
    retryServiceOrDisplayErrorMessage();
}

Make the timeout value configurable via a properties file. Don't hard code it. Here is a tutorial like example on timeout.


3. Application server configurations. For example, in Weblogic application server the JTA timeout is set via the server console

In Tomcat, via server.xml file

<Connector URIEncoding="UTF-8" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" enableLookups="false" maxHttpHeaderSize="8192" maxSpareThreads="75" maxThreads="150" minSpareThreads="25" port="7777" redirectPort="8443"/>

Most Java framework APIs provide timeout facilities, and make use of it.


Q. How will you go about performing outage testing of your application(s)?
A.

1. Setting up a dummy service to test.  Understanding dynamic proxies in Java with a practical example -- Service retry example
2. SSH tunneling via an SSH client like putty to a real service. Set up an SSH tunnel to the actual service, and then use the tunnel host/port number in the client application. Severe the tunnel connectivity, and see how your client application handles connection disruption.
3. By bringing down the actual service. Also known as the real outage testing. You need to notify other users of the actual service, and book a time for this outage testing to not impact other client applications.



Q. How  will you monitor real service outages of your java applications?
A.

• Java Management eXtentions (JMX) provides a way for you to connect to remote clients and monitor applications running on a JVM. 
• Monitoring your Java application infrastructures with Nagios, which provides complete monitoring of Java applications and servers. Nagios Offers effective JMX monitoring.
• Splunk can monitor your log files to predict behviors and proactively address potential outages

• Commercial monitoring tools like Tivoli from IBM and SiteScope from HP.