Google

Oct 22, 2014

Yammer metrics to monitor RESTful web services and report them via its admin web

This is a Yammer metrics tutorial to monitor RESTful web services. This extends RESTEasy web service tutorial basic and the basic stand-alone Yammer tutorial.

Step 1: The pom.xml file needs to have relevant dependencies defined relating to Yammer and Spring Web in addition to the JAX-RS librraies that was covered in the previous tutorial.


  <!-- Yammer Metrics -->
  <dependency>
   <groupId>com.yammer.metrics</groupId>
   <artifactId>metrics-core</artifactId>
   <scope>compile</scope>
   <version>${metrics.version}</version>
  </dependency>
  <dependency>
   <groupId>com.yammer.metrics</groupId>
   <artifactId>metrics-annotation</artifactId>
   <scope>compile</scope>
   <version>${metrics.version}</version>
  </dependency>
  <dependency>
   <groupId>com.yammer.metrics</groupId>
   <artifactId>metrics-spring</artifactId>
   <scope>runtime</scope>
   <version>${metrics.version}</version>
  </dependency>
  <dependency>
   <groupId>com.yammer.metrics</groupId>
   <artifactId>metrics-servlet</artifactId>
   <scope>compile</scope>
   <version>${metrics.version}</version>
  </dependency>
  <dependency>
   <groupId>com.yammer.metrics</groupId>
   <artifactId>metrics-web</artifactId>
   <scope>runtime</scope>
   <version>${metrics.version}</version>
  </dependency>

  <!-- Servelet API -->
  <dependency>
   <groupId>javax.servlet</groupId>
   <artifactId>javax.servlet-api</artifactId>
   <scope>provided</scope>
   <version>${javax-servlet.version}</version>
  </dependency>

  <!-- Spring -->
  <dependency>
   <groupId>org.springframework</groupId>
   <artifactId>spring-web</artifactId>
   <scope>compile</scope>
   <version>${spring.version}</version>
  </dependency>


Step 2: Add metrics annotation to the RESTful web service interface and implementation

package com.mytutorial;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;

import com.yammer.metrics.annotation.ExceptionMetered;
import com.yammer.metrics.annotation.Timed;

@Path("/myapp")
public interface SimpleRestWeb {

 @GET
 @Path("/name/{name}")
 @Timed
 @ExceptionMetered
 public String sayHello(@PathParam("name") String name);

}


package com.mytutorial;

import java.util.concurrent.atomic.AtomicLong;

import com.yammer.metrics.annotation.Gauge;
import com.yammer.metrics.annotation.Timed;

public class SimpleRestWebImpl implements SimpleRestWeb {

 @Gauge
 AtomicLong requestCount = new AtomicLong();
 
 
@Override
@Metered
 public String sayHello(String name) {
  String result = "Hello " + name;
  requestCount.incrementAndGet();
  return result;
 }

}


Note: A gauge "requestCount" was added to count the number of requests.

Step 3: The applicationContext.xml file under com/mytutorial with mterics bootstrapping.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:metrics="http://www.yammer.com/schema/metrics"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
                        http://www.yammer.com/schema/metrics http://www.yammer.com/schema/metrics/metrics.xsd">
    
    <metrics:metrics-registry id="rest-metrics"/>
    <metrics:health-check-registry id="rest-health"/>
    <metrics:annotation-driven metrics-registry="rest-metrics" health-check-registry="rest-health" proxy-target-class="true" />
    
     <bean id="simpleRestWeb" class="com.mytutorial.SimpleRestWebImpl"  />
    
    
</beans>


Step 4: The web.xml file with the Yammer admin servlet and relevant listeners defined.

<!DOCTYPE web-app PUBLIC
 "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
 "http://java.sun.com/dtd/web-app_2_3.dtd" >

<web-app>
 <display-name>Archetype Created Web Application</display-name>


 <context-param>
  <param-name>contextConfigLocation</param-name>
  <param-value>classpath:/com/mytutorial/applicationContext.xml</param-value>
 </context-param>

 <context-param>
  <param-name>resteasy.use.deployment.sensitive.factory</param-name>
  <param-value>false</param-value>
 </context-param>

 <context-param>
  <param-name>resteasy.servlet.mapping.prefix</param-name>
  <param-value>/rest</param-value>
 </context-param>


 <listener>
  <listener-class>org.jboss.resteasy.plugins.server.servlet.ResteasyBootstrap</listener-class>
 </listener>
 <listener>
  <listener-class>org.jboss.resteasy.plugins.spring.SpringContextLoaderListener</listener-class>
 </listener>
 <listener>
     <listener-class>com.mytutorial.MetricsContextLoaderListener</listener-class>
    </listener>

 
 <servlet>
  <servlet-name>resteasy-simple</servlet-name>
  <servlet-class>org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher</servlet-class>
 </servlet>
 <servlet>
  <servlet-name>admin-servlet</servlet-name>
  <servlet-class>com.yammer.metrics.reporting.AdminServlet</servlet-class>
  <load-on-startup>2</load-on-startup>
 </servlet>


 <servlet-mapping>
  <servlet-name>resteasy-simple</servlet-name>
  <url-pattern>/rest/*</url-pattern>
 </servlet-mapping>
 <servlet-mapping>
  <servlet-name>admin-servlet</servlet-name>
  <url-pattern>/admin/*</url-pattern>
 </servlet-mapping>
</web-app>


Step 5: The ServletContextListener com.mytutorial.MetricsContextLoaderListener defined in the above web.xml is a custom listener used to make MetricsRegistry available from the spring context to the Yammer admin servlet defined in the above web.xml.

package com.mytutorial;

import javax.servlet.ServletContext;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

import org.springframework.web.context.WebApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import com.yammer.metrics.core.MetricsRegistry;
import com.yammer.metrics.reporting.MetricsServlet;

/**
 * Application Lifecycle Listener for Metrics.
 * 
 *I need to fetch the MetricsRegistry from the Spring Context and put it into the ServletContext,
 * for the yammer admin-servlet to be able to find it.....
 *
 */
public class MetricsContextLoaderListener implements ServletContextListener {

    public MetricsContextLoaderListener() {

    }

 public void contextInitialized(ServletContextEvent event) {
        ServletContext context = event.getServletContext();
        MetricsRegistry metricsRegistry = getMetricsRegistry(context);
        setMetricsRegistry(metricsRegistry, context);
    }

 
    public void contextDestroyed(ServletContextEvent event) {

    }

   
    protected MetricsRegistry getMetricsRegistry(ServletContext context) {
        WebApplicationContext springContext = WebApplicationContextUtils.getWebApplicationContext(context); 
        return springContext.getBean(MetricsRegistry.class);
    }
    
   
    protected void setMetricsRegistry(MetricsRegistry registry, ServletContext context) {
        context.setAttribute(MetricsServlet.REGISTRY_ATTRIBUTE, registry);
    }
}


Step 6: Build and deploy the war file.

Hit the url: http://localhost:8080/tutorial/rest/myapp/name/arul to invoke the RESTful web service.
Hit the url: http://localhost:8080/tutorial/admin/metrics?pretty=true to get the metrics as json String.
Hit the url: http://localhost:8080/tutorial/admin to get the Yammer menu.

Step 7: The json output for metrics (e.g. http://localhost:8080/tutorial/admin/metrics?pretty=true) will look like

{
{
  "jvm" : {
    "vm" : {
      "name" : "Java HotSpot(TM) 64-Bit Server VM",
      "version" : "1.6.0_45-b06"
    },
    "memory" : {
      "totalInit" : 2.695299072E9,
      "totalUsed" : 2.8351824E8,
      "totalMax" : 5.953159168E9,
      "totalCommitted" : 2.606497792E9,
      "heapInit" : 2.155872256E9,
      "heapUsed" : 2.29205536E8,
      "heapMax" : 3.780509696E9,
      "heapCommitted" : 2.066087936E9,
      "heap_usage" : 0.06062821006450872,
      "non_heap_usage" : 0.02499863539883529,
      "memory_pool_usages" : {
        "Code Cache" : 0.06720225016276042,
        "PS Eden Space" : 0.16319328855716045,
        "PS Old Gen" : 0.009577277144031412,
        "PS Perm Gen" : 0.023997800623475327,
        "PS Survivor Space" : 0.0
      }
    },
    "daemon_thread_count" : 28,
    "thread_count" : 49,
    "current_time" : 1395644957593,
    "uptime" : 24,
    "fd_usage" : "NaN",
    "thread-states" : {
      "new" : 0.0,
      "timed_waiting" : 0.4897959183673469,
      "terminated" : 0.0,
      "waiting" : 0.16326530612244897,
      "blocked" : 0.0,
      "runnable" : 0.3469387755102041
    },
    "garbage-collectors" : {
      "PS MarkSweep" : {
        "runs" : 2,
        "time" : 134
      },
      "PS Scavenge" : {
        "runs" : 2,
        "time" : 33
      }
    }
  },
  "com.mytutorial.SimpleRestWebImpl" : {
    "requestCount" : {
      "type" : "gauge",
      "value" : 1
    },
    "sayHello" : {
      "type" : "timer",
      "duration" : {
        "unit" : "milliseconds",
        "min" : 2999.710987,
        "max" : 2999.710987,
        "mean" : 2999.710987,
        "std_dev" : 0.0,
        "median" : 2999.710987,
        "p75" : 2999.710987,
        "p95" : 2999.710987,
        "p98" : 2999.710987,
        "p99" : 2999.710987,
        "p999" : 2999.710987
      },
      "rate" : {
        "unit" : "seconds",
        "count" : 1,
        "mean" : 0.06855060368303201,
        "m1" : 0.0,
        "m5" : 0.0,
        "m15" : 0.0
      }
    }
  }
}


Take note of "requestCount", which counts the number of requests.

Labels: , ,

Oct 1, 2014

Yammer metrics to monitor RESTful web services with a Servlet filter

This extends Yammer metrics to monitor RESTful web services and report them via its admin web to demonstrate how a Servlet  filter can be added to gather metrics for
  • response codes for 200, 201, 404, etc to get a count.
  • active requests count
  • requests timing.

Step 1:  Decorate the "HttpServletResponseWrapper" class. HttpServletResponseWrapper is one particular implementation of HttpServletResponse which gives you a convenient way to wrap an existing response with some logic of your own without having to write a whole new implementation of the interface.



package com.mytutorial;

import java.io.IOException;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;

public class StatusServletResponse extends HttpServletResponseWrapper {
 private int httpStatus;

 public StatusServletResponse(HttpServletResponse response) {
  super(response);
 }

 @Override
 public void sendError(int sc) throws IOException {
  this.httpStatus = sc;
  super.sendError(sc);
 }

 @Override
 public void sendError(int sc, String msg) throws IOException {
  this.httpStatus = sc;
  super.sendError(sc, msg);
 }

 @Override
 public void setStatus(int sc) {
  this.httpStatus = sc;
  super.setStatus(sc);
 }

 public int getStatus() {
  return this.httpStatus;
 }
}



Step 2: Add a Servlet filter to capture the above metrics. This where Yammer metrics is used.

package com.mytutorial;

import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import java.util.concurrent.TimeUnit;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletResponse;

import org.springframework.web.context.WebApplicationContext;
import org.springframework.web.context.support.WebApplicationContextUtils;

import com.yammer.metrics.core.Counter;
import com.yammer.metrics.core.Meter;
import com.yammer.metrics.core.MetricsRegistry;
import com.yammer.metrics.core.Timer;
import com.yammer.metrics.core.TimerContext;

public class MetricsFilter implements Filter {

 private static final String NAME_PREFIX = "responseCodes.";
 private static final int OK = 200;
 private static final int CREATED = 201;
 private static final int NO_CONTENT = 204;
 private static final int BAD_REQUEST = 400;
 private static final int NOT_FOUND = 404;
 private static final int SERVER_ERROR = 500;

 private ConcurrentMap<Integer, Meter> metersByStatusCode;
 private Counter activeRequests;
 private Timer requestTimer;
 
 public MetricsFilter() {

    }
    

 public void init(FilterConfig filterConfig) throws ServletException {
  MetricsRegistry metricsRegistry = getMetricsRegistry(filterConfig);
  Map<Integer, String> meterNamesByStatusCode = getMeterNamesByStatesCode();

  this.metersByStatusCode = new ConcurrentHashMap<Integer, Meter>(meterNamesByStatusCode.size());
  for (Entry<Integer, String> entry : meterNamesByStatusCode.entrySet()) {
   metersByStatusCode.put(entry.getKey(),
     metricsRegistry.newMeter(this.getClass(), entry.getValue(), "responses", TimeUnit.SECONDS));
  }

  this.activeRequests = metricsRegistry.newCounter(this.getClass(), "activeRequests");
  this.requestTimer = metricsRegistry.newTimer(this.getClass(), "requests", TimeUnit.MILLISECONDS,
    TimeUnit.SECONDS);
 }

 public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException,
   ServletException {
  final StatusServletResponse wrappedResponse = new StatusServletResponse(
    (HttpServletResponse) response);
  this.activeRequests.inc();
  final TimerContext context = this.requestTimer.time();
  try {
   chain.doFilter(request, wrappedResponse);
  } finally {
   context.stop();
   this.activeRequests.dec();
   markMeterForStatusCode(wrappedResponse.getStatus()); 
  }

 }
 
 
  private void markMeterForStatusCode(int status) {
         final Meter metric = metersByStatusCode.get(status);
         if (metric != null) {
             metric.mark();
         } 
     }


 public void destroy() {

 }

 protected MetricsRegistry getMetricsRegistry(FilterConfig config) {
  WebApplicationContext springContext = WebApplicationContextUtils.getWebApplicationContext(config
    .getServletContext());
  return springContext.getBean(MetricsRegistry.class);
 }

 protected Map<Integer, String> getMeterNamesByStatesCode() {
  final Map<Integer, String> meterNamesByStatusCode = new HashMap<Integer, String>(6);
  meterNamesByStatusCode.put(OK, NAME_PREFIX + "ok");
  meterNamesByStatusCode.put(CREATED, NAME_PREFIX + "created");
  meterNamesByStatusCode.put(NO_CONTENT, NAME_PREFIX + "noContent");
  meterNamesByStatusCode.put(BAD_REQUEST, NAME_PREFIX + "badRequest");
  meterNamesByStatusCode.put(NOT_FOUND, NAME_PREFIX + "notFound");
  meterNamesByStatusCode.put(SERVER_ERROR, NAME_PREFIX + "serverError");
  return meterNamesByStatusCode;
 }
}

Step 3: Register this Servlet filter via web.xml

<!DOCTYPE web-app PUBLIC
 "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
 "http://java.sun.com/dtd/web-app_2_3.dtd" >

<web-app>
 <display-name>Archetype Created Web Application</display-name>


 <context-param>
  <param-name>contextConfigLocation</param-name>
  <param-value>classpath:/com/mytutorial/applicationContext.xml</param-value>
 </context-param>

 <context-param>
  <param-name>resteasy.use.deployment.sensitive.factory</param-name>
  <param-value>false</param-value>
 </context-param>

 <context-param>
  <param-name>resteasy.servlet.mapping.prefix</param-name>
  <param-value>/rest</param-value>
 </context-param>


 <listener>
  <listener-class>org.jboss.resteasy.plugins.server.servlet.ResteasyBootstrap</listener-class>
 </listener>
 <listener>
  <listener-class>org.jboss.resteasy.plugins.spring.SpringContextLoaderListener</listener-class>
 </listener>
  <listener>
   <listener-class>com.mytutorial.MetricsContextLoaderListener</listener-class>
  </listener>

 <filter>
  <description>
   Collects request metrics...</description>
  <display-name>MetricsFilter</display-name>
  <filter-name>MetricsFilter</filter-name>
  <filter-class>com.mytutorial.MetricsFilter</filter-class>
 </filter>

 <filter-mapping>
  <filter-name>MetricsFilter</filter-name>
  <servlet-name>resteasy-simple</servlet-name>
 </filter-mapping>

 <servlet>
  <servlet-name>resteasy-simple</servlet-name>
  <servlet-class>org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher</servlet-class>
 </servlet>
 <servlet>
  <servlet-name>admin-servlet</servlet-name>
  <servlet-class>com.yammer.metrics.reporting.AdminServlet</servlet-class>
  <load-on-startup>2</load-on-startup>
 </servlet>


 <servlet-mapping>
  <servlet-name>resteasy-simple</servlet-name>
  <url-pattern>/rest/*</url-pattern>
 </servlet-mapping>
 <servlet-mapping>
  <servlet-name>admin-servlet</servlet-name>
  <url-pattern>/admin/*</url-pattern>
 </servlet-mapping>
</web-app>


Step 4: Deploy the application to a web server.

Step 5: Invoke the RESTFul web service via :  http://localhost:8080/tutorial/rest/myapp/name/sam

Step 6: Invoke the Yammer metrics admin servlet via: http://localhost:8080/tutorial/admin and then click on "metrics".

Step 7: The metrics will be displayed as JSON data.Only a subset of JSON data is shown

"com.mytutorial.MetricsFilter" : {
    "activeRequests" : {
      "type" : "counter",
      "count" : 1
    },
    "requests" : {
      "type" : "timer",
      "duration" : {
        "unit" : "milliseconds",
        "min" : 3075.95668,
        "max" : 3075.95668,
        "mean" : 3075.95668,
        "std_dev" : 0.0,
        "median" : 3075.95668,
        "p75" : 3075.95668,
        "p95" : 3075.95668,
        "p98" : 3075.95668,
        "p99" : 3075.95668,
        "p999" : 3075.95668
      },
      "rate" : {
        "unit" : "seconds",
        "count" : 1,
        "mean" : 0.01570764346171342,
        "m1" : 0.015991117074135343,
        "m5" : 0.0033057092356765017,
        "m15" : 0.0011080303990206543
      }
    },
    "responseCodes.badRequest" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 0,
      "mean" : 0.0,
      "m1" : 0.0,
      "m5" : 0.0,
      "m15" : 0.0
    },
    "responseCodes.created" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 0,
      "mean" : 0.0,
      "m1" : 0.0,
      "m5" : 0.0,
      "m15" : 0.0
    },
    "responseCodes.noContent" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 0,
      "mean" : 0.0,
      "m1" : 0.0,
      "m5" : 0.0,
      "m15" : 0.0
    },
    "responseCodes.notFound" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 0,
      "mean" : 0.0,
      "m1" : 0.0,
      "m5" : 0.0,
      "m15" : 0.0
    },
    "responseCodes.ok" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 1,
      "mean" : 0.015706988535186733,
      "m1" : 0.015991117074135343,
      "m5" : 0.0033057092356765017,
      "m15" : 0.0011080303990206543
    },
    "responseCodes.serverError" : {
      "type" : "meter",
      "event_type" : "responses",
      "unit" : "seconds",
      "count" : 0,
      "mean" : 0.0,
      "m1" : 0.0,
      "m5" : 0.0,
      "m15" : 0.0
    }
  }
  
  

Labels: ,

Sep 2, 2014

Monitoring and gathering statistics with JMX based jconsole and Yammer metrics

This post extends  Yammer Metrics with Spring tutorial to demonstrate how the metrics can be monitored for the TradeEngineMain application.

Step 1: Run the TradeEngineMain  java class created in the Yammer metrics tutorial.

Step 2: From the DOS command prompt, type jconsole

C:\Users\akumaras>jconsole

Step 3:  The jconsole window pops up as shown below.



Step 4: You can monitor the statistics via the MBeans tab as shown below.



Drill down into each and observe the values.




With Yammer metrics, you can also output metrics to CSV, as JSON via RESTful web service, etc. It is quite a handy tool to monitor your applications.

Labels: , ,

Mar 14, 2014

Yammer Metrics with Spring tutorial

When you are running long term applications like web applications, batch jobs, or stand-alone status update jobs, it is good to know some statistics about them, like number of requests served or request duration. You can also gather more generic information like the state of your internal collections, how many times some portion of code is being executed, or health checks like database availability, or any kind of connection to an external system.

All this kind of instrumentation can be achieved by using native JMX or using a modular project like yammer Metrics. Metrics provides a powerful way to measure the behaviour of your critical components and reporting them to a variety of systems like, JConsole, System Console, Ganglia, Graphite, CSV, or making them available through a web services as JSON data.

Step 1: Required libraries. The metrics-core, metrics-annotation, and metrics-spring are the key libraries for gathering metrics.



Step 2: Artifacts used.


Step 3: Let's create a very basic TradeEngine, and monitor the number of requests and request execution times.


package com.writtentest12;

public interface TradeEngine {
 
    abstract void execute(Request... requests);
}



package com.writtentest12;

import java.util.concurrent.atomic.AtomicInteger;
import com.yammer.metrics.annotation.ExceptionMetered;
import com.yammer.metrics.annotation.Gauge;
import com.yammer.metrics.annotation.Timed;

public class TradeEngineImpl implements TradeEngine {

 @Gauge
 private final AtomicInteger currentRequests = new AtomicInteger();

 
 @Timed
 @ExceptionMetered
 public void execute(Request... requests) {
  executingRequests(requests.length);
  try {
   Thread.sleep(2000); //just to emulate some processing
  } catch (InterruptedException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
 }

 private void executingRequests(int count) {
  this.currentRequests.addAndGet(count);
 }
}

Take notice of @Gauge, @Timed, and @ExceptionMetered annotations used for monitoring. The Request class used in the TradeEngine.

package com.writtentest12;

public class Request {

}


Step 4: Spring applicationContext.xml file to wire up TradeEngine, metrics-registry, and jmx-reporter.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:aop="http://www.springframework.org/schema/aop"
    xmlns:metrics="http://www.yammer.com/schema/metrics"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd
                        http://www.yammer.com/schema/metrics http://www.yammer.com/schema/metrics/metrics.xsd">
    
    
    
    <metrics:metrics-registry id="trade-metrics"/>
    <metrics:health-check-registry id="trade-health"/>
    <metrics:annotation-driven metrics-registry="trade-metrics" />
    <metrics:jmx-reporter metrics-registry="trade-metrics"/>
    
 <bean id="tradeEngine" class="com.writtentest12.TradeEngineImpl"/>
    
    
</beans>


Step 5: Stand-alone TradeEngineMain class that runs until forcefully stopped. Run continuously so that you can monitor the metrics via JMX compliant jconsole.

package com.writtentest12;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class TradeEngineMain {

 public static void main(String[] args) {

  ApplicationContext context = new ClassPathXmlApplicationContext(
    new String[] { "com/writtentest12/applicationContext.xml" });

  TradeEngine tradeEngine = (TradeEngine) context.getBean("tradeEngine");

  try {
   while (true) {
    Request[] requests = new Request[2];
    requests[0] = new Request();
    requests[1] = new Request();
    tradeEngine.execute(requests);

    Thread.sleep(5000);
   }
  } catch (InterruptedException e) {
   e.printStackTrace();
  }

 }
}



You can run the  TradeEngineMain  application, and then start the jconsole from a command-line and monitor the number of requests processed and the stats on the execution time. I will cover this in the next post.


Labels: , ,

Aug 19, 2013

Visual VM for monitoring Java memory, CPU, threads, etc


VisualVM is a visual tool integrating several command line JDK tools and lightweight profiling capabilities. Designed for both production and development time use, it further enhances the capability of monitoring and performance analysis for the Java SE platform. It is packaged as an exe file. \


Step 1: You can start the visual vm by double clicking on %JAVA_HOME%/bin/jvisualvm.exe from Java 1.6 version on wards.

Step 2: Your local processes will be monitored under the local tab. The Visual vm can also used to open the heap dump files i.e *.hprof files to analyze the menory usages. You can find out the process ids of your local Java applications via

1. netstat -anp | grep 8088 or
2. In windows via the "Windows Task Manager" -->  Processes tab. You need to click on View --> Select  Columns and then "tick" PID (Process Identifier) check box.

Double click on the relevant PID in Visual VM console.

 



You can add remote processes by following the steps shown below.

1. Right click on  "Remote" and then select  "Add Remote Host...".
2. Provide the host name like "myapp.com".
3. It searches and adds the host.
4. Right click on the added host name and then select  "Add JMX Connection" and in the "Connection" field type the hostname:JMX port number like myapp.com:8083.
5. Double click on this JMX connection to monitor CPU, memory, thread, etc.






Labels: , , ,

Jul 4, 2013

Java Application Monitoring with Tivoli

Tivoli Management Framework (TMF) is a CORBA-based network management framework from IBM. BM Tivoli Monitoring helps you optimize IT infrastructure performance and availability. here is a high level overview of using Tivoli to monitor Java log files for presence of some text like errors and then raise support tickets at different levels like p1,p2,p3, and p4. The "P" is for prodyction environment and 1-4 are level of importance. Any p1 and p2 level issues need to be attended immediately. The SCIM is used to identify the source of the ticket, and it stands for System, Component, Item, and Module.


Firstly, you need to out put special text in the log to be picked up by the Tivoli.


Step 1: Use log4j to output text like "Monitoring|MyApp|TRADEFORECAST some exception followed ......". Say the exceptions are logged to a log file named myapp.log.

Step 2: The Tivoli config file needs to be configured as shown below with  log file name, regex pattern of the text to search, the SCIM name, queue name, and the ticket level as shown below.

[/local/myapp/applogs/myapp.log;(.*)Monitoring\|MyApp\|TRADEFORECAST(.*);Likely trade Feed Generation Failure]
warning.scim=APPLICATION/MYAPP/TRADE/ERROR
warning.page=QUEUE1
warning.prisev=P4


[/local/myapp/applogs/myapp.log;(.*)Monitoring\|MyApp\|TRADE_LINK(.*);Likely trade Feed Generation Failure.]
warning.scim=APPLICATION/MYAPP/TRADE/ERROR
warning.page=QUEUE1
warning.prisev=P4



"(.*)Monitoring\|MyApp\|TRADE_LINK(.*)" is the regex to look for in the myapp.log file, and "Likely trade feed generation failure" will be highlighted in the p4 ticket raised via the Peregrine ITSM system. Peregrine system is an ITSM (i.e IT Service Management) software. The SCIM is  "APPLICATION/MYAPP/TRADE/ERROR" to identify which system is raising a ticket, and QUEUE1 is the Peregrine system queue name.

Note: Nagios is a very popular monitoring tool and it is an open source tool. It is a network monitoring tool to test availability of servers, applications, etc.

Labels:

Jul 3, 2013

Splunk to analyse Java logs and other machine data

Q. What is Splunk and where will you use it?
A. Splunk is an enterprise-grade software tool for collecting and analyzing “machine data” like log files, feed files, and other big data in terra bytes. You can upload logs from your websites and let Splunk index them, and produce reports with graphs to analyze the reports. This is very useful in capturing start and finish times from asynchronous processes to calculate elapsed times. For example, here are the basic steps required.

Step 1: log4j MDC logging can be used to output context based logs and then  
Step 2: upload that into Splunk to index and produce elapsed times for monitoring application performance.


Q. What are the different ways to get data into Splunk?
A
  1. Uploading a log file via Splunk's web interface.
  2. Getting Splunk to monitor a local directory or file.
  3. Splunk can index data from any network port. For example, Splunk can index remote data from syslog-ng or any other application that transmits via TCP. Splunk can also receive and index SNMP events.
  4. Splunk also supports other kinds of data sources like FIFO queues and Scripted inputs to get data from APIs and other remote data interfaces and message queues. for example, here is a simple scripted script via input.conf file.
 
[script://$SCRIPT] 
<attrbute1> = <val1>
<attrbute2> = <val2>
...


Here is an example of using Splunk to write query against log files to monitor performance.


Step 1: Configure the Java application to output log statements as shown below using MDC (Mapped Diagnostic Context) feature offered by log-back or log4j.

  
2013-05-31 19:36:03,617 INFO  [Camel (MyApp) thread #24 - Multicast] c.j.w.a.c.s.i.MyAppForecastServiceImpl - [groupId=48937,jobType=CASH_FEED,ValuationDate=2013-01-04] - Total Time spent on group level cashforecast feed - 225 ms 

Step 2: Upload the test-myapp.log  file via the Splunk interface.







Step 3: Once the file is uploaded, you can write the search queries to your requirement.





The query shown above is

  
source="test4.aes.log" | 
search "Total Time" | search "group level cashforecast"  

The Splunk search language is very powerful. Here is an extended search language.

  
source="test.aes.log" | 
search "Total Time" | search "group level cashforecast"  |  rex field=_raw "feed - (?\d+)" | 
bucket _time span=5d | stats avg(timeTaken) as AvgDur, max(timeTaken) as MaxDur  by _time 

Labels: , ,