The Cloud Foundry Blog

Rethink Application Logging with RabbitMQ

Application logging is essential to monitoring applications and diagnosing problems, but in a cloud universe where file systems may be ephemeral and application instances are provisioned dynamically keeping track of your application logs can be a challenge. Fortunately with Cloud Foundry you can get fast, multi-instance, cloud-proof application logging in just a few steps with RabbitMQ In this blog post I’ll show you how to configure a Spring application so that it logs messages via AMQP. But first, what are the benefits? The primary motivations are that it’s fast and it doesn’t rely on the local file system of your application (which can disappear in a cloud environment). It’s also very easy to set up multiple applications to log to the same message broker, allowing you to aggregate log messages. Finally, you have the power to route and process those messages, for example allowing you to index them for search and store them in a blob store. So how do you go about logging messages via the broker? If you’re happy to use Spring, you’re already ahead of the game. One of the best kept secrets of the Spring AMQP project is an AMQP Log4j appender that allows you log messages to any AMQP broker. All you have to do is configure it.

Application changes

As an example, I’m going to use the Spring travel application. The same principles apply to any Java, servlet-based web application. In order to send log messages to a non-standard location, you have to attach the appropriate appender to a logger. Typically, you attach it to the root logger. The easiest way to do this is to add the appender to your Log4j configuration file, be it a properties file or XML. The Spring travel application uses XML configuration, so adding the AMQP appender is a simple case of inserting the appender definition into log4j.xml:


<?xml version="1.0" encoding="UTF-8"?>  
  <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/"> 
  ... 
 <appender name="amqp" class="org.springframework.amqp.rabbit.log4j.AmqpAppender"> 
    <param name="ExchangeName" value="amq.topic" /> 
    <param name="ExchangeType" value="topic" /> 
    <param name="RoutingKeyPattern" value="logs.spring-travel" /> 
    <param name="ApplicationId" value="spring-travel" /> 
  </appender> 
  ... 
 </log4j:configuration>

and then adding the appender to whichever loggers you want. In the case of the root logger:

 
<root> 
  <priority value="info" /> 
  <appender-ref ref="console" /> 
  <appender-ref ref="amqp"/> 
</root>

These simple changes will result in the Spring Travel application logging messages to a RabbitMQ broker – as long as it’s running on the default port and with the standard ‘guest’ account. You can configure the connection settings for the broker in log4j.xml if you know the broker host and port information in advance, but Cloud Foundry assigns these values dynamically. Fortunately it is easy to configure your Cloud Foundry application to use those dynamically assigned values. Since the connection settings for the various Cloud Foundry services are injected into an application at runtime (via an environment variable), we have to programmatically configure Log4j at runtime as well. This is a simple case of adding a ServletContextListener that checks whether the application is running inside Cloud Foundry and if so, updates the connection settings of the “amqp” appender. The full code is in the application sample, but the basic process involves initializing the logging from log4j.xml, getting the appender by name, closing it (so that any existing AMQP connections are dropped) and then updating the connection settings. Of course, we need to make sure the listener actually runs! So, drop the listener definition into the applications web descriptor:

 
<listener> 
  <listener-class>org.springframework.samples.travel.web.Log4jConfigListener</listener-class> 
</listener>

The last piece of the puzzle is to make sure that AmqpAppender is on the application’s classpath, which means you need to add the spring-rabbit library as a dependency:

 
<dependency> 
  <groupId>org.springframework.amqp</groupId> 
  <artifactId>spring-rabbit</artifactId> 
  <version>1.0.0.RELEASE</version> 
</dependency>

And because we’re using the CloudEnvironment class to extract the RabbitMQ connection settings from Cloud Foundry, we also need the cloudfoundry-runtime JAR, although that’s already part of the Spring Travel sample project:

 
<dependency> 
  <groupId>org.cloudfoundry</groupId> 
  <artifactId>cloudfoundry-runtime</artifactId> 
  <version>0.8.1</version> 
</dependency>

Everything is now in place, so we’re ready to deploy the application to Cloud Foundry. All that’s unique to this deployment is that you have to bind a RabbitMQ service called “rabbitmq-logs” to the application. This is because that’s the name of the service the code in Log4jConfigListener looks for. If you want to try this app yourself, just clone the repository and use the latest vmc to deploy it:

$ git clone https://github.com/pledbrook/spring-travel.git
$ cd spring-travel/server
$ mvn package
$ vmc push --path target
Would you like to deploy from the current directory? [Yn]: 
Application Name: spring-travel
Detected a Java SpringSource Spring Application, is this correct? [Yn]: 
Application Deployed URL [spring-travel.cloudfoundry.com]: koala-travel
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]:    
How many instances? [1]: 
Bind existing services to 'spring-travel'? [yN]: 
Create services to bind to 'spring-travel'? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 2
Specify the name of the service [mysql-6117e]: travel-rdbms
Create another? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 4
Specify the name of the service [rabbitmq-abd55]: rabbitmq-logs
Create another? [yN]: 
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [travel-rdbms]: OK
Binding Service [travel-rdbms]: OK
Binding Service [rabbitmq-logs]: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (41K): OK
Push Status: OK
Staging Application 'spring-travel': OK
Starting Application 'spring-travel': OK

Don’t forget to change the application URL if you’re deploying to cloudfoundry.com, otherwise you may be told that it’s already in use and the deployment will fail. Once you have successfully deployed the application and it starts up, those log messages will be flying! Hmmm…but where will they be flying to?

Who ate all the messages?

RabbitMQ will shift those log messages at a tremendous rate and retain them up to a point, but those messages need to be routed somewhere if you want to view them easily or search the logs in a week’s time. We obviously need to pick up those messages and put them into a data store for easy retrieval. For this blog post, I’m going to create a simple Grails application that consumes the messages and stores them in a MongoDB instance. I could use a relational database, but ACID and referential integrity aren’t important in this scenario. There are two parts to this application: a message consumer that stores the log messages in the database and a UI for viewing the messages. For the first part, I just declare dependencies on the RabbitMQ and MongoDB plugins (removing the Hibernate plugin) and create a handler service that consumes the messages and stores them into MongoDB:

package org.example 
import org.springframework.amqp.core.Message 
class MessageStoreService { 
  static transactional = false 
  static rabbitSubscribe = [ name: "amq.topic", routingKey: "logs.#", messageConverterBean: ""] 
  void handleMessage(Message msg) { 
    def props = msg.messageProperties 
    def logMessage = new LogMessage( message: new String(msg.body, props.contentEncoding ?: "UTF-8"), appName: props.receivedRoutingKey - "logs.", category: props.headers.categoryName, level: props.headers.level).save(flush: true) 
  } 
} 

The handleMessage() method is invoked each time a message is received from the amq.topic exchange whose routing key matches “logs.#”. It just extracts the relevant information from the AMQP message and stores that in a LogMessage instance, which maps to a MongoDB record:

package org.example 
class LogMessage { 
  String appName String message String category String level static constraints = { category nullable: true level nullable: true } 
}

Note that the application name is pulled from the routing key of the message. Finally, the application has a scaffolded controller for LogMessage so that users can view the log messages:

package org.example 
class LogMessageController { static scaffold = true }

The above are the only extra classes you need on top of a fresh Grails application. You will also need to add some RabbitMQ configuration to grails-app/conf/Config.groovy:

rabbitmq { 
  connectionfactory { 
    username = 'guest' 
    password = 'guest' 
    hostname = 'localhost' 
    consumers = 5 
  } 
  queues = { 
    exchange name: "amq.topic", 
    type: topic, 
    durable: true, 
    autoDelete: false 
  } 
} 

The key entry is the declaration of the ‘amq.topic’ exchange. The full source is of this application is available on GitHub and can be deployed directly to any Cloud Foundry account. All you need to do is create and bind a MongoDB service to the application and bind “rabbitmq-logs” (or whatever RabbitMQ service you’re using for logging). You will also need to download and install Grails 2 to try this application.

$ git clone https://github.com/pledbrook/cf-log-app.git
$ cd cf-log-app
$ grails compile
$ grails
grails> cf-create-service rabbitmq rabbitmq-logs
Service 'rabbitmq-logs' provisioned.
grails> cf-create-service mongodb mongo-logs
Service 'mongo-logs' provisioned.
grails> prod cf-push --appname=cf-log-apps --services=rabbitmq-logs,mongo-logs
Building war file
| Done creating WAR target/cf-temp-1332242548101.war
> 
Application Deployed URL: 'cf-log-apps.cloudfoundry.com'? 

Creating application cf-log-apps at cf-log-apps.cloudfoundry.com with 512MB and services [rabbitmq-logs, mongo-logs]:
2012-03-20 11:22:40,498 [main] WARN  client.RestTemplate  - GET request for "http://api.cloudfoundry.com/apps/cf-log-apps" resulted in 404 (Not Found); invoking error handler
 OK
Uploading Application:
  Checking for available resources:
 OK
  Processing resources:
 OK
  Packing application:
 OK
  Uploading (6K):
 OK

Trying to start Application: 'cf-log-apps'......
Application 'cf-log-apps' started at http://cf-log-apps.cloudfoundry.com

As before, don’t forget to change the application name if you follow the above instructions. You can also see the application live on cloudfoundry.com with some log messages from the grailstwitter sample application.

Power logging

This is of course a very basic application but it does demonstrate how easy it is to aggregrate the logs from multiple applications into a single location. It would also be fairly easy to add a separate application that archives logs to a blob service, all the while ensuring low latency inside the applications that are performing the logging. This is the power of logging via a message broker. Why stop at Java? In a polyglot environment, logging via AMQP will allow you to aggregate logs from all sorts of frameworks, such as Node.js and Ruby on Rails. As long as the consumer can handle any differences between message formats, you’re good to go.

- Peter Ledbrook Cloud Foundry Developer Relations

This entry was posted in Cloud Foundry. Bookmark the permalink.

9 Responses to Rethink Application Logging with RabbitMQ

  1. Vikas says:

    Don’t like the idea presented here, this’s classic case of single point of failure.

  2. Pid says:

    Your hard disk is a single point of failure too.

  3. There is a long-ago solved problem (1998). Most (all?) significant Java logging can use syslog(), which does aggregation very well, and is likely a lot more efficient than routing through a Rabbit.

    http://en.wikipedia.org/wiki/Syslog-ng

    Unless there is some reason you cannot use syslog()?

    • Peter Ledbrook says:

      Preston: Thanks for your feedback. syslog is certainly an option in many cases, but only if you have access to its configuration. The nice thing about using RabbitMQ, particularly on a PaaS, is that the developer can control everything, including where messages are routed to. I’m also not sure you can aggregate messages from multiple VMs unless you route them off the VMs to a shared location.

      I’m interested why you think syslog would likely be a lot more efficient than RabbitMQ. Is that because it runs locally?

      As an aside, I found this blog post after you mentioned syslog: http://blog.theroux.ca/syslog/advanced-logging-on-linux/ – an example of using syslog and RabbitMQ together. The fact he mentions MongoDB and Elastic Search as well is purely coincidental!

  4. Eric Genesky says:

    Peter,
    The java and cloud communities at DZone might be interested in reading this post, perhaps generating some further discussion. Do you mind if I repost it (with attribution to you as original author and a link back to the original)?

  5. Pid says:

    Maybe using Windows?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>