The issue Logging is as old as software programming itself. Where thin paper strips left a typing machine those days, masses of log appenders are configured in todays’ Enterprise Application servers. The extensive use can soon become overwhelming due to the billion of log entries written to plain log files.
The topic of efficiently processing these output streams has become more important in the last years. Errors in highly parallel threads, maybe cloned or clustered over many machines doing the same task slightly different due to user interaction or input data, can reside well hidden in the dark, only showing up in complex situations. Simple log file scanning la Matrix most times will not reveal the red-dressed blonde. Manually chasing stack traces in plain, most times clustered log files ends up in searching for the String ERROR, and after that for the needle in the haysack.
Many tools address this downside. It’s most popular commercial product seems to be splunk, an open source alternative is GrayLog2. The first may be called expensive, both look very time-consuming at first sight, because they demand extra infrastructure and administration.
In one of our inhouse tech-labs, we tried to prototype a lean solution which just gets along with a minimal appender an external store, and a Web Frontend, all together inside the source application maybe. The result was a Grails frontend, a MongoDB for log storage and many cool ideas what can be analyzed in fact.
All starts with this log4j definition:
<appender name="mongo" class="com.comsysto.logtopus.log4j.Log4JMongoDBAppender"> <param name="Threshold" value="DEBUG"/> <param name="databaseName" value="logtopus-db"/> <param name="host" value="10.10.0.42"/> <param name="applicationName" value="logtopus-example"/> <param name="version" value="0.1-DEV"/> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%5p %d [%t] (%F:%L) - %m%n"/> </layout> </appender>
The object behind stores every message document to a MongoDB, including some sophisticated values for later analysis. If your application already runs on MongoDB, you are just done.
The really interesting stuff comes up when thinking about what can be analyzed. We were also in touch with splunk before and felt the high degree of freedom it offers. But most times, log crawling can be reduced to some simple recurring questions: How does the log concern distribution look like? Were there any errors the last hour, day, week? Are there uncaught runtime exceptions?
These questions can be answered with simple queries against the message store. Some quick Google charts together make a nice dashboard.
The fancy ideas arise when thinking about correlations between different fields of a log message. We thought of the software version or build number for example. In genreal, this information is easily available for example by reading the Manifest file or defining it via the version property in the appender. With this small tweak we are now able to track the logging behavior and application quality over several versions! And we can prove that the correct version is live, which is not that easy sometimes.
The logging of java code debug details like source code line and called method is heavy duty for the logger, but allows another correlation feature which leads to quality hotspot analysis on class or even method level at runtime. Think about the signature a stored log message can have, maybe consisting of class, package, version, log level and occured exception if existing. Counting similar or same incidents leads to a code heatmap which is useful for optimization and bug tracing.
To put all that in a nutshell, it is not vital to build your own custom logging toolchain, but it leads to new ideas of runtime analysis. And in the end we found it easier to extend our logging appender to prepare our data for specific queries, rather than trying to insert key words in our log messages and building complex view and reports on it after that. If you are interested in more details, feel free to check out our small prototype on GitHub including appender, test data creator and Grails Frontend.