When dealing with an exception stack trace we have to not only construct a separate match for our grok filter but also make sure that all lines will be treated as one entry.

What we are going to build

In this post I’ll show you how to:

• configure Filebeat to merge all lines of an exception stack trace into one entry;
• parse it it with Logstash with a custom pattern;
• clear tags from a false “failure tag”.

Make Filebeat read all stack trace lines as one entry

Filebeat reads an input file line by line. We have to explicitly tell it to treat a stack trace as a whole by using the multiline option:

If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.

https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline-examples

Below you’ll find my configuration file for Filebeat. It reads logs from the all.log file, applies the multiline plugin on those that match the patterns and sends everything to Logstash:

pattern

The Filebeat documentation contains useful examples of dealing with Java exceptions and the pattern I used is copied from there. It will merge lines starting with ‘...‘, ‘at‘ and ‘Caused by‘ from the example input given below:

Read the Regular expression support docs if you want to construct your own pattern for Filebeat. They differ slightly from the Logstash patterns.

match and negate

The behaviour of multiline depends on the configuration of those two options. The default value for the negate option is false. For match I used ‘after‘. As a result, matching lines are joined with a preceding line that doesn’t match (‘Exception in thread "main“…’ is concatenated with all the following lines that match the pattern).

Add a match for exceptions to your Logstash configuration

In my logstash.conf file I need filters that will handle:

• a Java stacktrace:

• a regular Spring Boot log:

Advice: You can save a lot of time while constructing your patterns by verifying them in the Grok Debbuger. It’s also a good idea to browse the list of the available predefined patterns first.

Because I don’t want to list all patterns in one match section, every entry is being checked against both matches (I think the break_on_match is not working in this case). As a result the _grokparsefailure tag will be added to all entries, even those that matched one of my patterns. I’m going to add my custom tags to solve this issue:

To remove the _grokparsefailure tag I have to know that a particular entry was successfully matched by one pattern – the stacktrace or spring_boot_log tag will be present in such a case. Therefore, I can safely delete the _grokparsefailure tag for entries that have my custom tag.

Verify results

Make sure that you have an output configured. It can be send to the STDOUT of the shell running Logstash:

I’m sending parsed logs to ElasticSearch and use ElasticHQ app to show you the results in a more readable way. You can see how the original message with an exception was parsed on the screenshot below:

You can see the exception part separated from the rest of the message as well as the stacktrace added to the tags. There is also the multiline flag added automatically.

Useful links

• Handling Multiline Stack Traces with Logstash (if you don’t read logs with Filebeat)
• Using Multiple Grok Statements to Parse a Java Stack Trace
• Is there a way to tag for different grok matches?
• grok multiple messages and process them with different tags
• Logstash 7 GROK Filter plugin ‘break_on_match’ appears to be broken

Photo by The Lazy Artist Gallery on StockSnap