When dealing with an exception stack trace we have to not only construct a separate match for our grok filter but also make sure that all lines will be treated as one entry.
What we are going to build
In this post I’ll show you how to:
- configure Filebeat to merge all lines of an exception stack trace into one entry;
- parse it it with Logstash with a custom pattern;
- clear tags from a false “failure tag”.
Make Filebeat read all stack trace lines as one entry
Filebeat reads an input file line by line. We have to explicitly tell it to treat a stack trace as a whole by using the multiline option:
If you are sending multiline events to Logstash, use the options described here to handle multiline events before sending the event data to Logstash. Trying to implement multiline event handling in Logstash (for example, by using the Logstash multiline codec) may result in the mixing of streams and corrupted data.https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline-examples
Below you’ll find my configuration file for Filebeat. It reads logs from the
all.log file, applies the
multiline plugin on those that match the patterns and sends everything to Logstash:
The Filebeat documentation contains useful examples of dealing with Java exceptions and the pattern I used is copied from there. It will merge lines starting with ‘
at‘ and ‘
Caused by‘ from the example input given below:
Read the Regular expression support docs if you want to construct your own pattern for Filebeat. They differ slightly from the Logstash patterns.
match and negate
The behaviour of
multiline depends on the configuration of those two options. The default value for the
negate option is
match I used ‘
after‘. As a result, matching lines are joined with a preceding line that doesn’t match (‘
Exception in thread "main“…’ is concatenated with all the following lines that match the pattern).
Add a match for exceptions to your Logstash configuration
logstash.conf file I need filters that will handle:
- a Java stacktrace:
- a regular Spring Boot log:
Because I don’t want to list all patterns in one
match section, every entry is being checked against both matches (I think the break_on_match is not working in this case). As a result the
_grokparsefailure tag will be added to all entries, even those that matched one of my patterns. I’m going to add my custom tags to solve this issue:
To remove the
_grokparsefailure tag I have to know that a particular entry was successfully matched by one pattern – the
spring_boot_log tag will be present in such a case. Therefore, I can safely delete the
_grokparsefailure tag for entries that have my custom tag.
Make sure that you have an output configured. It can be send to the STDOUT of the shell running Logstash:
I’m sending parsed logs to ElasticSearch and use ElasticHQ app to show you the results in a more readable way. You can see how the original
message with an exception was parsed on the screenshot below:
You can see the
exception part separated from the rest of the message as well as the
stacktrace added to the tags. There is also the
multiline flag added automatically.
- Handling Multiline Stack Traces with Logstash (if you don’t read logs with Filebeat)
- Using Multiple Grok Statements to Parse a Java Stack Trace
- Is there a way to tag for different grok matches?
- grok multiple messages and process them with different tags
- Logstash 7 GROK Filter plugin ‘break_on_match’ appears to be broken