GELF

The Graylog Extended Log Format (GELF) is a uniquely convenient log format created to deal with all the shortcomings of classic plain Syslog. This enterprise feature allows you to collect structured events from anywhere, and then compress and chunk them in the blink of an eye.

Why do you need GELF?

Moving from unstructured to structured data is a chore on its own, but making sense of an event by parsing logs may be an impossible task when data types are absent. Even telling the difference between a number and a string can be extremely hard with structured Syslogs since their message categorization is rudimental at best. Plain Syslogs are ridden with several limitations such as the 1024 bytes length that will leave very little space for payloads like backtraces, or the absence of any form of compression.

Let’s face the truth: using the classic Syslog to keep track of data and events is like chopping wood with an ax. You can do it, but it will take a lot more time and effort than using a chainsaw. Since we want to help you do your logging as easily as possible, we’ve invented our nifty “chainsaw” – the GELF. Logging all your libraries in virtually any programming language and Syslog dialect has never been so easy.

Every log message in GELF is a dict with the following fields:

host (the creator of the message)
timestamp
version
long and short version of the message
other custom fields you can freely configure on your own

Any application can easily implement the GELF from within through their libraries for logging. You can find a broad range of appenders and libraries for nearly all programming languages as well.The GELF is a new log format that will help your team make sense of a ton of information beyond just your network’s or machines’ system messages. Every exception can be identified as a log message and sent to the Graylog cluster. GELF events can be sent to Seq via UDP to be compressed and chunked, so even TCP-related connection problems or timeouts will never be a problem again.

Chunking and Compressing

UDP datagrams can’t be larger than 8192 bytes, which are usually enough for GZIP’d information. “Usually” is the keyword here – and we all know what happens when that piece of critical data that you need to send is just a little bit larger. You can use TCP, of course, but unless you’re working in a high volume environment this protocol is not devoid of risks. Timeouts, slow connections, and network issues can bring your entire application down, causing significant disruption and a constant risk of data loss.

The GELF fixes this just by letting you chunk your message – and it does it smoothly and effectively. Just prepend a byte header to a GELF message to chunk it now, and reassemble it later by checking its ID or sequence number. GELF libraries allow you to choose whether you want to transport your logs via TCP, UDP, and sometimes via HTTP. Even better, they support chunking transparently, so you will be automatically alerted if a message can’t be sent in a single datagram because it’s too big. You can also have the option to save a ton of network bandwidth in exchange for some additional CPU load. Just select if you want to send your messages uncompressed, GZIP’d or ZLIB’d, and the Graylog nodes will automatically detect the compression type.

If you want to know more about the GELF and how to make it work, just check our documentation section here.