The inherent limitations of most log managers and the need to work within the constraints of your current hardware may force your enterprise to make some hard choices. Less useful data may be left unchecked, old information will eventually get deleted, and the amount of data that is accessed in real-time is sacrificed to reduce excess workload.
But Graylog isn’t your average log manager. Its advanced functionality and fully scalable model allow you to avoid picking only the most necessary and vital data. Smart use of its plugins and functions can bring in terabytes of data across multiple data sources without having to renounce anything you may find useful later.
SAVING SPACE AND HARDWARE THROUGH ARCHIVING
A standard function of every log management platform out there is the automatic deletion of older messages. Usually, after a certain period of time you set up, all messages get removed from the system to prevent storage size from growing too much. However, you may never know when that piece of data may prove to be useful, even years later. Having to painfully choose whether you want to keep a message stored or delete it to curb costs is not just “not ideal”, it’s plainly immoral. This is especially true when a lot of apparently non-relevant data must be stored for unnecessarily long periods of time due to compliance requirements like HIPAA or PCI.
Graylog’s Archiving functionality is a smart yet ingenious solution to keep all this extra data stored without having to scale up your architecture to Silicon Valley levels. By configuring this function, Graylog archives all log messages automatically to compressed flat files on the local filesystem until you need to re-import them for future analysis, audit or compliance requirements. You can either choose to set Archiving as an automatic function every time Elasticsearch’s retention cleaning kicks in or enact it manually through the web interface or a REST call.
The web interface also makes it very simple to move any or all these archived messages back into the original Graylog folder so you can temporarily import and analyze them. In other words, they’re still available if you need them for working purposes. Since flat files are vendor agnostic, data can be easily and freely accessed at any time. Even better, you now have all the liberty to make whatever you want with these archived files. You can print them on paper (think about compliance documents), move them to cheaper storage systems, or even write them on tape.
OVERCOMING SYSLOG LIMITS WITH THE GRAYLOG EXTENDED LOG FORMAT (GELF)
The plain Syslog is an efficient yet rudimentary system to keep track of your data. Albeit useful, the classic Syslog has many shortcomings, starting from a burdensome lack of any form of compression. Its limited length (just 1024 bytes) means that huge payloads like backtraces can’t be handled. The categorization of Syslog messages is extremely broad, and is based on a simple separation of the software that generates, stores, or analyzes them. Data types are absent even when data is structured, so even understanding the difference between a string and a number may be a chore. The massive number of Syslog dialects out there make it impossible to parse all of them despite the rigidity of the RFCs.
The Graylog Extended Log Format (GELF) is a new log format that allows your enterprise to make sense of much more than just your machines’ or network’s system messages. A unique feature of the Graylog Enterprise version, the GELF can be implemented with all applications by making use of their libraries for logging from within. Every exception can be sent as a log message to the Graylog cluster through GELF using UDP to avoid any risk of connection loss or timeouts associated with TCP. Chunked GELF is also supported whenever you want to send UDP datagrams which are larger than 8192 bytes. However, you may always choose to use TCP as transport (or, in some cases, even HTTP), especially in high volume environments. GELF messages can be compressed with ZLIB or GZIP whenever you want to trade some CPU load to save network bandwidth, and their compression type is detected automatically.
If you want to know more about the GELF, please check our documentation.
AUDIT LOG – A LOG MANAGER FOR KEEPING TRACK OF ALL CHANGES
The last but not least plugin in our list is the Audit Log. By keeping track of all changes made by all users in the Graylog system, this function brings even more granularity to your analysis. All audit log entries can be filtered and exported, adding a new layer of refinement by recording all database state changes automatically.
Especially useful for security purposes, the audit log plugin records when and what resources were accessed providing documentary evidence of all the activities that may have affected a given procedure or operation. Built to be easily written in a non-intrusive way, the audit log can be accessed at any time through the web interface to help you search and filter for audit events to obtain an even clearer picture of any events.
