Today we are releasing the first release candidate of Graylog v3.0.
This release brings a whole new content pack system, an overhauled collector sidecar, reporting capabilities, improved alerting with greater flexibility, a new script alert notification plugin, support for Elasticsearch 6.x, a preview version of an awesome new search page called Views, and tons of other improvements and bug fixes.
We are looking forward to your feedback on the new features and welcome any bug reports as we work on shipping the final GA release of Graylog 3.0.
Please read on for detailed descriptions of each feature.
DOWNLOAD LINK
Download v3.0 RC 1 here.
Please report bugs and any other issues in our GitHub issue tracker. Thank you!
NEW: VIEWS
In a single screen, collect all the information you need for a specific analysis you perform on a regular basis.
Built around the familiar layout of the Graylog search page, Views extend the familiar concept in many important ways. The most visible change is that you can rearrange and resize graphs, maps, statistical widgets that you create as part of your analysis just like you can on Dashboards. Those widgets are built using the Aggregation Builder widget, which allows for a dramatically more flexible approach when slicing and dicing your log data.
When the time comes to share your results, you can save the entire state of a View, share it with users in Graylog or simply pick a previously saved View to re-run it. What’s more, you can define parameters to be used in queries, so that your analysts don’t waste their valuable time typing in long queries: a single screen asking for a value is all they need to deal with. For example, simply enter a new IP address to run a pre-made analysis.
One less visible, but no less important, difference to the Graylog search page is that you can choose any value and use it to execute another saved View, thus allowing users to quickly run more detailed, pre-made analyses without copying and pasting many different search queries.
Views also support multiple queries alongside each other, so you can quickly gain an overview of what is happening in your environment.
We are actively working on extending the capabilities and integration of Views to provide you with the best value. Over time, we will build and ship default views most commonly used in IT, DevOps, and Security.
Let us know what you’d like to have included in our GitHub issue tracker.
NEW: REPORTS
Schedule dashboards to be delivered on a regular basis by email for review and to meet compliance procedures.
Using existing dashboard widgets from any number of dashboards you can graphically build reports that capture important information about your systems. They can be emailed to multiple recipients automatically, whether they are users in Graylog or external email addresses. You can also download a report from the Graylog console at any time.
Customize each report with a cover page, including your corporate colors and logo, should you want to automatically deliver reports to your own customers.
Configured reports can easily be shared using the new content packs, too.
UPDATED: ALERTS
Alert conditions can further filter messages for more precise control about with data is processed.
Before this release, all alert conditions applied to all messages in a stream. The problem with that was that you had to create another stream for each subset of messages you wanted to alert on.
For example, if you had a stream called “Windows EventLogs” and you wanted to trigger an alert whenever a message with Event ID 1102 (The audit log was cleared) came in, you’d have to create a new stream “Windows EventLogs: Audit Log Cleared” and then add a message count condition on that stream.
Graylog now has a new Query parameter for all alert conditions that lets you specify a search query to narrow down the alert result set. You can now add the alert condition to your global EventLog stream and specify a query like “event_id:1102”. The query language is the same as for all other searches, meaning you can use range operators, wildcards, and AND and OR operators.
NEW: SCRIPT ALERT NOTIFICATION
Call a custom script with the content of alert notifications to integrate with third-party services.
In cases where you need to interface with an internal or unsupported external service, or have specific requirements on how the alert notifications should look, Graylog Enterprise now gives you the ability to call a script or program in addition to the default notification handlers.
You can define the arguments passed to your script as well as receive the entire alert context, including the backlog, as a JSON object and perform any necessary action.
UPDATED: SIDECAR
Sidecar has been re-architected so you can now use it with any log collector out there.
The most important change is moving the collector-specific parts out of Sidecar, making it a pure process and file manager that can be controlled from within the Graylog web interface. This allows you to use any kind of collector that can be configured with a file, even unconventional ones like sysmon or osquery.
In addition to the already-supported Filebeat, Graylog has added support for three more common collectors in v3.0:
– Topbeat to collect system metrics like CPU and RAM usage
– Auditbeat to collect information about user activity and file integrity
– Packetbeat to collect network packets directly from the wire. For example, DNS requests or ICMP packets for threat hunting and network security.
In the new Sidecar UI, Graylog is able to display more status and error reporting information to make it easier to troubleshoot deployments.
UPDATED: CONTENT PACKS
Content Packs now support all stable Graylog features and can use parameters for almost any configuration element.
Content Packs have been completely overhauled to support all kinds of entities in Graylog. They are now versioned and can be customized by creating parameters for configuration settings that can be set when installing a content pack version.
For example, when including an input in a content pack, you can now prompt the user installing the content pack in a new cluster for the port that input should bind to. Graylog allows you to customize practically every configuration settings this way, making it possible to create Content Packs that can be adapted without having to edit them manually.
Because they are versioned it becomes possible to track, and if need be, roll back changes to Graylog configurations.
UPDATED: PIPELINE RULES
Improved rule code editing.
We fixed a couple of usability issues with the Pipeline source code editor, including support for undo/redo and filtering the function reference by name and description.
Additionally, Graylog 3.0 adds a number of important functions to the core:
- Added support for testing the data type of a value, for each built in type, which makes writing type conversion rules much simpler. The supported types are: boolean, number, double, long, string, collection, map, list, date, period, IP, JSON, URL.
- Various string functions: replace, regex_replace, starts_with, ends_with, join.
- URL encoding and decoding
- Handling JSON values to add them to a message without having to list the field names.
OTHER IMPROVEMENTS
- Graylog’s API has supported token authentication for multiple versions now, but previously there was no UI to configure them. Since the sidecar now uses them by default, Graylog now exposes UI for managing tokens per user.
- Creating Grok patterns could be a challenge, so Graylog now ships with an extended editor for them, supporting both lookup existing patterns as well as testing the overall pattern with sample data.
- A new search decorator that can render URLs as direct HTTP links in search results. Graciously contributed by GitHub user pbr0ck3r.
- The input list is now filterable.
NOTABLE CHANGES
- Graylog 3.0 drops support for Elasticsearch versions before 5.6.x. We recommend using the latest 6.x version.
- The previously deprecated Drools support has been removed. It was most often used to implement blacklisting of messages, which can be done in a more scalable way using Pipelines.
- Filesystem related settings have been updated to use a more consistent naming scheme and to allow plugins to have a more reliable way to refer to a writable path. Please check the upgrading notes for details if you customized these settings.
- HTTP related settings have been simplified, this applies mostly to deployments that use a reverse proxy in front of Graylog. Please refer to the upgrading notes.
- A few plugins, which were shipped by default, are now part of Graylog Core: Pipelines, CEF, Netflow and Beats inputs, Graylog Sidecar as well as the map widget.
- Improves Netflow compatibility with Fortigate producers.
