We could not be more excited to announce the first feature complete beta release of Graylog v2.0.
It’s been a little over a year since our last major release (v1.0), and in that time there has been tremendous growth in the Graylog ecosystem. There are now more than 10,000 Graylog deployments globally and 20,000 downloads a month. Graylog Marketplace has over 140 contributed plugins, content packs, and add-ons since its launch five months ago.
We’ve already seen the v1.0 series of Graylog displacing expensive, incumbent legacy solutions all over the world, and we’re confident that the v2.0 series will enable even more use cases and bring Graylog into even more data centers out there.
In v2.0, you will find significant architectural improvements as well as tons of new features. We built this series based on feedback from countless conversations with users in person and online via our various community channels. We thank everyone on the Graylog team for doing an incredible job, and we especially thank the community for helping with this huge release milestone.
$ git diff –shortstat 1.3.4..master
2465 files changed, 120524 insertions(+), 80224 deletions(-)
Important: This is a beta release and intended for testing purposes only. We have had users deploy our betas in production before, but use at your own risk. We do not guarantee production level quality yet.
Now let’s dive in and check out what’s new:
The web interface is no longer a separate process
We are now serving the web interface directly from graylog-server, and we have removed the extra graylog-web-interface process. The entire web interface code base has been rewritten with modern react.js technology, and many bugs or shortcomings were removed during that process.
With this unification of Graylog’s web interface and server processes, plugins now have the ability to include user interface components. The Graylog web interface becomes extensible by plugins, opening up endless possibilities for customization. You now have the power to make use of everything that Graylog has to offer today, while extending it with both back-end and front-end features at the same time. Many of the new Graylog features are being written as plugins, so you can be sure that the workflow of writing plugins will be rock solid (we are still working on updating documentation).
We now support Elasticsearch v2.x
Go ahead and upgrade to Elasticsearch v2.x to take full advantage of all the new goodness. Previous versions of Elasticsearch are no longer supported and it is important to know that Elasticsearch does not support a downgrade. Test Graylog v2.0 before going to production, as there will be no easy rollback.
Live tail
The much requested tail -f feature enables you to see what is being written to Graylog in real time. You can now configure your latest search results to reload automatically in 1 second to 5 minute increments, or any other time increment you want. Follow your latest log activities, like errors and exceptions after a deployment of your application, without having to hit the reload button all the time.
Message processing pipeline
Graylog has always provided several ways of processing log messages, like extractors and drools, but sometimes it’s been a bit confusing to use or simply not flexible enough. We now provide you with a more powerful solution that suits all of your processing needs. The new Message Processing Pipeline allows greater flexibility in routing, blacklisting, modifying, and enriching messages as they flow through Graylog.
The Message Processing Pipeline allows you to write custom rules and combine these rules in different processing pipelines that can transform incoming messages in almost any way you want. It will also be possible to write plugins extending the set of functions you can use in rules, meaning that you will be able to extend the system to suit your specific needs. Please take into account that this is the first version of the processor and we are still working heavily on it, so things may break or not work exactly as expected. You can find some initial documentation about it here. This will be improved over the next few weeks.
MAP WIDGET
This release adds the long awaited map widget and GeoIP resolver plugin to Graylog. Now you can enrich your messages with GeoIP data and visualize the results on an interactive map.
Archiving
Graylog currently enables you to configure a retention period and automatically deletes older messages – this is to help you control the costs of storage in Elasticsearch. But we know it’s not ideal deciding between keeping less messages in Graylog or paying more for hardware. Additionally, many of our customers are required to store data for long periods of time due to compliance requirements like PCI or HIPAA.
With the new Archiving functionality, Graylog now enables you to archive log messages until you need to re-import them into Graylog for analysis. You can instruct Graylog to automatically archive log messages to compressed flat files on the local filesystem before retention cleaning kicks in and messages are deleted from Elasticsearch. Archiving also works through a REST call or the web interface if you don’t want to wait until retention cleaning to happen. We think flat files are great in this scenario, because they are vendor agnostic so you will always be able to access your data.
You can then do whatever you want with the archived files: move them to cheap storage, write them on tape, or even print them out if you need to! If you need to search through archived data in the future, you can move any selection of archived messages back into the Graylog archive folder, and the Graylog web interface will enable you to temporarily import the archive so you can analyze the messages again in Graylog.
Archiving will be Graylog’s first commercial feature, meaning that it’s considered an add-on to the open source core that users will have to pay to use. Read more about the why and how in this blog post. The beta version is free for you to try out today. The beta license will expire shortly after the v2.0 GA release is out if you have not obtained a commercial license key.
Collector sidecar
We want to make your experience sending messages to Graylog as easy and flexible as possible. That is why we are introducing the new Collector Sidecar. The Sidecar runs next to your favorite log collector (like fluentd or nxlog) and configures it for you. Enjoy central configuration from the Graylog web interface, as well as no more tinkering with configuration files.
Those of you sending logs to Graylog through NXLog can start using the Collector Sidecar to manage its configuration from within the Graylog Web Interface. This first version only supports NXLog with a file input and a Windows EventLog input, but we will extend the Collector Sidecar to support multiple input types and log collector backends in the future (e.g. rsyslog).
The Graylog Collector Sidecar runs on Linux, Windows and Mac OS X. You can download a DEB package and a Windows installer from the repository release page for now. We have also listened to user feedback and wrote it in Go instead of Java. This means you’ll download a native binary instead and do not have to install any runtime dependencies like a JVM.
Streams filter
Some of you manage tens or hundreds of streams, and sometimes it’s difficult to find the stream you’re looking for among all the options. We have added a new search filter on the Streams page that lets you find streams by title or description.
Search surrounding messages
The new Show Surrounding Messages feature, one of our most requested features, allows you to investigate the events surrounding a given log message that was logged in the same context. This helps provide more context around a search result. What happened before and after this event? What else was happening on this host at the same time? Think of this like a grep -C [X] to find surrounding lines of a known line.
Query range limit
This adds a configuration option to limit the time range for searches. If you manage a team, some of your users on Graylog systems might be unintentionally overloading your Elasticsearch clusters by executing resource-intensive searches over an extended timeframe. If you set the query range limit to 1 month, for example, no one will be able to search for logs that are older than one month.
Go to System/Configurations in the navigation to configure your query range limit.
Configurable query ranges
The relative time range options for search in Graylog have always been fixed, and the maximum fixed time range was up to the last 30 days. But we had users who wanted to execute searches over longer time frames or other customized time frames. With the introduction of configurable query ranges, you can now change or delete the default values and also add new ones.
Go to System/Configurations in the navigation to configure your default time range options.
Tons of bugfixes and improvements
We touched around 200,000 lines of code, closed more than 200 issues on GitHub and squished a lot of bugs we found on the way. The list is too long to include here, so we encourage you to simply give the beta a try. If you find a bug, please report it using the community channels listed below.
Additionally, if you were using the previous alpha release, here is a list of bugfixes since v2.0-alpha.5:
- Fix build issue with maven. Graylog2/graylog-server2#1907 (Thanks @gitfrederic)
- Fix username in REST API access logs. Graylog2/graylog-server2#1815 Graylog2/graylog-server2#1918 (Thanks @mikkolehtisalo)
- Fix alert annotations in message histogram. Graylog2/graylog-server2#1921
- Fix problem with automatic input form reload. Graylog2/graylog-server2#1870 Graylog2/graylog-server2#1929
- Fix asset caching. Graylog2/graylog-server2#1924 Graylog2/graylog-server2#1930
- Fix issue with cursor jumps in the search bar. Graylog2/graylog-server2#1911
- Fix import of Graylog 1.x extractors. Graylog2/graylog-server2#1831 Graylog2/graylog-server2#1937
- Field charts will now use the stream and time range of the current search. Graylog2/graylog-server2#1785 Graylog2/graylog2-web-interface#1620 Graylog2/graylog2-web-interface#1618 Graylog2/graylog2-web-interface#1485 Graylog2/graylog-server2#1938
- Improve browser validations. Graylog2/graylog-server2#1885
- Fix Internet Explorer support. Graylog2/graylog-server2#1935
- Fix issue where a user was logged out when accessing an unauthorized resource. Graylog2/graylog-server2#1944
- Fix issue with surrounding search. Graylog2/graylog-server2#1946
DOWNLOAD IT NOW
Graylog v2.0-beta.1 can be downloaded from here.
Our virtual appliance in OVA format has also been updated for this release.
The operating system packages for v2.0-beta.1 are available in our repositories. See our documentation for details.
Docker images are available on Docker Hub.
UPGRADING FROM PREVIOUS V2.0-ALPHA.X RELEASES
You can just replace your existing installation. There were no database or index schema changes so a simple stop, upgrade, start procedure will work.
Please refer to the specific upgrade instructions of your installation method for details.
WE NEED YOUR FEEDBACK
We need feedback about what’s working and what’s broken in order to help everyone get the most out of Graylog v2.0. There are a variety of ways to provide feedback, all of which can be found on our community resources page:
- Report bugs and other issues in our GitHub graylog-server repo.
- Help with documentation in our GitHub documentation repo.
- Start a discussion in our Google Group mailing list.
- Or join the chatter on our #graylog Freenode IRC channel.
- New feature ideas are welcome in our product idea portal.
We’re super excited about releasing 2.0, and we value your feedback. So please go try out this release and let us know what you think!
