Graylog’s Archiving feature is used to store older log data on a separate storage such as an external HD, NAS, or Cloud storage. You can tell Graylog Enterprise and Graylog Security how long it should keep your stored data and write all messages of an index to flat files on a slow storage. You can apply any compression you want such as GZIP (default). Archives can be found in the interface underneath the enterprise drop down on the “Archives” submenu.
Managing Backends
As soon as you open it, go from the Overview tab to the Manage Backends one to select the backends where you are going to store your logs. Here you will find an overview of all your available backends, create new ones, or edit those currently available. By clicking on Edit you will find a title, a brief description, then you tell the base path where you will store your logs on that backend. For example, you can set a path based on variables like year-month-day if you want to create subfolders that will help you sort out your logs for later use. Just below the output base path, you can also see an example of how that will look when it’s stored.
The Configuration Tab
The Configuration tab is the panel where you can set all the archiving options you prefer. The first section the Backend one, and it’s used to choose the backend where do you want to store your log files from the ones you set up in the Manage Backends tab. Then, you can determine compression type – the default one is GZIP since it offers the best compression ratio, but you can choose the one you prefer. Be aware that some compression types are faster than other. The Restore Index Batch Size is used to determine how many logs per set are you going to re-import whenever you want to restore these archives. On the bottom you will find the streams of data that are actually imported in the backend of your choice. This way you can select, for example, to collect AWS and DNS logs in a specific storage, and firewall logs in a different one. This is important if you need to store a particular stream on a specific archive structure, or if you need to keep that data for a longer rotation period.
Indexing Your Data
Outside of those archives, the next thing you need to do is actually get to the Indices to set which log type is going to be collected. You can open up each index to find a complete overview of all the different indexes that you’ve collected so far, including those that are currently being written. You can find a lot of useful info such as the rotation period, the maximum number of indices, or the field type refresh interval. You can configure these indexes by clicking on the Edit key. Once in this new tab, you can set up a lot of variables to determine, for example, how long are you going to keep a set rotation time’s worth of logs before it’s deleted. You can also select the process after an archive is completed, so that it is not deleted if you don’t want to.
Archives Overview
The last tab of the archives feature is the Overview panel. Here you can see all the different archives that you have collected so far. You can find more details by opening up any one of these archives, such as where is the archive located at on the file system or if it is available. If you want to restore the archive, you can click on the Restore Index button to pull that data in again. You can also do that through our API.
