Time Zones: A Logger’s Worst Nightmare

When working with log messages, the log message’s timestamp must be accurate. Some problems that incorrect timestamps create include:

• Difficulty searching for log messages based on specific date/time
• Alerts not functioning properly

Log message timestamp issues are commonly caused by a mismatch of timezones between the log source (device sending the log) and log destination (device receiving the log, such as Graylog).

Time Zones and Graylog

For a Graylog cluster, time zones come into play in several ways:

• The timezone of your log sources
• The timezone of your Linux server’s operating system
• The timezone display settings for the user logged into Graylog

In order for log messages to be viewed with the correct timestamp and timezone, all of the above must be configured appropriately.

Log Sources

We strongly recommend configuring the timezone of any log source to send log messages with a timestamp in UTC. Many devices will default to this timezone.

Because not all log messages and not all date formats include the timezone offset, Coordinated Universal Time (UTC) should be used to prevent confusion about what timezone a timestamp is in and allow for consistent logging of timestamps regardless of how those logs are accessed or viewed. Consistent time stamps are crucial when correlating logs from multiple sources.

Timezone of the Linux Operating System where Graylog is receiving log messages

This applies to the Linux Operating system where the Graylog Input is running.

Some types of log sources and log formats, such as rfc3339, include a timezone aware timestamp.

Syslog has 2 different formats: rfc3164 and rfc5424 which is meant to replace rfc3164.

When log messages are sent to Graylog, Graylog needs to determine what timestamp to attribute to the log message. It does this in one of two ways:

1. 1. Using the timestamp contained in the log message
      1. IF the timestamp contains a timezone offset, Graylog uses that
      2. IF NOT, Graylog uses the timezone offset of the server operating system
   2. Using the date and time that the log message was received
      

Option one is more reliable and not prone to incorrect timestamps.

In order for option one to work correctly, especially for rfc3164 syslog messages, the timezone of the Linux operating system where Graylog is receiving log messages must match the timezone of the log source.

To view the timezone and offset on your Linux server, run the following command in a terminal:

date +”%Z %z”

This should output UTC +0000.

If the server timezone is NOT UTC, we recommend setting the timezone on any Linux server where Graylog is installed to UTC.

To set the timezone to UTC:

• Ubuntu, Debian, Redhat, CentOS:
  • sudo timedatectl set-timezone UTC
    

We also recommend configuring and enabling NTP to ensure the time on your server is always synchronized. Specifics on how to do this are out of scope for this article.

Timezone Display Settings for Graylog User

Each Graylog user can configure their display timezone. This changes what timezone is used to view log messages, but does not change the original timezone of log messages.

By default, Graylog displays all timestamps in UTC.

To change the Timezone:

1. Click the user/profile icon at the top right of the page
2. Click ‘Edit Profile’
3. Scroll down to the ‘Settings’ section
4. Use the ‘Time Zone’ drop down menu to choose the appropriate timezone
5. Click ‘Update Settings’

NOTE: the above does NOT apply for the default ‘admin’ user. The timezone for this user can only be set via the Graylog server server.conf configuration file using the ‘root_timezone’ property.

For example:

root_timezone = UTC

Common Issues

Messages are not viewable in Graylog, or are the timestamp is in the future

Source sends UTC, but Graylog server has a negative timezone offset (e.g. -6)

Message timestamps are in the past

Source sends with a timezone offset, but graylog server has no timezone offset (UTC)

My log timestamps are correct but the wrong time is shown in graylog

Timezone setting for Graylog web (set for user or configure in server conf for admin)

NOTE: Graylog 5.1 ads the ability to configure the timezone of Syslog inputs. This is a change in behavior where previous versions of graylog used the timezone of the server operating system where Graylog was installed and running.

You can configure the timezone of an existing syslog input via the inputs page (System / Inputs), finding the input, and choosing ‘Edit Input’ from the ‘More actions’ button. You can choose a timezone via the timezone select box at the bottom of the input configuration modal box:

Be sure to choose the timezone that matches the timezone of your log sources. For example, if your firewall is using Eastern Time (UTC-5), set the timezone of the syslog input to match.