How-To Guide: Securing Graylog with TLS

Welcome to our technical blog, where we’ll be diving into the world of Graylog and how you can secure your Graylog Server with Transport Layer Security (TLS). As an admin or a tech-savvy user, you know the importance of protecting your Graylog Server and the logs it manages from unauthorized access. TLS is a robust security protocol that can encrypt the communication between your Graylog Server and its clients, ensuring that sensitive data, such as log messages and user credentials, are safeguarded from potential eavesdropping or tampering. In this blog, we’ll guide you through the step-by-step process of implementing TLS on your Graylog Server. So, let’s roll up our sleeves and get ready to secure your Graylog Server with TLS for enhanced data protection!

Graylog Server TLS/HTTPS

Introduction

This document will cover everything you need to know to successfully configure graylog to use HTTPS. When configuring TLS there are a number of steps. These steps can result in an error if the store is not properly configured.

A common error is:

pkix path building failed: sun.security.provider.certpath.suncertpathbuilderexception: unable to find valid certification path to requested target

While not required, it is helpful to understand the basics of Public key infrastructure and Public-key cryptography.

Chain of Trust

TLS Certificates are validated using a chain of trust. In order for your client, such as a web browser, to trust a certificate, such as one served by a web server, the web browser (as well as the Device or Computer itself) must trust the certificate that “signed” and issued the certificate.

Every device (computer, phone, etc) that connects to the internet has a list of certificates of Trusted Public Certificate authorities (CA). These are commonly referred to as Root Certificates. This allows these devices to trust any certificate that is issued by a Public Certificate Authority (CA).

There are also private certificate authorities, that can issue certificates, but requires installing one or more certificates on any device accessing resources that using a private certificate authority (CA) certificate. Example of private certificate authorities are Microsoft Active Directory Certificate Services servers that issue certificates for a corporate or enterprise network. Any device that is a member of the same Active Directory Domain can be configured to automatically trust any certificate issued from the private certificate authority (CA).

The most important difference Public certificate authority (CA) signed certs and Private certificate authority (CA) signed certs, is that you must manually import and configure the private CA root certs on a device in order for it to trust certificates issued from that private certificate authority (CA), where as the device will automatically, by default, trust Public certificate authority (CA) signed certificates.

How web traffic is encrypted/decrypted

In order for TLS encryption to function, a web server, such as Graylog’s Web Interface, must be configured to use 2 different certificates:

• Public Key
• Private Key

When a client, such as a web browser, connects to a web server using TLS/HTTPS, the web server sends the client its public key. The client can then use this public key to encrypt any traffic sent to the web server. In order to decrypt this traffic, the corresponding private key must be used. Only the web server has access to this private key.

Prerequisites and Requirements

• SSH client
• SFTP Client (such as FileZilla)
• SSH access to your Graylog server
• Root access to your Graylog server
• Corresponding Private key and Public key certificate files
• Openssl command line utilities (typically installed by default on linux devices)

Review and Validation

Configuring HTTPS/TLS for a webserver typically requires 2 files:

• Public Key File
• Private Key File

This section will cover how to verify you have the correct files in the correct format.

PFX Certificate

If you already have separate certificate files for your private key and public key, you can move on to the next section ‘Verify your PUBLIC key certificate file’.

If your certificate is a single file, such as a .pfx file, you will need to convert the certificate to the correct format. A .pfx certificate contains both the Public and Private keys.

If you wish to verify your .pfx , you can use the following openssl command. Replace filename.pfx with the applicable file path and name to your .pfx certificate file.

openssl pkcs12 -info -in filename.pfx

Private Key

To extract separate public and private key files. Replace filename.pfx with the applicable file path and name to your .pfx certificate file.

Extract Key Pair File. You will be prompted to input the password used to encrypt the .pfx file.

openssl pkcs12 -in filename.pfx -nocerts -nodes -out keypair.key

Extract Private Key. You will be asked to input a password to encrypt the private key file.

openssl pkcs8 -in keypair.key -topk8 -out private.key

Public Key

Extract Public Key. You will be prompted to input the password used to encrypt the .pfx file.

openssl pkcs12 -in filename.pfx -clcerts -nokeys -out public.text.pem

Remove extra text from extracted public key certificate

openssl x509 -in public.text.pem -out public.cert.pem

Extract Certificate Chain. You will be prompted to input the password used to encrypt the .pfx file.

openssl pkcs12 -in filename.pfx -cacerts -chain -nokeys -out public.chain.text.pem

Remove extra text from extracted certificate chain (public keys of all certificates in the chain that signed this cert, such as the Root CA certificate and any intermediate CA certifiactes)

openssl x509 -in public.chain.text.pem -out public.chain.pem

Combine public certificate and certificate chain files:

cat public.cert.pem public.chain.pem > public.pem

Verify your PUBLIC key certificate file

Certificate file must be in PEM format.

The public certificate must start with

-----BEGIN CERTIFICATE----- and end with
-----END CERTIFICATE-----

It is common for the public key certificate file to contain more than one block of text that represents the certificate. This is referred to as the certificate chain, and included the public keys of all certificates.

For example:

• Root CA
  • Intermediate CA
    • Issued Certificate (e.g. mydomain.tld)

You can use a Certificate Decoder tool, such as https://www.sslshopper.com/certificate-decoder.html , to validate the certificate.

You can only paste a single certificate into the text box for the above link.

Example output:

Verify your PRIVATE key certificate file

Graylog currently only supports PKCS#8 private keys with PEM encoding.

The private key must start with

-----BEGIN ENCRYPTED PRIVATE KEY-----

If the private key starts with

-----BEGIN PRIVATE KEY----- (unencrypted PKCS#8) or
-----BEGIN RSA PRIVATE KEY----- (PKCS#1),

You will need to convert the format of the certificate:

openssl pkcs8 -in privkey.pem -topk8 -out privkey.pkcs8.pem

After completing the above step, rename the old and new private key files:

sudo mv privkey.pem privkey.pem.origsudo mv privkey.pkcs8.pem privkey.key

Prepare Cert Files

For the purposes of this document, we will store our public certificate and private key files in

/opt/graylog/tls/.

Copy Certificate Files to appropriate directory

On your graylog server, create this directory:

sudo mkdir -p /opt/graylog/tls/

Upload your certificate files to this directory (e.g. using SFTP via filezilla). Depending on your linux distribution, you may need to first upload your certificate files to your home directory and then copy the files to the above path.

E.g.

sudo cp ~/public.pem /opt/graylog/tls/

sudo cp ~/private.key /opt/graylog/tls/

Set and Verify Owner and Permissions

NOTE: This section assumes that you installed Graylog via an Operating System Packages, and that the graylog-server process(es) are running as user graylog.

Setting the owner of these files, as well as setting the permissions of these files is important for 2 reasons:

1. Graylog must have access to these files in order to serve pages via HTTPS
2. To prevent unauthorized or unwanted access to your certificate’s private key file

# Set files owner

sudo chown graylog:graylog /opt/graylog/tls/*

# set permissions of certificate private key so no other user can read the contents

# this allows ONLY the owner of the file to read it

sudo chmod 600 /opt/graylog/tls/private.key

# set permissions of certificate public key file

# this allows anyone to read the file but only the owner to edit/modify it

sudo chmod 644 /opt/graylog/tls/public.pem

Java Key Store

As discussed in the introduction section above, in order for a device or web client to trust a TLS certificate, it must trust the Root Certificates that signed and issued it. For java applications, such as Graylog, the java key store is used.

In order to be sure that Graylog fully trusts a configured TLS certificate we must complete the following tasks. The code block below includes commands for these tasks.

1. Make a copy of your existing java root certificates store (cacerts) (JKS = Java Key Store)

NOTE: Graylog now includes a “bundled” JDK, which means its possible to install graylog without instaling JDK. You will need to copy the cacerts file from one of 2 places, depending on if you do or do not have OpenJDK installed.

To see if you have an OpenJDK install:

ls /usr/lib/jvm/

If this returns a value, then you DO have OpenJDK installed. If there is no folder, you will get an error such as ls: cannot access ‘/usr/lib/jvm/’: No such file or directory.

# find the current home path for java and save as a variable

tmpjavahome=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | grep '/.*' -o)

# make a copy of the default java cacerts file

cp $tmpjavahome/lib/security/cacerts /etc/graylog/graylog.jks

IF the above ls command does NOT return a value, use the bundled JDK cacerts file

sudo cp /usr/share/graylog-server/jvm/lib/security/cacerts /etc/graylog/graylog.jks

Set owner of file to graylog user

sudo chown graylog:graylog /etc/graylog/graylog.jks

Import Certs

2. Import the public certs from your Private Certificate Authority (CA)

NOTE: certificates MUST be in PEM format (Base64 ASCII) (NOT DER binary)

Use openssl to convert a der format to pem format:

sudo openssl x509 -inform der -in enterpriseRootCA.cer -out enterpriseRootCA.pem

Import Root and Intermediate certificates

sudo keytool -importcert -keystore /etc/graylog/graylog.jks -storepass changeit -alias cachain -file  /etc/graylog/enterpriseRootCA.pem

Repeat the above step for all root and intermediate CAs. For example, if you have a private Root CA that issues certs to a sub CA, add both the Root CA and Sub CA public certs to the above JKS

Configure Graylog

3. Once your Java Key Store (JKS) file has the required certificates imported, you can configure graylog to use it as a trustStore.

Edit the JVM settings using a text editor, such as Vim or Nano (Be sure to use sudo when launching your text editor to ensure you can save changes to the file.)

• Debian/Ubuntu
  • /etc/default/graylog-server
• Red Hat/CentOS
  • /etc/sysconfig/graylog-server

See also Default File Locations.

Add the following to the first line starting with GRAYLOG_SERVER_JAVA_OPTS:

-Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks

Which should look like:

GRAYLOG_SERVER_JAVA_OPTS="-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks"

Configure Graylog Server.conf

Now that you have valid certificate files, and they are prepared and located in the appropriate directory, you can modify your Graylog server server.conf configuration file to configure HTTPS/TLS.

By default, this file is located at:
/etc/graylog/server/server.conf

Edit the graylog server configuration file using a text editor, such as Vim or Nano. (Be sure to use sudo when launching your text editor to ensure you can save changes to the file.)

# IP and Port that the web interface will bind to and listen on
#     By default graylog binds to TCP 9000
#     NOTE: graylog will fail to start unless
#         `AmbientCapabilities=CAP_NET_BIND_SERVICE` is added to
#         `graylog-server.service`
#         See next section in this document.
http_bind_address = 0.0.0.0:443
# Fully qualified domain name (FQDN) that will be used to accesss
#     this graylog server's web interface
http_publish_uri = https://subdomain.domain.tld
# Enable TLS
http_enable_tls = true
# The X.509 certificate chain file in PEM format to use for securing the HTTP interface.
http_tls_cert_file = /opt/graylog/tls/public.pem
# The PKCS#8 private key file in PEM format to use for securing the HTTP interface.
http_tls_key_file = /opt/graylog/tls/private.key
# The password to unlock the private key used for securing the HTTP interface.
# NOTE: do NOT enclose the password in double quotes (")
http_tls_key_password = thePasswordGoesHere

Configure Graylog service to allow listening on ports <1024

By default, non root linux users cannot bind to network ports lower than 1024. This means that if graylog is installed via an Operating System Packages, graylog will execute as user graylog and will be unable to start when its web interface is configured to use TCP 443 (HTTPS).

To address this, you need to add to graylog’s service file (Versions Prior to V5.2 only. Now included in systemd service):

AmbientCapabilities=CAP_NET_BIND_SERVICE

To find the source of the graylog server service file:

systemctl status graylog-server

The default path is /lib/systemd/system/graylog-server.service which should be reflected in the output of the above command:

root@graylog-ubu:~# systemctl status graylog-server graylog-server.service - Graylog server 
Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)

Add required line to Graylog Server service file and perform a systemctl daemon-reload:

sudo sed -i '/^LimitNOFILE=64000.*/a AmbientCapabilities=CAP_NET_BIND_SERVICE' /usr/lib/systemd/system/graylog-server.service

sudo systemctl daemon-reload

Now you can restart Graylog-server, which once started, should be accessible via HTTPS.

sudo systemctl restart graylog-server

References/Resources

• Using HTTPS (docs.graylog.com)