Welcome to our technical blog, where we’ll be diving into the world of Graylog and how you can secure your Graylog Server with Transport Layer Security (TLS). As an admin or a tech-savvy user, you know the importance of protecting your Graylog Server and the logs it manages from unauthorized access. TLS is a robust security protocol that can encrypt the communication between your Graylog Server and its clients, ensuring that sensitive data, such as log messages and user credentials, are safeguarded from potential eavesdropping or tampering. In this blog, we’ll guide you through the step-by-step process of implementing TLS on your Graylog Server. So, let’s roll up our sleeves and get ready to secure your Graylog Server with TLS for enhanced data protection!
Graylog Server TLS/HTTPS
|NOTE: if you have any sidecars installed and connecting to a non HTTPS URL/address (e.g. HTTP), you will need to verify: |
This document will cover everything you need to know to successfully configure graylog to use HTTPS.
Chain of Trust
TLS Certificates are validated using a chain of trust. In order for your client, such as a web browser, to trust a certificate, such as one served by a web server, the web browser (as well as the Device or Computer itself) must trust the certificate that “signed” and issued the certificate.
Every device (computer, phone, etc) that connects to the internet has a list of certificates of Trusted Public Certificate authorities (CA). These are commonly referred to as Root Certificates. This allows these devices to trust any certificate that is issued by a Public Certificate Authority (CA).
There are also private certificate authorities, that can issue certificates, but requires installing one or more certificates on any device accessing resources that using a private certificate authority (CA) certificate. Example of private certificate authorities are Microsoft Active Directory Certificate Services servers that issue certificates for a corporate or enterprise network. Any device that is a member of the same Active Directory Domain can be configured to automatically trust any certificate issued from the private certificate authority (CA).
The most important difference Public certificate authority (CA) signed certs and Private certificate authority (CA) signed certs, is that you must manually import and configure the private CA root certs on a device in order for it to trust certificates issued from that private certificate authority (CA), where as the device will automatically, by default, trust Public certificate authority (CA) signed certificates.
How web traffic is encrypted/decrypted
In order for TLS encryption to function, a web server, such as Graylog’s Web Interface, must be configured to use 2 different certificates:
- Public Key
- Private Key
When a client, such as a web browser, connects to a web server using TLS/HTTPS, the web server sends the client its public key. The client can then use this public key to encrypt any traffic sent to the web server. In order to decrypt this traffic, the corresponding private key must be used. Only the web server has access to this private key.
Prerequisites and Requirements
- SSH client
- SFTP Client (such as FileZilla)
- SSH access to your Graylog server
- Root access to your Graylog server
- Corresponding Private key and Public key certificate files
- Openssl command line utilities (typically installed by default on linux devices)
Review and Validation
Configuring HTTPS/TLS for a webserver typically requires 2 files:
- Public Key File
- Private Key File
This section will cover how to verify you have the correct files in the correct format.
If you already have separate certificate files for your private key and public key, you can move on to the next section ‘Verify your PUBLIC key certificate file’.
If your certificate is a single file, such as a .pfx file, you will need to convert the certificate to the correct format. A .pfx certificate contains both the Public and Private keys.
If you wish to verify your .pfx , you can use the following openssl command. Replace filename.pfx with the applicable file path and name to your .pfx certificate file.
openssl pkcs12 -info -in filename.pfx
To extract separate public and private key files. Replace filename.pfx with the applicable file path and name to your .pfx certificate file.
Extract Key Pair File. You will be prompted to input the password used to encrypt the .pfx file.
openssl pkcs12 -in filename.pfx -nocerts -nodes -out keypair.key
Extract Private Key. You will be asked to input a password to encrypt the private key file.
openssl pkcs8 -in keypair.key -topk8 -out private.key
Extract Public Key. You will be prompted to input the password used to encrypt the .pfx file.
openssl pkcs12 -in filename.pfx -clcerts -nokeys -out public.text.pem
Remove extra text from extracted public key certificate
openssl x509 -in public.text.pem -out public.cert.pem
Extract Certificate Chain. You will be prompted to input the password used to encrypt the .pfx file.
openssl pkcs12 -in filename.pfx -cacerts -chain -nokeys -out public.chain.text.pem
Remove extra text from extracted certificate chain (public keys of all certificates in the chain that signed this cert, such as the Root CA certificate and any intermediate CA certifiactes)
openssl x509 -in public.chain.text.pem -out public.chain.pem
Combine public certificate and certificate chain files:
cat public.cert.pem public.chain.pem > public.pem
Verify your PUBLIC key certificate file
Certificate file must be in PEM format.
The public certificate must start with
-----BEGIN CERTIFICATE----- and end with -----END CERTIFICATE-----
It is common for the public key certificate file to contain more than one block of text that represents the certificate. This is referred to as the certificate chain, and included the public keys of all certificates.
- Root CA
- Intermediate CA
- Issued Certificate (e.g. mydomain.tld)
- Intermediate CA
You can use a Certificate Decoder tool, such as https://www.sslshopper.com/certificate-decoder.html , to validate the certificate.
You can only paste a single certificate into the text box for the above link.
Verify your PRIVATE key certificate file
The private key must start with
-----BEGIN ENCRYPTED PRIVATE KEY-----
If the private key starts with
-----BEGIN PRIVATE KEY----- (unencrypted PKCS#8) or -----BEGIN RSA PRIVATE KEY----- (PKCS#1),
You will need to convert the format of the certificate:
|You will be asked to input a password. You will need to input this password in the Graylog server.conf configuration file in a later step so make sure you store this password so you can access it later.|
openssl pkcs8 -in privkey.pem -topk8 -out privkey.pkcs8.pem
After completing the above step, rename the old and new private key files:
sudo mv privkey.pem privkey.pem.origsudo mv privkey.pkcs8.pem privkey.key
Prepare Cert Files
For the purposes of this document, we will store our public certificate and private key files in
Copy Certificate Files to appropriate directory
On your graylog server, create this directory:
sudo mkdir -p /opt/graylog/tls/
Upload your certificate files to this directory (e.g. using SFTP via filezilla). Depending on your linux distribution, you may need to first upload your certificate files to your home directory and then copy the files to the above path.
sudo cp ~/public.pem /opt/graylog/tls/
sudo cp ~/private.key /opt/graylog/tls/
Set and Verify Owner and Permissions
NOTE: This section assumes that you installed Graylog via an Operating System Packages, and that the graylog-server process(es) are running as user graylog.
Setting the owner of these files, as well as setting the permissions of these files is important for 2 reasons:
- Graylog must have access to these files in order to serve pages via HTTPS
- To prevent unauthorized or unwanted access to your certificate’s private key file
# Set files owner
sudo chown graylog:graylog /opt/graylog/tls/*
# set permissions of certificate private key so no other user can read the contents
# this allows ONLY the owner of the file to read it
sudo chmod 600 /opt/graylog/tls/private.key
# set permissions of certificate public key file
# this allows anyone to read the file but only the owner to edit/modify it
sudo chmod 644 /opt/graylog/tls/public.pem
Java Key Store
As discussed in the introduction section above, in order for a device or web client to trust a TLS certificate, it must trust the Root Certificates that signed and issued it. For java applications, such as Graylog, the java key store is used.
In order to be sure that Graylog fully trusts a configured TLS certificate we must complete the following tasks. The code block below includes commands for these tasks.
- Make a copy of your existing java root certificates store (cacerts) (JKS = Java Key Store)
NOTE: Graylog now includes a “bundled” JDK, which means its possible to install graylog without instaling JDK. You will need to copy the cacerts file from one of 2 places, depending on if you do or do not have OpenJDK installed.
To see if you have an OpenJDK install:
If this returns a value, then you DO have OpenJDK installed. If there is no folder, you will get an error such as ls: cannot access ‘/usr/lib/jvm/’: No such file or directory.
# find the current home path for java and save as a variable
tmpjavahome=$(java -XshowSettings:properties -version 2>&1 > /dev/null | grep 'java.home' | grep '/.*' -o)
# make a copy of the default java cacerts file
cp $tmpjavahome/lib/security/cacerts /etc/graylog/graylog.jks
IF the above ls command does NOT return a value, use the bundled JDK cacerts file
sudo cp /usr/share/graylog-server/jvm/lib/security/cacerts /etc/graylog/graylog.jks
Set owner of file to graylog user
sudo chown graylog:graylog /etc/graylog/graylog.jks
- Import the public certs from your Private Certificate Authority (CA)
NOTE: certificates MUST be in PEM format (Base64 ASCII) (NOT DER binary)
Use openssl to convert a der format to pem format:
sudo openssl x509 -inform der -in enterpriseRootCA.cer -out enterpriseRootCA.pem
Import Root and Intermediate certificates
sudo keytool -importcert -keystore /etc/graylog/graylog.jks -storepass changeit -alias cachain -file /etc/graylog/enterpriseRootCA.pem
Repeat the above step for all root and intermediate CAs. For example, if you have a private Root CA that issues certs to a sub CA, add both the Root CA and Sub CA public certs to the above JKS
- Once your Java Key Store (JKS) file has the required certificates imported, you can configure graylog to use it as a trustStore.
- Red Hat/CentOS
See also Default File Locations.
Add the following to the first line starting with GRAYLOG_SERVER_JAVA_OPTS:
Which should look like:
GRAYLOG_SERVER_JAVA_OPTS="-Xms8g -Xmx8g -XX:NewRatio=1 -server -XX:+ResizeTLAB -XX:-OmitStackTraceInFastThrow -Djavax.net.ssl.trustStore=/etc/graylog/graylog.jks"
|NOTE: Java/JVM heap size is also declared here using the -Xms AND -Xmx arguments. You may have different values.|
Configure Graylog Server.conf
Now that you have valid certificate files, and they are prepared and located in the appropriate directory, you can modify your Graylog server server.conf configuration file to configure HTTPS/TLS.
By default, this file is located at:
# IP and Port that the web interface will bind to and listen on # By default graylog binds to TCP 9000 # NOTE: graylog will fail to start unless # `AmbientCapabilities=CAP_NET_BIND_SERVICE` is added to # `graylog-server.service` # See next section in this document. http_bind_address = 0.0.0.0:443 # Fully qualified domain name (FQDN) that will be used to accesss # this graylog server's web interface http_publish_uri = https://subdomain.domain.tld # Enable TLS http_enable_tls = true # The X.509 certificate chain file in PEM format to use for securing the HTTP interface. http_tls_cert_file = /opt/graylog/tls/public.pem # The PKCS#8 private key file in PEM format to use for securing the HTTP interface. http_tls_key_file = /opt/graylog/tls/private.key # The password to unlock the private key used for securing the HTTP interface. # NOTE: do NOT enclose the password in double quotes (") http_tls_key_password = thePasswordGoesHere
Configure Graylog service to allow listening on ports <1024
By default, non root linux users cannot bind to network ports lower than 1024. This means that if graylog is installed via an Operating System Packages, graylog will execute as user graylog and will be unable to start when its web interface is configured to use TCP 443 (HTTPS).
To address this, you need to add to graylog’s service file:
To find the source of the graylog server service file:
systemctl status graylog-server
The default path is /lib/systemd/system/graylog-server.service which should be reflected in the output of the above command:
root@graylog-ubu:~# systemctl status graylog-server graylog-server.service - Graylog server Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled)
Add required line to Graylog Server service file and perform a systemctl daemon-reload:
sudo sed -i '/^LimitNOFILE=64000.*/a AmbientCapabilities=CAP_NET_BIND_SERVICE' /usr/lib/systemd/system/graylog-server.service
sudo systemctl daemon-reload
Now you can restart Graylog-server, which once started, should be accessible via HTTPS.
sudo systemctl restart graylog-server
- Using HTTPS (docs.graylog.com)