The following is a transcript:
Hardening Graylog is where we will help you encryptify your log supply. The real subtitle is, Why You Should Use TLS. It’s easier than you may think. Typical Graylog deployment looks something like this, where you have your clients and your web browsers, your users, log into Graylog’s server over, through HTTP, and you have log sources that are sending logs through protocol, through TCP, UDP, and GELF.
Connections to Encrypt
Then you have the connection between Graylog web server and the OpenSearch backend. You also have open search connections between other open search nodes. As you can see there, it’s all within a separate little network there. All these communications can be encrypted and authenticated by default. They are not, they are completely open and clear text.
Why do you want to encrypt? You think, okay, we want to trust that network. We have all our security controls in place. Everything should be fine where we use a VPN or we have strict policies. And you may think that TLS is too complicated. I can attest as a support engineer that we do get a lot of tickets on certificates.
Walking Through This
I will walk you through that. You will be experts to those three main reasons why you need to encrypt:
- Privacy. If everything is clear text It can be read by anyone listening.
- Trust, you need to be able to trust all the nodes in your deployment in Graylog open search log sources and Authentication because even if you do have encryption and trust you need to you only want certain Devices and certain users to be able to access that data for privacy problems. Clear texts, like I just said, user logins, completely clear incoming logs that can make contain sensitive data are completely clear and the connection to the open search backend is in the clear.
So, the solution of course is to encrypt and you can encrypt the web UI and inputs. To trust, there’s no by default. There’s no mechanism to verify that the server you are connecting to is in fact the server you’re intending to connect to. This includes server and application names, and it’s prone to man in the middle attacks or spoofing, and self signed certs are just gross and should not be used in an enterprise deployment.
Making Certificates
So the solution of course is to make certs. But just like self signed certs, you can’t just make certs and call it a day. It needs to be within an organized and planned, public key infrastructure. With PKI you can encrypt the Graylog API, which is also the web interface, the open search API. And then with this, you have a fully trusted certificate chain using either your internal CA, which a lot of people do, or use something public.
We have log sources that are sending in and are not authenticated. Any log source can send in to an open port. So we definitely want to address that log data in the open search backend without user authentication. There’s any user, including the Graylog application itself that would be able to read sensitive data stored in OpenSearch.
Inputs, OpenSearch, Graylog TLS
And so the solution here, not surprisingly, is more certificates. Inputs can use mutual TLS, which is basically certificates on TCP connections. And then there’s OpenSearch includes an internal user database with the OpenSearch security plugin. This is what a certificate looks like if you just Plain text file.
It looks very organized. I will personally ship this Graylog t shirt to anyone who can read this and tell me what the common name is without using open SSL, which is what we use to parse this. We feed that file through open SSL commands with a few other flags and it spits out the actual structured human readable data.
A few key lines in here, and there’s a lot of information, but a few key lines are, the subject and the issuer is the certificate authority that signed this certificate. In P. K. I. there’s levels of trust service certificates are always signed by and verified by a certificate authority, which you can tell an authority is the authoritative source of trust for that certificate.
The subject is the entire distinguished name or DN of the certificate itself. This one here in our example, it’s a wildcard certificate, which you can tell by the asterisk in the common name that covers all sub domains under the logfather.org domain. But there are other fields you can have as well.
There’s organization, organizational unit and Localities, but all those fields are not required. The only one that’s required is the common name or CN. The first page was just the top half because slides don’t really go vertical, but next part here has the extensions, which is as an extension on the X five or nine protocol.
Key Usage
But there’s a few key things here. There’s a key usage. Configuration where it’s this key of the certificate can only be used for certain things and in this case, we needed to say digital signature and key and cypher mint. There’s several others. We won’t get into those though. Extended key usage is just more of the same, more of the things that you just say certain things.
The CA flag needs to be false for all service certificates because they’re not CAs. But of course for CA certificates, then CA would be set to true. Lastly, the subject alternative name. Now, this may actually be the most important field in the entire certificate because modern browsers, modern as in probably since 2015, I think, all require, they base the verification on what’s in the subject alternative name or SAN and not the common name.
Basically, the subject alternative name lists all DNS and or IP addresses that this certificate is supposed to represent. In our case, we just need it to represent “asterisk.logfather.org” , but of course, if you have a very tightly controlled IP address layout, then you can put your IP addresses in there as well.
What We need!
All right, so enough talking. Let’s go. What do we need? We need application host names or IP addresses like I just mentioned. Before you deploy your PKI, you need to know all the host names that you’re going to be using for communication between the nodes. And this includes application names, so sometimes you can have a certificate that represents just the word Graylog, and as long as that’s the word you put in the browser or whatever connection string you’re using, that’s what the certificate will represent, and only what it’ll represent.
In order to use host names, for this, you need to have a well tuned DNS resolution, your clients that type in “graylog.company. com” their DNS server needs to point to your Graylog server that is presenting the Graylog dot company. com certificate. If it goes to somewhere else, then whatever’s on the other side is not.
It could be a name mismatch and it would not work. You need certificates with SANs. The SAN is definitely required in modern browsers. The CA certs need to be installed on all your clients. If you have an internal CA, the Let’s Encrypt in CA is by default trusted by pretty much every browser. But if you have an internal company CA, then the certificate of that CA needs to be installed on all clients so that those clients know to trust your CA.
Securing the Graylog Web Interface
A super nerdy Hitchhiker’s Guide reference that I won’t embarrass myself by going through. So let’s start. Securing the Graylog web interface. Step one, instill fear by demoing the dangers of cleartext connections. Okay, you do not want to, there’s If you’re watching this, you probably know this already, but we will show you exactly why clear text connections are no good.
And then to remedy this, we will configure Graylog with certificates and keys and then configure the Graylog internal Java key store. And I have a super slick demo here for you to watch, and I’ll just walk you through it here. First, I’m going to start with a TCP dump on port 9000. This is by default Graylog out of the box, no encryption, port 9000 being the web port.
And we’re going to watch what comes over the wire when I perform a login on the web interface.
Graylog Admin Password
Admin, this is a super duper secret secure password. And so I’m logging in and we will see the login. And then what do we see here? Username, admin, password, yabba dabba doo. So it’s completely clear text. This is exactly what goes across the wire. So obviously. This is, this needs to be fixed. Let’s configure the Java key store now.
So that way we have a place to install our certificates. Every Java based application uses a Java key store. Here is the certificate chain. This is basically the service certificate and the CA certificate. And we’re going to munch them together here. We’re opening up the Java key store. We encrypted the private keys that way it’s actually as a password.
Protected private key is one extra step never hurts the key tool command is what we use to interface with a key store and by default your shell doesn’t know where it exists and with Graylog 5. 0 we bundle all the java binaries with it so we have to go tell your shell hey go to the Graylog bundled version of the key tool command We can just type key tool without specifying that entire path there.
You see the path works now. And so here we’re going to add the certificate to the key store.
So here is the entry in the key store. And we’re going through just verifying that this is the right certificate, all the different parameters that we need. The subject alternative name, the alias is completely arbitrary, but it’s, it doesn’t have to match anything. But of course, it’s just best so you,
what entry it is. So, we said yes, we’re going to trust this crazy certificate and We’re going to double check. This is the actual entry that shows up. So, this is why aliases are important this is how you’re going to be able to tell what file it is the Certificate is now in the Java key store, which we all uses to as its internal database of configuration files.
The Graylog Server.conf File
The top one here is our Graylog server. conf Change in the bind address to port 443, which is optional by the way, you can leave it at 9000, but I just don’t like putting port numbers in URLs. Change, and by extension we need to change the publish URI, which if you didn’t change the port, you need to add it here.
- The publish URI is important because this is what your certificate needs to represent. Notice here, “williamtrelawnylab.logfather.org” matches the wildcard pattern. So, the certificate will work with this when I put this into my browser.
- Enable TLS true. TLS cert file and key file are just the paths to the files and then the password if you specified one for the private key.
Which, I don’t know if specifying a password of test is actually doing anyone any good. But this is a POC. And the Graylog server Java options, we just need to modify this because by default, our keystore didn’t exist before. So we have to tell Graylog where to go to find the keystore. We have the path there.
If you change the Bind address port to something to a privileged port under 1024 Then you need to give the Graylog server service a certain capability to bind to that kind of port That’s what the bottom part is. Let’s demo an encrypted login and see what we see So we’re going to do a TCP dump on port 443 and we’re going to grab for password because that’s what we know shows up when you log in you see there the URL is HTTPS and they’re It’s a fancy log pad.
But now you see we don’t see anything. Why is this? So, we grep for passwords. Let’s take off the password part and see what shows up. I’m going to do another login. Oh, this is a bunch of encrypted garbage. Of course, that was before the login. So, we’re going to do another login now. That’s what happens when you’re sniffing an encrypted login.
Securing Graylog Inputs
Next part, we need to secure the Graylog inputs. Another demo of incoming logs in clear text, which is going to be really bad. Configure inputs for encryption, because you have to tell them where to get the certificates, just like we did for the web server. And then we’re going to do a little demo on why encryption alone is not enough, and how to, and then how to remedy that.
GELF TCP Ingestion
Here we’re going to do another TCP dump on port 12201, which is the GELF port. GELF is really cool, by the way. You should be using it. The Graylog extended log format. It’s got everything you need.
And here, we’re going to send in a test log through curl, on the same host. So it shows up here on the, in the web UI, but we’re going to see here that, Oh yes, you see the, there we go. That’s in that scale right there, version host and short name. See real simple. You should all abandon this log now an encrypted log.
You can see everything. So imagine what could be in that log file that would be nasty for people too. So then we go into the input and we specify the path to the certificate, the key, enable TLS and the password. Now, let’s see what it looks, that same message looks like through an encrypted input. Do another TCP dump, same port because we didn’t change the port number.
GELF HTTP Ingestion
So we’re going to send in another log. This time, of course, changing the input to specify HTTPS. Because GELF is HTTP, it looks like it. I don’t know, it’s JSON. It’s good. You should use it. So here, all you can see is the William Trelawney lab.logfather.org. And that’s for the MTLS. That’s just to clarify.
That’s the only thing that comes across by the host name of the Graylog server that you’re saying log to, as you can see, the rest of it is more garbage, but encryption alone enough, right? Even if a stream of data is encrypted, we still face an issue with. Now anything and everything in the entire internet can be sending encrypted data on this port and our Graylog or our poor Graylog will be ingesting it.
All those super efficient extractors and pipelines that you’ve written will love you. So, we’re going to do another TCP dump.
And we’re getting, here we go. There we go. This actually is my actual laptop. It’s not the lab environment that the last log went through. As you can see by the super authoritative name: “willslaptop.home”. I’m going to send a log to the actual, over the internet, to my lab instance. And you see it shows up right there.
If this were a live demo, which I was going to do, you all could have sent logs to this instance and totally demolished the entire presentation. But it would have been a good demonstration of why authentication is just as important as encryption. You see the log is encrypted there, same thing. But again, it’s from a completely untrusted, unsolicited log source.
The special thing about inputs that I don’t feel like a lot of people know about, based on my customer interactions. So this is, this hasn’t really come up too much. You can use certificates to authenticate log sources and it’s just another, configuration option there below the other ones we had.
Certificate Chain and Circle of Trust
And you specify a certificate chain. That goes up to the mutual CA that your Graylog and your log sources can both trace their chain of trust back to. So, if you have an internal CA that signs your Graylog web server cert, as well as all your firewall log, firewall devices and all that, they share the same CA route, then only log sources that are signed by that CA.
Let’s of course, try another one. So, this time I’m going to tail the server dot log, because there’s going to be some special output we see there. So, we’re going to do the same thing for my personal laptop. We’re going to send another GELF log via curl. Curl is going to hang for a bit and we’re going to see that it spits out some SSL error.
And it’s very cryptic as all open SSL errors are. But basically it’s saying it failed and you don’t see the log there in Graylog. And the server log, but an associated SSL error, basically saying that the client did not present a certificate. So it dropped the log. So now we’ve prevented, now we’ve prevented some random log source from connecting him.
Securing OpenSearch
The final leg of securing Graylog is of course, securing the open search backend and open search. Conveniently ships for free with the open search security plugin, which we will heavily utilize for this. So first the demo, why we need to encrypt, which as you can see, you’re still asking, but you’re in the wrong place.
You haven’t been paying attention and then we do the open search configuration. Tell open search to use the security plugin and then configure the security plugin with encryption and authentication. We are going to start with a TCP dump. On port 9200, which is going to catch traffic from Graylog to open search.
And we’re going to send some log data in.
This is a log that contains credit card information. That’s my actual credit card. Feel free by some Graylog swag credit limit of 5 though. So I don’t know how much you’re going to get. You see the log comes in and log makes it in. We see the log in clear text being sent to the open search backend.
Anyone on your internal network, we saw a previous presentation about people focusing on perimeter security but neglecting internal security. If someone makes it through in a breach or is an internal actor, they could be sniffing this traffic just as easily. Even though your log came into Graylog encrypted.
It is, at this point, written to OpenSearch unencrypted. Of course, this is a major security hole. So first we’re going to enable the OpenSearch plugin. Setting plugins. Security disabled to: false. The double negative there pretty much means that it is enabled. Both the HTTP and the transport layer to use our OpenSearch certificates.
OpenSearch Nodes & TLS
They’re separated out because the HTTP refers to clients connecting into OpenSearch, such as Graylog. Then the transport section is for OpenSearch nodes talking to each other; because we definitely do want that to be encrypted as well. Then, finally at the bottom, we have the admin, the OpenSearch admin certificate.
Now, the CN here is largely irrelevant because it doesn’t have to map to actual username. But this is just basically, this is you telling OpenSearch that, Any client presenting this certificate can do admin things, user authentication.
There are basically four steps to configuring the OpenSearch plugin.
Four big steps, because the plugin is very big and there’s lots of intricacies that I’m not going to cover on the presentation. First step is:
- to generate a password hash using the convenient hash.sh script that ships with OpenSearch, the security plugin.
- Delete all the demo configuration inserts and OpenSearch will complain if you have demo certificates still present.
- Add the OS admin to user database. You give the admin user the admin role.
- And then we run the initialization script and punch it chewy into hyperspace.
- This is the security admin.sh script that ships with all this stuff ships with OpenSearch security plugin.
Okay, so here we go. We’re going to initialize the OpenSearch security plugin.
You see, we have a lot of options here:
- We need to run the command as the OpenSearch user. So that’s why the “sudo -U” and I’m setting the OpenSearch Java home path because your shell most likely doesn’t know where the OpenSearch Java binaries are. So we can set that in line as an environment variable there the way.
- We don’t need that environment variable to be set outside of this. The full path to security admin. sh. This CN stands for cluster name. Which is conveniently confusing for talking about common names and certificates, but two different, two different things.
- And then the cluster name has to match what is set in OpenSearch.yaml as your, I think it’s cluster name in there as well. Next a path to your ca cert, which is a ca cert chain, basically the chain of trust up to your ca. Server certificate key, of course, key password. CD is the configuration directory with all the security plugin files.
- And lastly, H is the host name of the OpenSearch server that you’re securing.
Let’s see it all in action.
This is me going through the OpenSearch security configuration, the “hash.sh”. Basically, I’m just going to go through all the steps I just outlined. So we copy the hash that out. We set a super high security million-character password there. And now we’re going to configure OpenSearch. The, in the tenants file, we need to delete the demo tenant.
We don’t need the tenant because Graylog is our tenant in this case. And all the demo users in the internal user’s database, which is just a YAML file. Here we have the hash password and the admin role set to the admin user. And now we’re going to run the security “admin.sh” script. Tie it all together. And all the options I laid out in the previous slide are here.
I’m just going to highlight them again, the pass itself. Yes, sure. The key password, hostname. I’m going to watch the magic happen.
All those success messages. There we go. Basically, saying that yes, this has worked. It’s a lot of output for a successful, but I guess that’s a good thing so you can watch what happens. And now, so we’ve set up OpenSearch. To use the security plugin and we’ve configured the security plugin.
Now we need to update Graylog because Graylog has been sitting there and if you’re, if you’ve been watching your “server.log” at this point, you were going to see a bunch of messages about, they say something else. Some other, I forget what it says is some cryptic open SSL error, but basically, it’s Graylog saying that it’s not able to connect to open search because open search is expecting.
Validate TLS Between Graylog and OpenSearch
HTTPS and it’s still talking HTTP. Go back into Graylog, change Elasticsearch hosts to the URL, including user authentication that Graylog is going to use to connect to OpenSearch. We restart Graylog server to make those changes go into effect. We look in the server log for the line saying it’s successfully connected to OpenSearch.
Because if you’ve worked with OpenSSL before and TLS certs, then you know that something always goes wrong. Getting to this point took me more tries than I care to admit. So always verifying that Graylog is happy with everything is always good. So here, do a TCP dump and capture an encrypted log being sent to OpenSearch.
So do another TCP dump on port 9200 port. Port numbers don’t change and I’m, yeah, I’m going to use a curl command, but also supply the certificate so that the curl client is actually supplying the certificate because that’s part of the configuration we did earlier. So we see the log came in and now we see, we don’t see anything because of the grip.
Of course, we’re grepping for a string. Grep doesn’t see because it’s encrypted. So we’re going to take off the grep and just see what all comes in. Now I’m constantly sitting in logs, so there’s going to be a, and plus this is also just Graylog, just checking in with open search all the time because they’re really chatty, but yeah, there’s a lot to keep up with.
Everything is now using TLS
So it makes sense. And yeah, so there, there’s the encrypted log coming in. This is the last step in encryption. So now all communications logs, users coming into Graylog logs, coming into Graylog. And Graylog talking to its OpenSearch backend are all encrypted, authenticated, and super duper secure. Do we have any questions?
I know that was a lot. And if you have any hard questions, feel free to send it to our ProServe team. I heard they deal with everything that we can’t deal with live.
Questions
Okay, my, my dear coworker, Ed has a question. He’s going to grill me. Yeah, it’s okay. Okay. Am I supposed to hug you? How does this work? So I didn’t see this covered, but whenever you’re doing the import of the cert, don’t use the store pass a switch because what can happen. Is a lot of people don’t realize this, the default password, it’s changing.
What about the passwords?
And so all of this hard work that he’s done, if you don’t change that password, it’s going to be in the history. No, that’s a good point. And that’s something that I did forgot that I did it in the demo. And if you were following along, you probably saw that I changed the default password. I forgot to say it.
So good call though, for sure. That, yeah, definitely good call. The question I had was, so you did about 30, 35 minutes here, 40 minutes, even. And you went through a ton of information, the part where you’re actually doing the real configuration, what would you estimate that would have been?
Basically, what’s the length of your demos?
What’d you say? Oh, to do all the commands, five or 10 minutes, maybe max. And again, of course, love five or 10 minutes max is awesome. That was my question. And now I’m saying at the end of it, you just did a great presentation where you talked about encrypting basically the database layer.
Yeah. In transit and in flight from the collector. As well as front to back from the UI in five to 10 minutes. Yeah I’m just doing that. That’s why then, I appreciate it. That’s why the real subtitle was like, it’s easier than you think. There’s a lot that goes into the planning and there’s a lot can go wrong, but overall, yes.
Thanks for Watching
It’s just a, it really is. Just a few commands. Thanks again for showing up. Lemme turn to the thank you slide. There we go. Thank you . Have a good one, folks.
