Environment Review
Before we start to look at the installation and configuration, I just want to take a moment to briefly go over the environment that we’re working with today. So conceptually, this is what we’re looking to replicate. We’ve got a head office here on the left hand side with some various log sources, a load balancer and two Graylog forwarders that connect over public infrastructure to a data center where our great log cluster is running. In reality this is running in a lab environment, so it’s running on VMs and this is actually what we’ve got.
So we’ve got a Graylog cluster running on one node, and a Graylog forwarder box and a second Graylog forwarder box, and we’ve got a one Syslog source in this environment just to prove that the connectivity’s there at the end of the demonstration. So, that’s the sort of conceptual overview. If we jump back now, we can see that on the right hand side of the screen, I’ve got the Graylog documentation open, and I would recommend before you undertake any configuration work like this, to check out the latest version of the documentation which can be found at docs.graylog.org.
As I just mentioned, I would take the time to familiarize yourself with the steps contained in the documentation. I already know the steps so we are going to jump straight into the UI and get cracking.
Create Input Forwarders
The first thing that we want to do in the Graylog UI is create an input for our forwarders. So if we jump into the system menu, inputs, and we’re going to create a new input, that’s going to be a forwarder type. We’re going to launch the input and we’re going to call it, “lab forwarder”. I’m also going to disable TLS in this instance, it’s just that I don’t have the specific authority locally, and so it’s just not a live environment so it’s not really important that we encrypt the traffic between the forwarders and Graylog. Obviously in a production environment, particularly those environments where you are using public infrastructure to transport log data, you definitely want to enable TLS so, I would recommend enabling that where possible. So if we hit the save button, that’s going to create the input and start it for us.
Starting the Forwarder Configuration
The next step we need to do is to come across and to jump into our enterprise menu and our forwarder submenu. As you can see, we’ve not got any forwarders there yet, so we’re going to hit the get started button and now, we’re going to start the configuration to walk through the wizard and we’re going to start a new forwarder.
We’re going to continue across the first step, and we’re going to jump straight into “Create API token.” Now this API token, we’re going to reuse on both of our forwarders and I’ll explain why we’re doing that later on in the video. But for now, we’re just going to call it “head office”. We’re going to create the token and what the Wizard has done is it has been presented as the config that’s associated with this API token. We’ve got this really handy copy configuration snippet button here, which we’re going to take advantage of. We’re going to drop that into our Notepad++, and we’re going to replace the one that I had in there before I’d set this video up to record.
So the next thing that’s really important to do at this stage is to hit the continue button. And what that does is that saves the API token configuration into Graylog, and that it sort of puts Graylog in a position where it’s now ready to receive connections from forwarders. So that’s our cradle configuration done for the time being, we’re going to jump across now and build our forwarder. So we want to run the prerequisite installations before we get onto the forwarder installation.
Building the Forwarder
The first thing we’re going to do is update our OS. We grab the latest patches for that and this can take a couple of minutes depending on how old your OS is. So, we’ll jump back into this once this installation finishes. Okay, so that installation of the OS update is completed and the next thing we’re going to do is install our prerequisite packages. So we’re going to hit that, we’re going to install them and that shouldn’t take too long to complete. Okay, so now it’s completed, we’re going to crack on and get the packages we need for our forwarder installation. So we’re going to run through the installation steps here.
Installing the Forwarder
Okay, so that’s our repository setup, and now we are actually going to install the Forwarder package, and now that’s complete. Now, the last thing we need to do on the forwarder side is edit our “forwarder.conf” file to tell it where and how to connect to our Graylog cluster. So when we open the file up, we’ve got this template configuration that’s in here. So we’re going to delete the top two lines, because we have that config stored already along with some extra configuration, from our little handy code snippet that we had earlier.
So we’re going to paste that in and you can just see there the top line is the IP address. The Graylog of cluster, the next line is the actual API token itself. We’ve got a configuration port and a transmission port, used for communication between the forwarder and Graylog and the next line is the fact that we’ve disabled TLS. Okay, so let’s write that config and then the next step for us now is to start the service. So let’s copy that command.
Okay, so the service just started. We can check that now using the systemctl command again and we can just add this status, switch to it, and you can see that the service is up and running. The other way that we could check to see if everything’s running okay, is just to tail the log file, which can be found at var/log/graylog-forwarder/forwarder.log and as you can see there, we’ve just got some info messages that have just been written basically during the start phase of the forwarder.
So if we jump across, back now to our Graylog UI, we can see that GL forwarder, which is our first forwarder machine has appeared now in the Wizard so we can continue on with that forwarder. So I’m going to call this “lab forwarder one”, which is not the most imaginative name, but it’ll suit our purposes.
And now we come to create an input profile. So, there’s a little explanation on the right hand side of what an input profile is. The way I like to think of it is a collection of inputs that can be reused across multiple forwarders and where I think this would be particularly handy is if you’re setting up multiple forwarders on the same site, and you’re expecting to collect the same log sources from that site. So maybe you’ve got a low balance in front of your forwarders and you’re going to collect from the same log sources. So that could be like Windows and Syslog devices from maybe network, or maybe you’ve got some Linux that environment or whatever it might be. So what we’re going to do is we’re going to create a profile. We’re going to call it “head office”, but we’re going to spell it correctly.
And now we’re going to add inputs to that profile. In this case, we’ve got a Syslog source, we’re just going to transmit in raw, plain text over UDP. So I’m going to call this syslog/raw/udp. We’re going to create the input and we’re going to save the input. Now we’ve only got one input for this particular profile but that’s just a limitation of the environment we’re working in. So we’re going to exit the configuration, and now we come to our sort of forwarder’s overview page, and we can see on the left hand side, we’ve got our “lab forwarder one”, which is what we called it. It’s also got the forwarder ID associated with that forwarder. We’ve got our status of connected. We can see the host name of the machine there, the input profile, and the fact that we’ve received no messages.
So if we actually click on the title of the forwarder, we can see that we’ve got the Syslog raw UDP input, that has just switched from unknown to a running state who are now ready to receive raw Syslog over UDP on this forwarder. So if we jump across to our show received messages, just make sure we hit the play button at the top there to make sure that this is going to update. It’s going to open a putty session and I’m just going to file some logons and then you can see there that the file logon attempts have now appeared. So that’s this logon is being sent to the forwarder, and the forwarder’s then packaging that up and sending along to Graylog, and that’s what we’re seeing here in the UI.
So we’ve proved sort of end to end connectivity for that. The last thing I want to do in this video is show you how to add a forwarder to this configuration. And so, what we’re going to do is in true lab fashion, we’re going to jump across to our second forwarder machine. I’ve already taken the liberty of installing and running all the prerequisites and we are now at the stage, where we just need to update the forwarder configuration file.
So we’re going to copy the command from our instructions, and you can see that we’ve got the default template in here. What we’re going to do is we’re going to reuse the command that we got from our code snippet and we’re going to paste that in here, I should really set this as the default. There you go. So now we use the reusing the same configuration that we used before. We’re going to write that to the file, and then we’re going to start our service. Okay, so our service just started and if we jump back across to the UI, we come back into enterprise into forwarders, we can see that we’ve got our second forwarder here and we’ve got a disconnected state that should switch to connected in a second, there we go.
And then what we do is we’re going to hit the configure button, and that’s what’s going to give us a chance to name it. When a forwarder has just made connection back into Graylog, sometimes this screen can take a little while to update. Sometimes if it doesn’t update after a while, you can give it a refresh and you can jump into the config that way. So we’re going to call this “lab forwarder two”, and we’re going to add forwarder inputs and instead of creating a new input profile. We are going to select our head office input and we’re going to finish configuration, exit the configuration.
And now we can see that we’ve got two forwarders, we can see the host names there, we can see the input profiles, the head office and indeed, if we check our lab forwarder two, we can see that we’ve got the Syslog raw UDP input and that’s just starting up now. Now, the advantage of doing things this way is obviously we’ve had to not recreate the input so, imagine that you’ve got a collection of inputs in this profile, maybe five or six and we’ve not had to recreate them on an individual basis. And because we’ve reused the API token, that means that if we were to reject that API taken from the Graylog cluster, that all the forwarders from that site would then no longer be out to connect back into Graylog.
So if you’re decommissioning a whole site, that becomes really handy to be able to manage all of those forwarders in one go. You could of course create individual API tokens per forwarder. It’s just an extra step in my view and you can make a case either way. That brings us to the end of the forwarder video, I hope you found it really informative and useful, but obviously if you do get stuck at any stage during the configuration and building of any forwarders, please don’t hesitate to contact Graylog support and thanks very much for watching, and I will catch you in the next one.
