Introduction to Data Node Migration
In this demonstration, we’re showcasing the amazing work that the Data Node team has been doing on Data Node migration.
Initial Setup
Existing Graylog Instance
I’ve got a Graylog instance open here. This is a vanilla Graylog setup with Linux CIS log feeding into it.
System Overview
If I go over to my system overview, I’m using a self-managed open-source cluster that I installed independently. Right now, I would have to patch, upgrade, and configure this independently to manage my setup.
Data Node Information
What is Data Node?
I can now go into my Data Node information to read more about it. If I want to migrate from my self-managed version of Open Source Open Search to leverage Data Node, I can do that. Data Node helps set the versioning, patch level, and configuration automatically, simplifying management.
Migration Overview
If I want to migrate, I can read about the steps involved. The first part of the migration involves setting up a certificate, so I can use Open Search securely.
Setting Up a Certificate Authority
Creating a New Certificate Authority
The team has made this process ridiculously easy. You can create a new Certificate Authority (CA) called “Graylog.” If you have your own certificate authority, you can upload that as well.
Setting the Renewal Policy
Once the Certificate Authority is created, you can set up the renewal policy. The default is 30 days, and it automatically renews every 30 days. This ensures that if an attacker manages to compromise a certificate, it’s only valid for a limited time, and as a manager, this happens automatically without requiring manual intervention.
Migration Steps
In-Place Migration
There are two types of migration. I’m going to choose an in-place migration, which is what 90% of our customers do. This method replaces the self-managed Open Search nodes with Data Node while retaining all existing data.
Preparing Open Search Configuration
There are a few configuration steps that need to be done, but we provide step-by-step instructions to help. I’ve already edited my Open Search configuration for this migration. However, I will need to make one more minor edit soon.
Compatibility Check
Next, I run a compatibility check to ensure that we can perform the migration. The check confirms that we’re good to proceed.
Setting Up the Data Node
Configuring Data Node
Now that we’ve passed the compatibility check, we can set up the Data Node using the certificate we just created. The system tells me that it’s currently unconfigured but is preparing for migration. Once everything is ready, I can proceed to the next step.
Journal Size Warning
The system provides a journal size warning, which lets me know the throughput isn’t very high. This is a safeguard to ensure there is no data loss during the migration process. Since my journal size is fine, I’m good to move forward.
Stopping Open Search
Before proceeding, I need to stop Open Search since we will no longer be using it. After this migration, Open Search will be managed through Graylog services, so it will no longer need to be manually started or stopped.
Final Configuration Changes
Editing Server Configuration
The last step involves removing Graylog’s reliance on the independent Open Search instance. I’ll do this by editing the server configuration file (`server.conf`) and commenting out the related settings. This ensures that Graylog will now use Data Node instead of Open Search.
Restarting Graylog
Once I’ve made the changes, I need to restart Graylog for the new configuration to take effect. After restarting, the migration is complete.
Post-Migration Check
Verifying the Migration
After restarting Graylog, my system is back up, and the migration is finished. When I go back to the system overview, I can see that it’s no longer an Open Search cluster but a Data Node cluster.
Data Validation
Let’s run a search across all time to confirm that all my messages are still intact. As expected, everything is there. The Data Node migration was a success—easy peasy!
