In this post, we will go through creating your own processing pipeline function. Some Java experience will be helpful, but not necessary. We will be taking it step-by-step from understanding a pipeline, to implementing and installing your function.
Let’s begin!
What is a Graylog Processing Pipeline again?
Graylog Message Processing Pipelines were released in Graylog v2.0 and are a great way to flexibly route, blacklist, modify and enrich log messages.
What does this mean?
Essentially, every message that is sent to Graylog will be evaluated by your pipeline configuration. Depending on your rules, the message will either be modified or discarded.
Pipeline Basics:
A pipeline consists of rules that trigger functions. A rule can look like this:
rule “Example: Uppercase a message field”
when
has_field(“some_field”)
then
let x = uppercase(to_string($message.some_field));
set_field(“some_field”, x);
end
The rule above states that if a message has the field, ‘some_field,” then convert the value into uppercase letters.
Now let’s create a custom function. This function will have an easy domain logic so that we can focus on the plugin creation and installation.
Function: string_length(string) This function will return the length of a string we pass to it.
Creating a plugin skeleton
(You can find the full example code on GitHub)
Let’s start with some good news! We are providing a Maven Archetype that will create a complete plugin skeleton that you can load into your favorite Java IDE with no effort. All the boilerplate code and base configuration will be included as well. .
Before we begin, make sure you have a Java 8 JDK and maven (the Java build tool) installed:
$ java -version
java version “1.8.0_91”
$ javac -version
javac 1.8.0_91
$ mvn -version
Apache Maven 3.2.5
(Protip: You’ll find all this in homebrew if you are on OSX)
We recommend IntelliJ IDEA for your Java IDE, but Eclipse or most others will work as well.
Now let’s create the plugin skeleton project.
In your workspace, run this: (It will create a new folder with all your plugin files in it)
$ mvn archetype:generate -DarchetypeGroupId=org.graylog -DarchetypeArtifactId=graylog-plugin-archetype
This will interactively ask you a few questions about versions and names of your new plugin.
Use the example below::
groupId: com.example.plugins
artifactId: graylog-plugin-function-strlen
version: 1.0.0-SNAPSHOT
package: com.example.plugins.strlen
githubRepo: YourGitHubUsername/graylog-plugin-function-strlen
ownerEmail: [email protected]
ownerName: John Doe
pluginClassName: StringLengthFunction
You will see a new folder called graylog-plugin-function-strlen.
Use the Open Project dialog of your IDE and select this newly created folder. Since the archetype created a full Maven project, your IDE is able to import it without any other configuration required.
Preparing the plugin
Open the StringLengthFunctionMetaData class and look at the code that was generated for you.
There are four methods in there that you want to touch:
Now open the pom.xml file which configures the build of our plugin. We’ll need to add a dependency to the Graylog pipeline processor code here so we can implement certain functionality.
Search for the <dependencies> section and add this:</dependencies>
<dependency></dependency>
<groupid>org.graylog.plugins</groupid>
<artifactid>graylog-plugin-pipeline-processor</artifactid>
<version>1.1.1</version>
<scope>provided</scope>
Like with all other Graylog components, make sure to use the latest stable version. At the time this guide is being written it is 1.1.1, but you can always double check on Maven Central.
Your IDE should ask you if you want to update the Maven settings. Confirm and also set it automatically update Maven settings in the future if the pom.xml file is changed. This will make your life easier. (IntelliJ IDEA will offer to do that automatically)
Implementing the function
Open the StringLengthFunction class and let it implement Function<integer>: We use Integer because the return type will be the number of characters in the string that was passed to it. </integer>
package com.example.plugins.strlen;
import org.graylog.plugins.pipelineprocessor.EvaluationContext;
import org.graylog.plugins.pipelineprocessor.ast.expressions.Expression;
import org.graylog.plugins.pipelineprocessor.ast.functions.AbstractFunction;
import org.graylog.plugins.pipelineprocessor.ast.functions.Function;
import org.graylog.plugins.pipelineprocessor.ast.functions.FunctionArgs;
import org.graylog.plugins.pipelineprocessor.ast.functions.FunctionDescriptor;public class StringLengthFunction extends AbstractFunction {
public static final String NAME = “string_length”;
private static final String PARAM = “string”;@Override
public Integer evaluate(FunctionArgs functionArgs, EvaluationContext evaluationContext) {
return null;
}@Override
public FunctionDescriptor<integer> descriptor() {</integer>
return null;
}
}
Note that we also added a constant NAME and PARAM that hold the name of this function and the name of the parameter it takes. We’ll use it later.
All we need to do is implement these two functions:
Let’s start by implementing the FunctionDescriptor. Add a member variable that describes the one parameter our function will take (the string of which we want to calculate the length of):
private final ParameterDescriptor valueParam = ParameterDescriptor
.string(PARAM)
.description(“The string to calculate the length of. For example, passing ‘foo’ will return 3.”)
.build();
Now implement the FunctionDescriptor:
@Override
public FunctionDescriptor<integer> descriptor() {</integer>
return FunctionDescriptor.<integer>builder()</integer>
.name(NAME)
.description(“Returns the length of a string”)
.params(valueParam)
.returnType(Integer.class)
.build();
}
All that is left is implementing the actual function logic:
@Override
public Integer evaluate(FunctionArgs functionArgs, EvaluationContext evaluationContext) {
String target = valueParam.required(functionArgs, evaluationContext);
if (target == null) {
return 0;
}return target.length();
}
That’s it! Now we just need to instruct the graylog-server instance that loads this plugin that we have a new pipeline function. We’ll need to load it and make it available for the Graylog users.
Instructing Graylog to load the new function
Open the StringLengthFunctionModule class and add three methods that will allow us to install the function:
protected void addMessageProcessorFunction(String name, Class<? extends Function<?>> functionClass) {
addMessageProcessorFunction(binder(), name, functionClass);
}public static MapBinder> processorFunctionBinder(Binder binder) {
return MapBinder.newMapBinder(binder, TypeLiteral.get(String.class), new TypeLiteral>() {});
}public static void addMessageProcessorFunction(Binder binder, String name, Class<? extends Function<?>> functionClass) {
processorFunctionBinder(binder).addBinding(name).to(functionClass);
}
Now in the configure() method, install our new function:
@Override
protected void configure() {
addMessageProcessorFunction(StringLengthFunction.NAME, StringLengthFunction.class);
}
Done! Now let’s compile the plugin and move it into the graylog-server plugin directory.
Compiling and installing the plugin
From the pom.xml file, remove the <profiles> section. This will disable the attempt to build Graylog web interface plugin code which we do not need.</profiles>
Navigate into the graylog-plugin-function-strlen folder and compile the plugin:
$ mvn package
You should see a target/ folder that will hold a graylog-plugin-function-strlen-1.0.0-SNAPSHOT.jar file. Copy it to your graylog-server plugin_dir directory (configured in graylog-server.conf) and restart graylog-server.
$ cp target/graylog-plugin-function-strlen-1.0.0-SNAPSHOT.jar ../graylog2-server/plugin
Make sure not to move the original-graylog-plugin-function-strlen-1.0.0-SNAPSHOT.jar which has also been generated.
During launch of the server, you should see this:
2016-09-17 16:55:15,094 INFO : org.graylog2.bootstrap.CmdLineTool – Loaded plugin: String length pipeline function 1.0.0-SNAPSHOT [com.example.plugins.strlen.StringLengthFunctionPlugin]
Congratulations! Now let’s use our new function.
Using the function
Here is a test rule that uses our new function:
rule “Testing our new plugin”
when
has_field(“source”)
then
let length = string_length(to_string($message.source));
set_field(“source_length”, length);
end
This will calculate the length of the string in the message field source and then store the result in a new message field called source_length.
You will also see our new function in the quick reference:
To test, add the rule to a new pipeline and wire that pipeline to the default stream. Then, send in messages and confirm that the function and rule are working.
And that’s it! You have successfully written and implemented your own Graylog Processing Function! Go you!
You can find the full example code on GitHub.
WRITING MORE COMPLEX FUNCTIONS
Now that we have gone through the full process of creating a function with a single call to String#length(), you will be able to write more complex functions. A more sophisticated example is the Graylog Threat Intelligence Plugin that deals with calls to other web services, dynamic configuration from the web interface, and caching.
Lastly, remember to share your function on the Graylog Marketplace if it is open source.
