Syslog was developed in the 1980s by Eric Allman as part of the Sendmail project and adopted by many systems over the years. When looking at Syslog, there are a few protocol options, each with slight differences. In this reference guide, I’ll break down the differences so that you have a guide to see these formats when utilizing this protocol.
The Syslog protocol has been incorporated into many network hardware and operating Systems. Some have adopted it in addition to other protocols like JSON format and Kafka. In this article, each section will have a reference and a sample message will be provided. The fields will be identified, and the messages will help tell the story of the entire message. Utilizing Graylog as a Syslog server is very straightforward, and if would like to, you can find info here.
A list we will cover in this guide will be:
- RFC 3164 (BSD Syslog Protocol)
- RFC 5424 (Structured Syslog Protocol)
- RFC 3195 (Reliable Delivery for Syslog)
- RFC 6587 (Syslog over TCP)
- Syslog Kafka (formatted as per RFC 3164 or RFC 5424)
- Syslog AMQP (Advanced Message Queuing Protocol)
- Syslog TCP/UDP (Multiline format)
RFC 3164
Here’s a sample Syslog message formatted according to RFC 3164 (also known as the BSD Syslog protocol):
<34>Oct 3 10:15:32 mymachine su[12345]: 'su root' failed for user on /dev/pts/0Details of the message:
1. Priority (<34>): This represents the priority value (Facility * 8 + Severity). In this case:
– Facility = 4 (Auth)
– Severity = 2 (Critical)
– Priority = 4 * 8 + 2 = 34
2. Timestamp (Oct 3 10:15:32): The date and time of the log entry in the format Mmm dd hh:mm:ss.
3. Hostname (mymachine): The hostname or IP address of the machine where the log was generated.
4. Tag (su[12345]): The program name that generated the log message, followed by the process ID in square brackets.
5. Content (‘su root’ failed for user on /dev/pts/0): The actual log message describing the event.
RFC5424
Here’s a sample Syslog message formatted according to RFC 5424 (The Syslog Protocol):
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0Details of the message:
1. Priority (<34>): Same as RFC 3164. Priority = Facility * 8 + Severity. Here:
– Facility = 4 (Auth)
– Severity = 2 (Critical)
– Priority = 34
2. Version (1): Indicates the version of the Syslog protocol. RFC 5424 always uses version 1.
3. Timestamp (2025-01-03T14:07:15.003Z): A full timestamp in ISO 8601 format, including milliseconds and time zone (Z for UTC).
4. Hostname (mymachine.example.com): Fully Qualified Domain Name (FQDN) of the machine. If unavailable, the IP address can be used.
5. App-name (su): The name of the application or process generating the log.
6. Process ID (12345): The process ID of the application generating the message.
7. Message ID (ID47): An identifier for the type of message. This can be used for categorization or tracing purposes.
8. Structured Data (-): Placeholder for structured data. If no structured data is present, it is represented by a single dash (-).
9. Message (‘su root’ failed for user on /dev/pts/0): The free-text portion of the log entry describing the event.
This structured format provides much more detail and is better suited for modern logging needs compared to RFC 3164.
RFC 3195
RFC 3195 (Reliable Delivery for Syslog) does not define a specific message format. Instead, it specifies how Syslog messages should be transmitted reliably over a network using protocols like BEEP (Blocks Extensible Exchange Protocol). The content of the Syslog message is typically based on either RFC 3164 (BSD Syslog) or RFC 5424 (structured Syslog).
Here’s an example of how a Syslog message might look if transmitted using RFC 3195 with an RFC 5424 message:
Syslog Message:
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0Sent Over RFC 3195:
This message would be transmitted as part of a BEEP frame. For example:
BEEP Frame Example:
MSG 1 0 . 200 104
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0
ENDDetails of the message:
1. BEEP Header:
– MSG: Indicates the beginning of a message.
– 1: Message channel number.
– 0: Message sequence number.
– .: No immediate replies expected.
– 200: Payload size in bytes (depends on the actual message length).
– 104: Example length of the Syslog message.
2. Payload: The actual Syslog message in the format specified (e.g., RFC 3164 or RFC 5424).
3. END: Marks the end of the BEEP frame.
Key Features of RFC 3195:
– Reliable Delivery: Ensures Syslog messages are delivered without loss, unlike the unreliable UDP used in RFC 3164.
– Support for Structure: Can encapsulate both traditional and structured Syslog messages (e.g., RFC 5424).
– BEEP Protocol: Provides multiplexing and reliable message delivery.
RFC 3195 is more about the transport mechanism than the message format, so it works alongside existing Syslog message definitions.
RFC 6587
RFC 6587 specifies how Syslog messages are transmitted over TCP (Transmission Control Protocol) to ensure reliable delivery. The primary formats defined are octet-counting and non-transparent framing. The Syslog message itself can follow the format defined by either RFC 3164 or RFC 5424.
Here’s an example of how a Syslog message might look when sent using RFC 6587:
Example Using RFC 6587 with Octet-Counting:
114 <34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0Details of the message:
1. Octet Count (114): Specifies the exact length of the Syslog message in bytes, including the message itself. This ensures the receiver knows where the message ends.
2. Syslog Message (<34>1 …): The actual Syslog message, formatted according to RFC 5424.
Example Using RFC 6587 with Non-Transparent Framing:
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0The message is followed by a delimiter, typically a newline (\n) or other agreed-upon character.
Key Points for RFC 6587 Framing Methods:
1. Octet-Counting:
– Preferred for environments with variable-length messages.
– Eliminates ambiguity by including the length at the beginning of the message.
– Example: 114 <34>1 … (where 114 is the byte count).
2. Non-Transparent Framing:
– Simpler to implement.
– Relies on a delimiter to separate messages (e.g., newline \n).
– Requires careful message formatting to avoid delimiter collisions.
Sample Syslog Message Explained (Using Octet-Counting):
114 <34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/01. 114: The byte count of the entire message.
2. <34>: Priority (Facility 8 + Severity = 34).
3. 1: Syslog protocol version.
4. 2025-01-03T14:07:15.003Z: ISO 8601 timestamp.
5. mymachine.example.com: Hostname of the sender.
6. su: Application name generating the log.
7. 12345: Process ID.
8. ID47: Message ID.
9. -: Placeholder for structured data.
10. ‘su root’ failed for user on /dev/pts/0: The actual log message.
RFC 6587 enhances the reliability of Syslog by providing clear message framing, ensuring no message is lost or misinterpreted in a TCP stream.
Syslog Kafka
Syslog messages can be sent to Kafka for centralized logging, often used in modern distributed systems for reliable message processing. Kafka doesn’t define a specific Syslog format but acts as a transport layer where Syslog messages (typically RFC 3164 or RFC 5424) are encapsulated and published to a Kafka topic.
Here’s an example of how a Syslog message would be formatted for Syslog Kafka ingestion:
Example Syslog Message in Kafka (Using RFC 5424 Format):
{"key":"syslog", "message":"<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0"}Details of the message:
1. Kafka Message Key (“key”:”syslog”):
– Optional key used for partitioning or identifying the message type.
– In this example, the key is syslog.
2. Kafka Message Value (“message”:”…”):
– Encapsulates the actual syslog message.
– The Syslog message is formatted using RFC 5424 for structured logging.
Example Syslog Kafka with Additional Metadata:
{
"key": "syslog",
"metadata": {
"topic": "syslog_topic",
"partition": 1,
"offset": 23456,
"timestamp": "2025-01-03T14:07:15.003Z"
},
"message": "<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0"
}Breakdown with Metadata:
1. “metadata”:
– Contains Kafka-specific metadata such as:
– Topic: The Kafka topic where the Syslog message is published (e.g., syslog_topic).
– Partition: The partition in Kafka where the message is stored (e.g., 1).
– Offset: The offset of the message in the partition (e.g., 23456).
– Timestamp: When the message was received by Kafka.
2. “message”:
– The actual Syslog message formatted as RFC 5424 or RFC 3164.
Sample Message Sent to Kafka Broker:
If using a command-line producer or programmatically publishing to Kafka, a typical example might look like this:
bash
echo '{"key":"syslog", "message":"<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0"}' | kafka-console-producer --broker-list localhost:9092 --topic syslog_topicNotes for Syslog Kafka Integration:
1. Message Parsing:
– Syslog messages can be parsed and enriched (e.g., add host metadata) before publishing to Kafka.
2. Topic Structure:
– Different Syslog facilities or applications can map to different Kafka topics for better organization (e.g., auth_logs, system_logs).
3. Schema Registry:
– If using Avro or JSON with a schema registry, Syslog messages can be validated and structured more consistently across systems.
4. Consumer Applications:
– Downstream Kafka consumers process and analyze these Syslog messages, often storing them in Graylog, or databases for querying and visualization.
Syslog Kafka bridges the gap between traditional logging protocols and modern event-driven architectures.
Syslog AMQP
When Syslog messages are sent over **AMQP (Advanced Message Queuing Protocol)**, such as with RabbitMQ, the Syslog messages are encapsulated in an AMQP message format. The content of the Syslog message typically follows RFC 3164 (BSD Syslog) or RFC 5424 (structured Syslog).
Here’s an example of how a Syslog message might look when sent via Syslog AMQP:
Example Syslog Message Encapsulated in AMQP:
json
{
"exchange": "syslog.exchange",
"routing_key": "syslog.auth",
"message": "<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0",
"properties": {
"content_type": "text/plain",
"delivery_mode": 2,
"priority": 5,
"timestamp": 1735918035
}
}Details of the message:
1. exchange:
– The AMQP exchange to which the Syslog message is published (e.g., syslog.exchange).
– Can be a topic exchange, direct exchange, or fanout exchange.
2. routing_key:
– The routing key determines which queues receive the message. For example:
– syslog.auth: Logs related to authentication.
– syslog.system: Logs related to system events.
3. message:
– The actual Syslog message formatted according to RFC 3164 or RFC 5424.
– Example uses RFC 5424 format:
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/04. properties:
– Metadata about the AMQP message:
– content_type: Specifies the format of the message (text/plain or application/json).
– delivery_mode: Persistent (2) or non-persistent (1) message.
– priority: Priority level of the message (0–9).
– timestamp: Unix timestamp indicating when the message was published (e.g., 1735918035 for 2025-01-03T14:07:15Z).
Sending a Syslog Message via AMQP (RabbitMQ Example):
If sending using a script, the following Python example uses Pika (a Python AMQP library):
import pika
import time
# Connection parameters
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
# Declare exchange
exchange_name = 'syslog.exchange'
channel.exchange_declare(exchange=exchange_name, exchange_type='topic')
# Syslog message
syslog_message = "<34>1 2025-01-03T14:07:15.003Z mymachine.example.com su 12345 ID47 - 'su root' failed for user on /dev/pts/0"
routing_key = 'syslog.auth'
# Publish message
channel.basic_publish(
exchange=exchange_name,
routing_key=routing_key,
body=syslog_message,
properties=pika.BasicProperties(
content_type='text/plain',
delivery_mode=2, # Persistent
priority=5,
timestamp=int(time.time())
)
)
print(" [x] Sent syslog message")
connection.close()
Notes for Syslog AMQP Integration: 1. Exchange and Queue Design: - Use topic exchanges for flexibility (e.g., route messages based on facility or severity). - Example routing keys: - syslog.auth.error - syslog.system.info
2. Message Persistence:
– Ensure Syslog messages are marked as persistent (delivery_mode=2) for reliability.
3. Structured Data:
– If using RFC 5424, structured data within the Syslog message can be parsed by consumers for enhanced logging insights.
4. Consumer Applications:
– AMQP consumers (e.g., log processors, monitoring tools) can retrieve Syslog messages and process them further, such as storing them in OpenSearch.
5. Priority Handling:
– AMQP supports message priorities, allowing Syslog messages to be processed based on severity or importance.
Syslog over AMQP provides reliable delivery and advanced message routing, making it ideal for distributed logging architectures.
Syslog Multiline Format: A Tricky Thing!
The Syslog Multiline Format is not a specific protocol or standardized format defined by an RFC. Instead, it refers to techniques and practices used to handle log messages that span multiple lines, which is common in scenarios where:
1. Log messages include detailed information (e.g., stack traces or multi-step process outputs).
2. Applications output complex logs that don’t fit within a single line.
Multiline Syslog messages are challenging because traditional Syslog formats (e.g., RFC 3164 or RFC 5424) are designed for single-line messages, and multiline logs can break parsers or disrupt logging workflows.
Key Approaches for Handling Multiline Syslog
1. Escaping Newlines
– Newlines in the log message are escaped or replaced with a special character or sequence (e.g., \n).
– Example:
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com app 12345 - - "Error occurred:\nLine 1 of the stack trace\nLine 2 of the stack trace"– Advantages:
– Keeps the log message within a single Syslog entry.
– Disadvantages:
– Can make logs harder to read without additional processing.
2. Structured Data in RFC 5424
– RFC 5424 allows for structured data fields, which can encapsulate multiline data within a single message.
– Example:
<34>1 2025-01-03T14:07:15.003Z mymachine.example.com app 12345 ID47 [details@12345 stack="Error:\nLine 1\nLine 2"] "Error processing request"– Advantages:
– Fully compatible with structured logging systems.
– Disadvantages:
– Requires support for RFC 5424 and structured data.
Challenges with Multiline Syslog
1. Parsing Complexity:
– Multiline logs are harder to parse, especially with traditional Syslog parsers.
2. Potential for Log Corruption:
– If newlines are not handled correctly, multiline messages can corrupt the log stream.
3. Compatibility:
– Not all Syslog systems or tools support advanced multiline handling techniques.
Best Practices for Multiline Syslog
1. Use Structured Formats:
– JSON or RFC 5424 with structured data is ideal for complex logs.
2. Configure Log Forwarders:
– Use tools like Fluentd or Filebeat and NXLog to preprocess multiline logs into JSON or GELF format.
3. Escape Newlines When Necessary:
– Escape newlines (\n) if the receiving system cannot handle multiline entries.
4. Delimit Consistently:
– Use consistent delimiters for easier parsing.
In Closing
Syslog continues to be a protocol supported for logging across many applications and hardware. It is still being used today and it is a very common logging format. Graylog supports Syslog and the standards: RFC 5434 and RFC5424. When Syslog is sent in the message field as JSON, Graylog can easily parse this info into fields using the JSON parsing functions.
Many new logging formats have come out over the years. Syslog is still here and will be for quite some time. Try Graylog as a Syslog server!
