Revised 2/2/22
Log management tools improve security, help troubleshoot issues, and enable system monitoring. This log management guide defines key terms and explains the benefits of centralized tools.
WHAT IS A LOG?
A log file is data produced automatically when certain events occur in systems, networks, and applications. They produce records documenting activities for:
- Users
- Servers
- Networks
- Operating systems
- Applications/software
For example, event logs may track:
- When a computer was backed up
- Errors preventing an application from running
- Files requested by users from a website
Security and IT operations teams use them to investigate and respond to abnormal system activity.
Logs can come in two different formats. Some can be opened and read by a human. Others are kept for auditing purposes and are primarily machine-readable only.
Some examples of log types include:
- Audit logs
- Transaction logs
- Event logs
- Error logs
- Message logs
- Security logs
Finally, logs come in a broad variety of formats or extensions, like
- .log
- .txt
- JSON
- .csv
- .dat
Depending on the extension and readability, you can open log files with:
- Standard text editor like Notepad
- Word processing applications like OpenOffice or Microsoft Word
- Command-line application like PowerShell
- Microsoft Excel
- OpenOffice Calc
- LibreOffice Calc
WHAT IS LOG MANAGEMENT?
Log management is the process for managing event logs, including the following activities for log:
- Generating
- Transmitting
- Storing
- Analyzing
- Disposing
Log management is critical to compliance. Since the event logs contain all the data about the activity occurring in the environment, they act as documentation for audits. For example, log management helps meet some compliance requirements, including those for:
- Federal Information Security Management Act of 2002 (FISMA)
- Health Insurance Portability and Accountability Act of 1996 (HIPAA)
- Sarbanes-Oxley Act of 2002 (SOX)
- Gramm-Leach-Bliley Act (GLBA)
- Payment Card Industry Data Security Standard (PCI DSS)
- General Data Protection Regulation (GDPR)
WHAT IS CENTRALIZED LOG MANAGEMENT?
A centralized log management solution is a technology that enables organizations to manage all the various types of logs. These include those from across their on-premises, cloud, and hybrid environments by enabling the following:
- Collection: ingesting logs from across diverse systems, networks, applications, and devices
- Aggregation combining the increased number, volume, and variety of log sources in a single location
- Parsing: pulling out the important information from each event log for usability
- Normalization: creating a standard format for all event log data
- Correlation: combining information about events across all environments for greater visibility into activity
- Analysis: automating correlated data points to create high-fidelity alerts
A centralized logging tool gives operations and security high-fidelity alerts so they can detect, investigate, and respond to issues faster.
You can use log management tools like Graylog to ingest different applications or systems logs. This way, you can view and extract valuable data from them.
HOW ARE LOG MANAGEMENT TOOLS USED?
With a centralized log management solution, everyone gains visibility into what’s happening across the diverse IT environment. More importantly, centralized log management makes it easier to take action on things like down servers or security incidents. However, it’s also important to understand how centralized log management solutions work to understand their value better.
LOG COLLECTION
The first step in log management is determining how to collect log data and store it. This is the fundamental value of a centralized log management solution.
Different parts of the IT environment that generate log data include:
- operating systems,
- firewalls,
- servers,
- switches,
- routers,
- Workstations
- Applications
- Intrusion detection systems (IDS)
- Intrusion prevention systems (IPS)
- Anti-virus solutions
- Endpoint detection and response (EDR) solutions
Each system, software, and device can generate dozens of EPS (events per second). This is why using a log collector that can handle the corresponding amount of logs matters.
A centralized log management solution allows you to configure and customize the log data you want.
STRATEGIES FOR COLLECTING LOGS
You want to configure log data collection settings to leave out redundant data and ensure that you gather all relevant information. This is the minimalist approach that improves performance and efficiency.
The other way to approach log collection is to go with a maximalist strategy. This means collecting every possible piece of information so the log management tool can sort and analyze it. While this approach has many drawbacks, the two biggest ones are:
- Increased costs: Storing a large amount of data is expensive and requires additional staffing to manage the process
- Reduced efficiency: Extremely large data sets stored online will reduce overall performance.
LONG-TERM LOG STORAGE AND RETENTION
Collection gives you the ability to have long-term log storage and retention. Many compliance mandates have log storage and retention requirements, so you should be considering this as part of your log collection. Generally, best practices suggest storing log data for at least one year in case there is a need for investigation.
When storing logs, you can choose to back up data to on-premises servers or in the cloud. This decision often comes with a company’s decision to undergo a digital transformation and migrate its resources online.
LOG ROTATION
Log rotation automates the process of renaming, resizing, moving, or deleting log files that are too large or too old.
You can choose a time interval after which the log will be:
- Deleted,
- Compressed to save space
- Emailed to another location.
This way, new storage space opens up for more recent log files.
Graylog streamlines log indexing and rotation for optimized archival.
LOG AGGREGATION
Aggregating all log data in one place can be challenging without a centralized log management solution. Some of the challenges include:
- High volumes of data
- Log data accuracy
- Diverse formats across systems, networks, applications, and devices
Even if your log management system can handle large volumes of data, what matters is how fast this data is generated. Log management tools should be able to keep up with this speed. This is why the EPS of a tool is something you should consider when choosing one.
LOG PARSING
You want to focus on the data that matters the most to meet your needs. Log parsing is the process of pulling out the data you need the most.
Every event log entry is classified by type. Typically, you’ll have the following event log types:
- Information: basic description of something happening that was supposed to happen
- Warning: Notification of an event that may not be significant now but indicates an issue that could occur later
- Error: Something went wrong, leading to a significant problem
- Success audit: Security log indicating the successful completion of an audited security event
- Failure audit: Security log indicating that an audited security event did not complete successfully
In addition, each audit log may contain the following information:
- Date
- Time
- User
- Computer/Device
- Event ID
- Source
- Type of event
- Description of event
LOG NORMALIZATION
Parsing data helps you get the pieces of data you need. Log normalization helps you create a standard format across all event logs.
For example, the following are some of the different types of log formats that you may be collecting:
- Syslog: event messages from network devices like routers and switches
- JavaScript Object Notation (JSON): format that can be read by both humans and machines
- Windows Event Log: records from Windows-based operating systems and applications
- Common Event Format (CEF): text-based, extensible format that is easily readable
JSON
{“post code”: “90210”, “country”: “United States”, “country abbreviation”: “US”, “places”: [{“place name”: “Beverly Hills”, “longitude”: “-118.4065”, “state”: “California”, “state abbreviation”: “CA”, “latitude”: “34.0901”}]}
Syslog
Your environment is likely collecting event logs that use two or more of these formats, like Syslog and Windows Event Log. However, because the formats name the pieces of information differently, parsing only extracts those pieces. Normalizing standardizes how the data is presented so that you gain meaningful insights.
For example, the Graylog Extended Log Format (GELF) standardizes complex and diverse event logs. It allows you to collect structured events from anywhere to get the insights you need.
‘{“short_message”:”metrics”, “_cpu”:”88″, “_mem”:”1024″, “_disk”:”90″, “_guid”:”b375dc4c-dd07-479b-9288-b152aa583318″}’
Correlation of Events
In a modern, complex IT environment, you have a lot of interconnected systems, networks, and applications. You need to understand all the dependencies and be able to trace the issue to the root cause.
Correlating events is the process of combining multiple events to see the relationships that exist. For example, a server outage may impact application performance. However, your IT team received a help desk call about the slow application.
By correlating the events, you can do the root cause analysis faster.
Log Analysis
With log analysis, you make use of the data that you’re collecting, parsing, normalizing, and correlating.
Centralized log management tools automate and simplify the process of log data analysis. Visualizations like charts and plots emphasize the correlations and similarities between events and data. This way, it’s easier to detect issues and track down their causes.
The top use cases for log analysis include:
- compliance
- security
- troubleshooting
- performance improvement
WHAT ARE THE KEY BENEFITS OF LOG MANAGEMENT & MONITORING
Log management is important because it allows you to take a systematic approach for real-time insights into operations and security.
Some of the benefits of log management and monitoring include:
- System monitoring
- High-fidelity alerts
- Enhanced security
- Faster troubleshooting
- Optimized resource usage
- Improved compliance posture
SYSTEM MONITORING
One of log management’s benefits is enabling you to monitor everything happening across your diverse IT environment. Additionally, multiple people gain the same visibility for better communication as part of system monitoring. For example, centralized system monitoring enables:
- IT operations
- DevOps
- Developers
- System Administrators
- Security operations
This visibility can provide insight into performance issues that indicate potential or future issues.
With Graylog, you can share information using emails, collaboration tools, and ticketing systems.
HIGH-FIDELITY ALERTS
Centralized log management’s ability to correlate and analyze event log data also means that you can create high-fidelity alerts. You can adjust monitoring settings to track a custom selection of events with customizable real-time alerts. These alerts allow you to react faster and reduce downtime.
ENHANCED SECURITY
High-fidelity alerts also enable enhanced security. You may have a dedicated security team or people within the IT team managing security functions. In both cases, everyone is overwhelmed by the large number of alerts, many of which are false positives.
You can reduce the number of false positives to prioritize security response activities by correlating events. This improves detection, reduces response times, and mitigates risk. The less time threat actors are in your environment, the less damage they can do.
Additionally, you can use log management to turn your SIEM into a proactive security tool with threat hunting.
FASTER TROUBLESHOOTING
Log management gives you better control over and insight into the processes across your environment. Log management solutions come equipped with the means for data mining. They can dig through volumes of log data to discover patterns that would remain hidden otherwise.
You can customize search and analysis to benefit from the large amount of data stored in logs with log analytics. Advanced search capabilities allow you to use both structured and unstructured logs. This way, you can gather information about specific events that help determine the root cause.
This makes it easier to:
- Recreate the timeline of problematic events
- Discover connections with other events
- Pinpoint the source of the issue.
OPTIMIZED RESOURCE USAGE
Performance monitoring can help you track resource usage. Often, someone will send a ticket to the help desk saying that an application is not responding. However, this may be the result of an overloaded server.
Centralized log management gives you visibility into performance issues and bottlenecks. By optimizing your resource use, you reduce the burden on your IT team.
For cloud environments, centralized log management can give you visibility into usage around:
- Workloads
- Applications
- Multiple environments
- Forgotten assets
Visibility into this can help you optimize cloud spending.
IMPROVED COMPLIANCE POSTURE
Log reporting summarizes the search and analysis activities with numbers and visual attributes.
You can use log reports to present your findings to someone who doesn’t have a technical background. This information can be shared with senior leadership or the Board of Directors. By reviewing these reports, they have the data to prove governance over the security program, which is important to compliance.
HOW ARE LOG MANAGEMENT TOOLS USED?
It is possible to perform the steps of log management in-house. However, it is a time-consuming process that requires a lot of customization and planning. Handling log management without logging tools is similar to programming from scratch instead of utilizing existing libraries and scripts – doable but wasteful in terms of time and resources.
Graylog’s centralized log management solution was purpose-built with IT teams in mind. We wanted them to optimize the value of their logs.
Graylog offers high-fidelity alerts and lightning-fast search capabilities. These reduce key metrics used for IT operations and security reporting. This way, you can focus your attention on the important daily activities of managing a complex IT environment.
