The future of enterprise IT stacks is the cloud. In fact, according to a 2019 Gartner post, when we say “cloud infrastructure,” 81% of people really mean multi-cloud. Considering the analyst took this survey prior to the pandemic, we can safely assume that the number of companies with multi-cloud stacks is probably higher than this. Companies choose a multi-cloud strategy for a lot of reasons, including making disaster recovery and migration easier.
A multi-cloud stack is the three-flavored banana split of IT. Each of the Big Three – AWS, Microsoft Azure, and Google Cloud Platform (GCP) – each have their own distinct “flavor.” Azure, for example, is an enterprise favorite because it supports the enterprise 0365 purchases. AWS offers better computer and storage options. GCP has live migrations for virtual machines. To get visibility into your environment, you want a centralized log management that supports your multi-cloud strategies.
WHAT ARE THE MULTI-CLOUD ENVIRONMENT LOG MANAGEMENT CHALLENGES?
If something is easy, everyone would do it. Log management for multi-cloud can be difficult, which is why a lot of companies struggle. In fact, every company with multi-cloud strategy faces at least one of the following three log data challenges.
TOO MANY FORMATS
If you were just managing the providers’ clouds, you’d only be looking at three different formats. However, you also connected applications to each cloud. In reality, the average enterprise uses 1,295 cloud services.
With this many data sources connected to your multi-cloud ecosystem, you can’t expect them to all use the same format. You have the potential for at least four potential log formats – CEF, Windows Event Log, JSON, and Syslog. If you’re trying to correlate information as part of a log analysis, this becomes a nightmare. Aggregating these multiple forms of log types and formats will allow you to normalize them. Once logs are normalized you can create a common path to analyze this valuable information. This standard approach in the formatting and enrichment of your log data will give you the ability to create high fidelity alerts and notifications for specific events in your logs.
TOO MANY TOOLS
Each provider offers its own tool. At first, this is great because it’s included in the subscription cost. If you’re paying for something, you don’t want to add redundant technologies to your stack.
Unfortunately, comparing metrics for each cloud usinge vendor-supplied tools gets messy. Cross-referencing each tool by hand adds time to your processes. It also leaves you open to human error risks. Multi-cloud monitoring for security means risk can be the difference between being breached and proactively mitigating a risk.
TOO MANY DIFFERENT DATA POINTS
As if having different formats and tools wasn’t bad enough, each cloud service provider collects similar data but in different ways. For example, you can be dealing bringing in data from different cloud (e.g., web-based project management tools, business productivity tools, HR applications, ITOps ticketing systems, etc.).
TO MUCH TIME
Whether you’re threat hunting or monitor system resources, using multiple tools becomes too time-consuming. The more time threat hunting takes, the longer the dwell time is. If you can’t compare data effectively as part of your system resource management, you can’t provide valid metrics.
HOW DOES CENTRALIZED LOG MANAGEMENT HELP SOLVE THESE PROBLEMS?
Centralized log management solutions bring all your log files together in one place. With all information aggregated in a central location you can normalize your to create common events from different sources and you get a more holistic data set for searching and reporting. Additionally, a central log management solution can create a unified event log format that gives you the ability to make sense of data collected beyond just your network or machine system messages. With all data standardized in a single location, you can make better correlations and save time.
5 SUGGESTIONS FOR AN EFFECTIVE MULTI-CLOUD LOGGING STRATEGY
Creating an effective log management strategy for your multi-cloud strategy can be challenging. In order to get the most out of your purchase, you want to plan in advance.
1. FIND THE RIGHT SOLUTION
Starting any project means finding a solution that gives you the services you need at a price you can afford. For a multi-cloud log management solution, you want a one that can:
- Translate all logs into one format
- Aggregate data in real-time
- Collect API call logs
- Manage inconsistent identity controls and markers
- Identify instances using time and events
- Be able to manage changes in processing power and storage needs
- Integrate directly with 3rd Party tools (e.g., ticketing systems)
2. PLAN A LOGGING ARCHITECTURE
When you move to the cloud, you probably also added a lot of Software-as-a-Service (SaaS) applications because they make collaboration easier. When you set up your centralized log management, your ITOps team will use them to want to troubleshoot issues like performance or failed logins, your DevOps team will use them to debug issues in your applications, your Security Ops team will use them for security analytics. As a result you will want to take into account how you plan to manage things like:
(Note that these are examples and not an exhaustive list.)
System Logs: monitor server health while giving visibility into security and should capture:
- Device
- Service
- Operating system module
Security Logs: monitor authentications, users, permissions, changes to privileges so you can detect, investigate threats, etc.
Application logs: consist of informational, warning, error, or critical event logs help understand operational status so you can find bug or other problems and should capture:
- General execution requests
- General execution times
- COnfiguration setting
Database logs: monitor database access and speed and should capture:
- General query data
- Slow query data
- Error
- Resource status
- Database load
- QPS
Web (Server) Access Logs: document success or failure when processing a request and should capture:
- Client IP address
- Agent software
- Requested URL
- Response code
Event logs:give visibility into other cloud resources like browsers or third-party software and should capture:
- User login
- Failed login
3. DECIDE HOW YOU WANT TO USE THE INFORMATION
You can collect and many companies do all of your logs. However, your strategy for what to monitor needs to cover the right logs. In other words, you choose what you want to monitor. This means that to get the most benefit from your centralized log management solution, you need to think about how you want to use your logs and then collect the data that gives you the most insight. When you’re planning your multi-cloud centralized log management strategy, try considering:
(Note that these are examples and not an exhaustive list.)
- Event logs for a variety of different activities
- Environments that you need to monitor in real-time
- KPIs you want to measure
- Infrastructure performance and availability
- Threat intelligence
- etc.
4. DON’T FORGET ABOUT COMPLIANCE
While no one loves discussing compliance, event logs are the primary documentation used to meet the requirements. This means that you need to think about what data your auditors need and how you can meet data retention requirements for regulatory compliance.
5. GET THE SPEED YOU NEED
The reason you bring everything together in one place is to save time. Your security team needs a tool that makes their threat hunting efficient and effective. Your network operations team needs to investigate wireless DHCP usage. Your centralized log management solution should let you build search workflows that you can save and share.
Some things to consider when looking at search workflow capabilities include the ability to:
- Set up pipelines
- Chain queries together
- Limit search to relevant data
- Save workflows for later use
- Set multiple parameters
- Share workflows
- Keep Workflows Private or Secured
GRAYLOG CENTRALIZED LOG MANAGEMENT FOR MULTI-CLOUD ENVIRONMENTS
Graylog’s centralized log management solution makes managing a multi-cloud environment easier. Graylog Extended Log Format (GELF) introduced extractors that take the important data from event log message fields, no matter what form they are originally in. This gives you a standardized log format which makes managing information across your multi-cloud environment easier. Both our open source and enterprise services offer this capability.
Our Search Workflows let you create shared searches so that team members don’t need to start from scratch every time. You can also drill-down into the charts that the workflow creates and turn them into a dashboard.
Our Enterprise solution offers an additional feature, Views. You can use prebuilt Views or create ones unique to your needs. For example, our pre-built DNS: overview View lets you add data showing the DNS servers in use, how much memory they used, and what domains were requested most often. In a multi-cloud environment, Views gives you easy visibility into a complex architecture so that you can search for and analyze data faster.
