As the affordable choice for cloud computing, Google Cloud Platform (GCP) is catching up to its competitors, like AWS and Microsoft Azure. As a business, you need the speed and scalability that the cloud provides, but you want to limit your costs to ensure you hit revenue targets. With GCP, you found a digital services business partner to help you meet your business objectives, a technology that gives you the service availability you want at the speed you need. Simultaneously, you still want to use other cloud providers, or maybe you have an on-premises data center that you’re managing as well. While Google Cloud Logging and Cloud Monitoring give you some of what you need, you’re struggling to get a complete picture of how well your services perform and how secure your data is.
To manage your Google cloud logs and service performance monitoring, a centralized log management solution can help you reduce costs, improve communications across multiple teams, and streamline security incident investigations.
What are the types of logs in Google Cloud?
Since Google Cloud offers two different types of logging and monitoring tools, you need to add the basic lingo to your vocabulary so that you can understand where different logs go and what they do. Although some overlap exists between the different log types, Google has five basic categories that you need to know.
Platform logs
Written by Google Cloud services, you use these logs to understand your services. For example, VPC Flow Logs can help you debug or troubleshoot issues with virtual machine (VM) instances because they record network flow samples.
Component logs
Written by Google-provided software components, you use these logs to understand the components running on your system. For example, Google Kubernetes Engine (GKE) logs contain the metadata that your help desk can use to understand issues with VMs and data centers.
Security logs
Google provides two types of security logs that help you track down:
- Who: User or users
- What: Engaged in activities
- Where: Resources impacted, user location
- When: Day and time activity occurred
To help you trace activities occurring in your environment, Google provides two different types of logs:
- Cloud Audit Logs: administrative access and activities within Google Cloud resources used for security, auditing, and compliance
- Access Transparency: Google staff activities and access within your Google Cloud content
User-written logs
Your custom applications and services generate these logs, using one of the following methods:
- Ops Agent or Logging agent
- Cloud Logging API
- Cloud Logging client libraries
Multi-cloud and hybrid-cloud logs
These logs come from other providers, like AWS and Microsoft Azure. You can also collect these from your on-premises infrastructure.
What is Cloud Logging in Google Cloud?
Cloud Logging is the Google-supplied log-management system that enables companies to store, search, analyze, and monitor their infrastructures and third-party applications. It is part of the Google Cloud Operations Suite, previously called Stackdriver. When using the fully managed service, companies can collect log data from application components, on-premises systems, and hybrid cloud systems.
People typically use Cloud Logging to:
- Collect logs from applications
- Troubleshoot and analyze performance of services and applications
- Monitor security-related events and trends over time
Cloud Logging gives you various ways to store your logs, including:
- Cloud Logging log buckets: combining all Cloud Logging data with other data upgraded to use Log Analytics
- Cloud Storage buckets: long-term log data storage in JSON format
- Google Cloud projects: destination project’s Log Router receives and processes the logs
- BigQuery datasets: combining Cloud Logging data with other data sources to create searchable BigQuery datasets
- Pub/Sub topics: third-party integrations for JSON formatted log entries
Cloud Logging automatically routes the following logs to the _Required bucket without counting them towards the monthly log ingestion allotment:
- Cloud Audit logging: Admin Activity and System Event audit logs
- Google Workspace Admin Audit, Enterprise Groups Audit, and Login Audit logs
- Access Transparency logs
Google applies fixed usage and size limitations to Cloud Logging, including:
- Log entry: 256KB
- Audit log entry: 512 KiB
- Labels: 64 per log entry
- Length of log entry label key: 512 B
- Length of log entry label value: 64KB
- Length of query: 20,000 characters
- Oldest timestamp: 30 days in the past
Google provides a free monthly allotment for:
- Ingestion: First 50 GiB/project
- Storage: Logs retained for default retention period
What is Cloud Monitoring in Google Cloud?
Cloud Monitoring collects and stores performance information about Google Cloud services and third-party applications. With Cloud Monitoring, system administrators can create data visualizations to compare current data to previous data so that they can identify outliers and trends.
Typically, people use Cloud Monitoring to:
- Monitor service load
- Monitor website availability
- Get alerts for performance issues, like CPU load or latency
- Create resource groups to manage resources as a collection
To visualize trends, Cloud Monitoring offers the following tools:
- Google Cloud dashboards: automatically generated based on Google Cloud project’s resources
- Custom dashboards: user-defined data and visualizations, like metrics data, alerting policies, and logs stored in a Google Cloud project
- Charts: time-series data
Google provides a free monthly allotment for:
- All non-chargeable Google Cloud metrics
- Initial 150 MiB per billing for chargeable metrics
- First 1 million Read API calls pr billing account
- 1 million executions per Google Cloud project
3 challenges organizations face using Cloud Logging and Cloud Monitoring
While Cloud Monitoring supports your IT team’s observability needs, Cloud Logging supports your security initiatives. However, organizations that try to use Google-supplied products face several significant challenges.
Different data models limit correlation
Since Cloud Monitoring and Cloud Logging have different use-cases, two services use different data models.
Cloud Logging uses the following model:
- Timestamp
- Information about the entry log’s source called a monitored resource
- Payload, or message
- Name of the log, including full path of the resource followed by an identifier
Cloud Monitoring uses the following data model:
- Metric captured, like CPU utilization
- Time series, time stamp, information about the source, and meaning of metric
Since the data models are different, IT and security teams struggle to correlate activities to analyze issues that impact both operations and security, like a Distributed Denial of Service (DDoS) attack.
Limited hybrid and multi-cloud support for complex environments
The GCP provided tools offer limited visibility into complex environments. While Cloud Logging can ingest data from AWS and Azure, Cloud Monitoring provides little visibility into those services’ performance. Further, Cloud Monitoring provides very little visibility into private servers or data centers.
Increased costs as environment scales
Your costs increase in parallel with your environment’s complexity. As you create and use more resources, your Cloud Logging and Cloud Monitoring costs increase right along with you. Since each service gives you different information and insights, you need to maintain both of them, ultimately increasing costs for each.
Slow search capabilities for security
Google uses BigQuery for searching logs. Unfortunately for security teams, BigQuery’s focus is scalability, not speed. For speed in BigQuery, you want to create as precise a query as possible. By limiting the amount of data that the query runs against, you can optimize speed. Unfortunately, when investigating a security incident, you may need to run your query against terabytes of historical data which can become time-consuming.
Graylog: The single source of operations and security information
While Cloud Logging and Cloud Monitoring use different data models, they rely on the same basic information – event logs. With Graylog, you can build a single source of log information that enables observability and visibility across a complex environment. Graylog ingests all log data, no matter what service generates it, then applies a standardized data model so that you can correlate and analyze all events. Since your IT operations and security teams share the same information, they can communicate more effectively.
Further, with Graylog’s lightning-fast search capabilities, your security and IT teams can get the answers they need, even when they’re searching terabytes of data. Purpose-built for modern log analytics, Graylog gives you the two-for-one solution necessary to improve performance and reduce cybersecurity risk. Our cloud-native capabilities and out-of-the-box security content give your teams the ability to collaborate effectively, reducing service downtime and alert fatigue.
To learn how Graylog can help you save money and respond more effectively to issues, contact us today.
