Juraj Kosik, an Infrastructure Security Technical Lead at Deutsche Telekom Pan-Net, has written a detailed case study of how his organization implemented Graylog to centralize log data from multiple data centers exceeding 1 TB/day. His case study provides thorough insights into real-world issues you might run into when implementing and operating a log management platform in a large-scale cloud environment.
The DT Pan-Net Security team had a goal to build a scalable platform for events collection based on Open Source, offering both log management and SIEM capabilities. Their challenge was to ensure the consumption of an unpredictable amount of data and to select technology compatible with high-volume data in motion and data at rest. One of the key requirements was solid community support and the ability to integrate with CICD workflows using APIs and message broker systems used in the company.
Before selecting the technology they planned to use for centralized log management, Juraj and his colleagues defined dos and don’ts for their operational model, knowing they wanted the following as part of their implementation:
- All components virtualized or containerized in the Openstack cloud
- Open Source used as much as possible
- DevOps
- Continuous integration/continuous delivery (CICD)
- Infrastructure as Code (IaC)
The case study walks through the organization’s thought process in selecting Graylog and how they configured the architecture and platform size for their needs. It also covers several lessons learned in the implementation, such as:
- How to set up various environments to receive the maximum benefit for CICD purposes
- How to set up load balancers for optimum performance in data replication and data flow across environments
- Best practices for parsing log data
- Scaling the infrastructure for maximum storage performance
- How to set up log aggregation for reliable and secure data consumption
- What they did to ensure zero downtime and resiliency
- The best Graylog plug-ins to replicate SIEM functionality
Check out the full case study with all the nitty-gritty details here: https://bit.ly/2J6Bxnp
