Docker containers have solved many technical challenges, but also introduced a few new ones.
Log collection from Docker containers has been difficult because containers do not persist for a long time, and thus don’t leave a reliable trail of historical information.
As a result, tasks that rely on log data, such as application debugging, have become more difficult.
sender as a logging driver to the Docker 1.8 core platform.
The GELF logging driver enables a container to send STDOUT and STDERR messages formatted as GELF messages to a remote, centralized system like Graylog.
The GELF logging driver replaces log forwarders or manual methods for collecting logs inside or outside of a container.
As log forwarder containers are removed, users will see a decrease in configuration requirements, image bloat, and CPU/memory utilization.
COLLECTING CONTAINER LOGS WITH DOCKER 1.8
SETUP
Start with an existing Graylog instance or use our pre-configured Docker image.
docker run -t -p 9000:9000 -p 12201:12201 graylog2/allinone
Next, create a UDP GELF input through the Graylog web interface.
Now Graylog is listening for your data and you are ready to send logs
from any container to Graylog.
docker run –log-driver=gelf –log-opt gelf-address=udp://<graylogip>:12201 busybox echo Hello Graylog</graylogip>
USAGE
Additional context is very useful for figuring out which application and service was impacted if there was an issue.
Below is a list of additional log message fields included in every log message from Docker:
These additional fields combined with the search functionality of Graylog, gives the user the ability to analyze the output of containers across multiple host systems.
Figuring out what went wrong with a Docker container is now much easier, you can search by host, image or even the container itself.
Below is an example of searching by container ID.
Since we can collect all of the log messages centrally, it’s a simple process to search across all log messages from all Docker hosts and containers.
Another example would be to display all logs from a Redis cluster running in multiple Docker containers.
Assume we run a cluster with more than one Redis instance and we don’t know the exact container IDs.
It is possible to simply search for the image name because all containers were built from the
In the same way all other fields can be used to find the necessary information in a tangle of log data.
To do further breakdowns of log data, use Graylog’s quick values feature.
With one click a user can analyze every field data from Docker.
This can be used to get an overview of most used images or commands executed by users.
Additionally Graylog provides statistical data for numerical fields.
With this it is possible to get mean, min, max or standard derivation for certain log data.
DASHBOARD
All searches from above can be used to create widgets and dashboards in Graylog to provide a better overview for Docker admins.
Just click on the top-right to add the data in question to a Docker dashboard.
This dashboard can then be shared with other users or put on wall-mounted display.
ALERTING
Graylog is able to send out alerts based on certain log events. For example, if we want to send out alerts whenever a Tomcat container runs out of memory, we can create a simple stream looking for ‘OutOfMemoryError’ in the Tomcat container message.
Based on the type of error and the number in a given time period we can send out alert emails, notifications or other kinds of alerts to different systems i.e. Slack, PagerDuty
etc.
CONCLUSION
Log management used to be a challenge with Docker, but it’s now a lot easier since native GELF support is integrated into the core Docker platform.
Graylog users will also get the benefit of advanced log management features working right out of the box like alerting, log retention and dashboards.
System Administrators and Application Developers are now able to quickly zero in on and analyze problems in their Docker environments without the additional complexity and configuration/management overhead of the past.
Tell us about your experience! We’d love to hear how your Docker logging experience changes with this new addition to the core Docker platform.
