How to Run Docker in Verbose Mode for Debugging

As a Docker power user, you likely know that running Docker with the -d flag detaches the container and runs it in the background. This is convenient for most containerized applications. However, when your Docker containers or images aren‘t working as expected, debugging them can be tricky without visibility into what‘s happening under the hood.

That‘s where Docker‘s verbose logging mode comes in handy. By running Docker in verbose mode, you can view detailed logs of everything happening in the Docker daemon. This enables you to more easily identify issues so you can get your containers and images running properly again.

In this comprehensive guide, I‘ll explain:

• How the Docker daemon and containers work
• When to use Docker‘s verbose/debug mode
• How to enable verbose logging in Docker
• How to read the verbose Docker logs
• Docker verbose logging best practices

So if you‘re ready to level up your Docker troubleshooting skills, let‘s get started!

Understanding the Docker Daemon

Before we jump into using Docker‘s verbose logging, it helps to understand what the Docker daemon is and how it works.

The Docker daemon (dockerd) is a persistent background process that manages Docker objects like images, containers, networks, and volumes. It provides a server-like interface for clients like the Docker CLI to interact with these objects.

For example, when you use a command like docker run, you are communicating with the Docker daemon process, which then goes and starts the container you requested.

By default, the Docker daemon runs silently in the background with minimal logging. This helps it run efficiently without generating unnecessary logs.

But when things aren‘t working properly, we need more visibility from the daemon. Verbose logging gives us that debugging insight.

When to Run the Docker Daemon in Verbose Mode

The most common scenario for running Docker in verbose mode is when you are troubleshooting problems with your Docker environment.

Here are some examples of when it can be helpful to enable verbose Docker logging:

• Containers won‘t start: If docker run or docker start are failing with cryptic errors, verbose logs from the daemon could reveal what‘s happening behind the scenes.

• Images won‘t build: If you have a Dockerfile that fails partway through, verbose daemon logs show each step attempted during the build process.

• Networking issues: Problems with container connectivity, DNS resolution, or port mapping can be debugged by inspecting parameters and progress from verbose networking logs.

• Volume mount issues: Trouble mounting host directories or external volumes into containers may be diagnosable from daemon logs.

• Performance problems: Verbose CPU, memory, and I/O stats could help analyze resource contention, starvation, or other bottlenecks.

In summary, anytime a Docker command isn‘t working as expected and you want to understand what the daemon is doing in more detail, consider running Docker in verbose/debug mode.

How to Run the Docker Daemon in Verbose Mode

There are two primary ways to enable verbose logging from the Docker daemon:

1. Start the daemon manually with verbose flags
2. Edit the daemon configuration file

Let‘s go through both options.

Start Docker Daemon Manually with -D

The easiest way to quickly troubleshoot an issue is to start the Docker daemon manually with debug flags:

dockerd -D

The -D flag tells Docker to run in debug mode, which enables more verbose logging.

If you already have an instance of the daemon running, stop it first:

sudo systemctl stop docker 

Then run the above dockerd -D command to start a new daemon process manually.

This may generate a lot of output, so be sure to redirect it to a file you can analyze:

dockerd -D > /var/log/docker-debug.log 2>&1

Whenever you are done troubleshooting, be sure to kill the daemon process and restart the standard system-managed instance:

killall dockerd
sudo systemctl start docker

This is the quickest way to enable verbose logging. But for persistently debugging problematic daemon issues, editing the configuration file is better.

Edit the Docker Daemon Configuration

For ongoing verbose logging, you‘ll want to edit the Docker daemon configuration file and restart the standard daemon process.

The default location of this configuration file depends on your Linux distribution:

• Ubuntu/Debian: /etc/docker/daemon.json
• RHEL/CentOS: /etc/docker/daemon.json
• Fedora: /etc/containers/registries.conf

If the file doesn‘t already exist, go ahead and create it. Then add the following entry:

{
  "debug": true
} 

Save and close the file.

Next, restart Docker to load the new configuration:

sudo systemctl restart docker

The daemon will now run in debug mode persistently across daemon restarts.

To disable, simply edit the config file again and set "debug": false.

This provides an easy way to enable verbose logging without having to manually start the daemon each time.

Reading Verbose Docker Logs

Once verbose logging is enabled, you can access the Docker daemon debug logs at:

• Ubuntu/Debian – /var/log/daemon.log
• RHEL/CentOS – /var/log/messages
• Fedora – journalctl -u docker.service

Search these logs for the time frame when you were running the failing Docker command. This should provide much more detail about each daemon operation related to images, containers, volumes, etc.

Look for errors, stack traces, or any unexpected behavior in the logs to try diagnosing the problem.

Here are some key things to look for:

• Connection issues – Problems interfacing with container runtimes, storage drivers, network plugins, etc.
• Permission errors – The daemon running as the wrong user without access to sockets, config files, or volumes.
• Resource faults – Out of disk space, memory exhaustion, TCP port conflicts, missing device files, etc.
• Configuration problems – Invalid or incompatible daemon config values.
• Image issues – Errors pulling image manifests and layers during docker pull or docker build.
• Networking failures – Problems creating network namespaces, managing iptables rules, or configuring interfaces.

Pay attention to the first error logged to identify the initial cause of the failure. Then assess subsequent errors to understand how the issue compounded across different components.

This should help you pinpoint the specific problem and solution.

Docker Verbose Logging Best Practices

Here are some tips for getting the most out of Docker‘s verbose logging when debugging daemon issues:

• Log selectively – Only enable debug logging temporarily when needed. Leaving it on all the time can consume disk space and impact performance.

• Isolate environments – If possible, reproduce the problem with a small-scale test system instead of large complex ones to simplify logs.

• Capture early context – Start logging before running the problem command to capture additional context like daemon start up configuration.

• Inspect interactively – Go through logs interactively via tail -f rather than post-mortem to catch errors in real-time.

• Understand log flow – Get familiar with the various components involved – containerd, runc, CNI plugins, storage drivers, etc. – to better interpret log flow.

• Correlate user errors – Review CLI usage and client logs before inspecting daemon logs to distinguish client vs server errors.

• Interpret timeouts – Differentiate connectivity timeouts from processing timeouts to locate sources of delay.

• Reproduce reliably– Reliably reproduce the problem scenario to isolate deviations in verbose logs between working vs failed outcomes.

Getting into these debugging habits will help you become a Docker troubleshooting expert!

Conclusion

I hope this guide has shed some light on how to take advantage of Docker‘s built-in verbose logging capabilities.

When containers and images aren‘t operating correctly, your first steps should be:

1. Enable daemon debug mode
2. Reproduce the failure scenario
3. Inspect the verbose logs

This will reveal much more about internal daemon operations compared to standard baseline logging.

Look for discrepancy in the verbose logs compared to a working scenario. Try to pinpoint the first errors that ultimately caused the failure.

As you get more comfortable interpreting verbose Docker logs, you‘ll be able to debug and resolve daemon issues much faster. This will help minimize application downtime and deployment delays for yourself, your team, and users.

So the next time you run into problems with containers or images, remember to run Docker in verbose mode as the first troubleshooting step!