Why is my vm with docker running out of space?

I recently ran into a frustrating “No space left on device” error on my VM. The culprit? Unused Docker images, containers, and volumes. Here’s a quick solution to get you back up and running:

The Problem: Docker Sprawl

Unused Docker resources consume disk space, slowing down your system. Particularly the docker/overlay2 folder. Check this out for example:

/var/lib$ sudo du -h --max-depth=1 docker
4.0K    docker/swarm
328M    docker/buildkit
188K    docker/network
8.0K    docker/tmp
367M    docker/volumes
16K     docker/plugins
63M     docker/containers
4.0K    docker/runtimes
330G    docker/overlay2
26M     docker/image
331G    docker

330 gig!!!

Lets get to work.

Run the following command to remove unused data:

docker system prune --all --volumes --force

This command removes:

• Stopped containers
• Unused images
• Volumes
• Networks

Important Notes

• Backup important data before running this command.
• Be aware of the potential risks of removing unused data.

[Update 15-10-2024]

My drive filled up again! So back to the drawing board. I did some more digging and found that logging was causing the space issue.

The Overlay2 Folder: A Hidden Space Hog

As I dug deeper into the issue, I discovered that the overlay2 folder was the culprit behind my VM’s rapidly filling disk space. For those who may not be familiar, overlay2 is a storage driver used by Docker to manage container data. It’s a powerful tool that allows for efficient storage and management of container data, but it can also be a source of trouble if not properly configured.

In my case, I had an application with verbose logging enabled. This setting was invaluable during the app’s development stage, as it provided me with detailed insights into the app’s behavior and helped me troubleshoot issues quickly. However, as the app matured, I realized that the verbose logging setting was no longer necessary, but I didn’t want to turn it off entirely.

The Problem with Verbose Logging

Verbose logging can be a double-edged sword. On the one hand, it provides valuable information that can help you debug and troubleshoot issues. On the other hand, it can generate a massive amount of log data that can quickly fill up your disk space. In my case, the log data was being stored in the overlay2 folder, which was causing my VM’s disk space to fill up rapidly.

The Solution: Configuring Docker Logging

After some research, I discovered that Docker provides a way to configure logging settings for containers. By adding a logging section to my Docker Compose file, I was able to specify the logging driver and options that would be used for my containers.

In my case, I chose to use the json-file logging driver, which allows me to store log data in a JSON file. I also specified the max-size and max-file options to limit the size of the log files and prevent them from filling up my disk space. Here’s an example of how I configured logging for my app service:

app:
  build:
    context: .
    dockerfile: Dockerfile.prod
  volumes:
    - ./Aspnetcoreapp/resources:/app/build/resources
    - ./Aspnetcoreapp/wwwroot:/app/build/wwwroot
    - ./Aspnetcoreapp/Helpers/Templates:/app/build/Helpers/Templates
  environment:
    ASPNETCORE_URLS: "http://*:5000"
    ASPNETCORE_ENVIRONMENT: Production
    Application_URL: "https://www.wehireit.com.au"
  restart: unless-stopped
  depends_on:
    - db
  networks:
    - internal
  logging:
    driver: json-file
    options:
      max-size: "10m"
      max-file: "5"

As soon as I ran docker-compose up, I immediately saw 150gb freed up!!! Talk about a win!