Control Groups (cgroups) — The Resource Police

Imagine you’re running a buffet restaurant.

You have three types of customers:

• Regular customers
• VIP customers
• Kids

Without any rules, a single hungry customer could pile their plate with all the crab legs, leaving nothing for anyone else. One person could monopolize the dessert bar.

So you implement portion control:

• Regular customers: Maximum 2 plates
• VIP customers: Maximum 4 plates
• Kids: Maximum 1 plate

You also have servers monitoring each section:

• “Sorry, you’ve reached your crab leg limit.”
• “You can have more salad, but no more steak.”
• “You’ve had your dessert portion.”

This is what cgroups do for system resources.

Each process (or group of processes) gets assigned to a cgroup. That cgroup has limits:

• Maximum CPU usage: “You can use up to 50% of one CPU core”
• Maximum memory: “You can allocate up to 512MB of RAM”
• Maximum I/O: “You can read/write up to 10MB/s to disk”

If a process tries to exceed its limits, the kernel enforces them. The process gets throttled, blocked, or even killed.

Cgroups are organized in a hierarchy, like a directory tree.

The cgroup hierarchy

At the top level, you have controllers (also called subsystems):

• cpu: Controls CPU time allocation
• memory: Controls memory usage
• blkio: Controls block I/O (disk access)
• net_cls: Controls network packet tagging
• pids: Controls number of processes

Under each controller, you can create groups. Each group has:

• Limit settings (configured by an administrator)
• Processes assigned to it
• Statistics about resource usage

Example cgroup structure:

/sys/fs/cgroup/
├── cpu/
│   ├── docker/
│   │   ├── <container-id-1>/
│   │   │   ├── cpu.cfs_quota_us    (CPU limit)
│   │   │   └── cpu.cfs_period_us
│   │   └── <container-id-2>/
│   └── user.slice/
└── memory/
    ├── docker/
    │   ├── <container-id-1>/
    │   │   ├── memory.limit_in_bytes  (Memory limit)
    │   │   └── memory.usage_in_bytes  (Current usage)
    │   └── <container-id-2>/
    └── user.slice/

How limits are enforced:

CPU limits:

• If you set a CPU limit of 50%, the process gets 50% of one CPU core’s time.
• The kernel scheduler throttles the process, pausing it periodically to enforce the limit.
• The process runs normally until it hits the limit, then gets paused for the rest of that time period.

Memory limits:

• If you set a memory limit of 512MB, the process can allocate up to 512MB.
• If it tries to allocate more, the kernel refuses the allocation (malloc fails).
• If memory pressure is high, the kernel’s OOM (Out of Memory) killer may kill the process.

I/O limits:

• If you set a disk read limit of 10MB/s, the process can read up to that rate.
• If it tries to read faster, the kernel throttles the I/O operations.

When you start a container with resource limits, Docker creates cgroups and assigns the container’s processes to them.

Example 1: Memory limit

docker run -d --name limited --memory=512m nginx

Docker creates a cgroup under /sys/fs/cgroup/memory/docker/<container-id>/ and sets:

memory.limit_in_bytes = 536870912  # 512MB in bytes

All processes in this container are assigned to this cgroup. If they try to use more than 512MB total, the kernel enforces the limit.

Example 2: CPU limit

docker run -d --name limited --cpus=0.5 nginx

Docker creates a cgroup under /sys/fs/cgroup/cpu/docker/<container-id>/ and sets CPU quota to allow 50% of one CPU core.

Example 3: Combined limits

docker run -d \
  --memory=1g \
  --cpus=2 \
  --blkio-weight=500 \
  nginx

Docker creates cgroups for memory, CPU, and block I/O, each with their respective limits.

Let’s see cgroups in action.

Start a container with limits:

docker run -d --name demo --memory=512m --cpus=0.5 nginx

Find the container’s cgroup:

First, get the full container ID:

CONTAINER_ID=$(docker inspect demo --format '{{.Id}}')
echo $CONTAINER_ID

View memory cgroup:

cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.limit_in_bytes

This shows the memory limit in bytes (should be ~536870912 for 512MB).

View current memory usage:

cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.usage_in_bytes

This shows how much memory the container is currently using.

View CPU cgroup:

cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.cfs_quota_us
cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.cfs_period_us

The quota divided by the period gives you the CPU limit. For 0.5 CPUs:

• quota = 50000
• period = 100000
• 50000 / 100000 = 0.5 (50% of one CPU)

Test the memory limit:

Run a command inside the container that tries to allocate too much memory:

docker exec demo bash -c 'tail /dev/zero'

This command tries to read infinite data into memory. The container will be killed when it exceeds the 512MB limit. Check with:

docker inspect demo --format '{{.State.OOMKilled}}'

If it says true, the container was killed by the Out-of-Memory killer due to the cgroup limit.

Without cgroups, containers would be isolated (thanks to namespaces) but not safe.

One container could:

• Starve all others of CPU
• Consume all available RAM
• Saturate disk I/O
• Create millions of processes (fork bomb)

Cgroups ensure fair resource sharing and protect the host system.

In production environments, setting resource limits is critical:

• Prevents runaway containers from taking down the entire host
• Ensures predictable performance for critical services
• Enables multi-tenancy (multiple users’ containers on the same host)

Cgroups are the reason you can confidently run dozens or hundreds of containers on a single server.

---