Pod Log Storage¶

Every container's stdout and stderr is written to disk by kubelet before Pixee's log collector ever sees it. Where those files live decides which filesystem fills up when a container logs more than expected, and whether Kubernetes notices in time to protect the node. This page explains when to move them and how to do it at install time. Moving them on a cluster that is already running is a supported migration Pixee support will walk you through.

Why the location matters¶

kubelet writes container output to a pod log directory, /var/log/pods by default, and rotates each file once it passes containerLogMaxSize (10 MiB by default). Rotation is enforced by a periodic sweep rather than a hard filesystem quota, so a container that writes faster than the sweep runs can leave files far larger than the nominal cap until the sweep catches up. Rotation limits the steady state; it does not bound the worst case.

Kubernetes has a second defense for exactly that case: kubelet evicts pods when the filesystem it calls nodefs runs low on space. nodefs is the filesystem backing kubelet's root directory, which Embedded Cluster places under --data-dir. If pod logs sit on a different filesystem from --data-dir, that defense cannot see them — pod logs can fill their volume to 100% while kubelet reports plenty of nodefs space and never evicts anything.

Moving pod logs onto the --data-dir volume closes both halves at once: the logs get that volume's headroom, and they land on the filesystem kubelet's eviction thresholds actually watch.

Do you need this?¶

Compare the filesystem holding /var/log with the one holding your --data-dir:

df -h /var/log /var/lib/embedded-cluster

Same filesystem — the default single-root layout — and there is nothing to do. Pod logs already share the filesystem kubelet watches, and relocating them within it changes nothing.

Different filesystems, with --data-dir on the larger one, and you have the gap described above. A host with a small dedicated /var and a large data volume is the common shape it takes: the large volume has room to spare, the logs go to the small one, and nothing intervenes before it is full. A full /var takes containerd, the admin console, and support bundle collection down with it. Relocate the pod log directory.

The destination must be the --data-dir filesystem, not merely a bigger one

Any other volume, however much space it has, leaves you exactly where you started. kubelet measures free space on the filesystem backing its own root directory, so pod logs on a third filesystem are invisible to eviction in the same way pod logs on /var were — the disk just fills up somewhere new. Kubernetes requires the two to share a filesystem for its accounting to hold. Pick a directory under --data-dir.

What changes about the failure mode

This does not stop a container logging too much; it changes what happens when one does. Before, pod logs filled a volume Kubernetes was not watching, and the node degraded with nothing intervening. After, they fill the volume that also holds Pixee's database and object storage — but kubelet is watching that one, so it starts evicting pods to protect the node instead of letting it wedge. Recovery becomes a Kubernetes event you can see rather than a full disk you discover afterwards. Size the --data-dir volume with that in mind.

Pixee does not set this for you. The path has to name a directory on your --data-dir volume, and the Embedded Cluster Config supports no template functions, so any value shipped in a release would be a fixed literal — right for deployments that took the default --data-dir and wrong for every other one. It stays yours to set.

Relocating at install time¶

The example throughout this page installs with --data-dir /apps/pixee/embedded-cluster, so the pod log directory it names is /apps/pixee/embedded-cluster/k0s/pod-logs — the k0s/pod-logs part is the only piece that is a convention rather than your choice. Substitute your own data directory everywhere it appears.

Write an overrides file naming a path under it:

# pod-logs-overrides.yaml
apiVersion: embeddedcluster.replicated.com/v1beta1
kind: Config
spec:
  unsupportedOverrides:
    k0s: |
      config:
        spec:
          workerProfiles:
            - name: default
              values:
                podLogsDir: /apps/pixee/embedded-cluster/k0s/pod-logs

Pass it to the installer alongside your data directory:

sudo ./pixee install --data-dir /apps/pixee/embedded-cluster \
  --overrides pod-logs-overrides.yaml

kubelet creates the directory itself on first start; you do not need to pre-create it. Keep the file — Embedded Cluster re-applies these overrides on every upgrade, which is what makes the setting durable.

Then complete Giving the log collector access below. On SELinux-enforcing hosts, also work through Checking SELinux labelling.

Giving the log collector access¶

Whichever way you set the pod log directory, the log collector needs read access to it. In the admin console, go to Config → Observability, set Pod log directory to the same absolute path, then save and deploy.

This step is not optional, and skipping it fails quietly. The collector finds pod logs through /var/log/containers, a directory of symlinks that kubelet keeps in place wherever the pod log directory points. It already has read access to /var/log and /var/lib, so a path under either of those needs nothing here. A path outside both — such as one under /apps — leaves the collector resolving symlinks to files it cannot open: log ingestion stops, and every pod stays Ready with no error to notice.

Helm deployments

On a Helm deployment you manage kubelet yourself, and the equivalent of this step is a read-only hostPath on the collector. See the commented extraVolumes and extraVolumeMounts block under victoria-logs-collector in the pixee-enterprise-server-observability chart's values.yaml — the collector lives in that sibling chart, not in pixee-enterprise-server. The mount path must equal the host path, because the symlinks kubelet writes are absolute. Setting global.pixee.observability.podLogsDir on the main chart is separate: it only reports the path in support bundles and grants no access.

Relocating on a running cluster¶

Relocating after installation is a one-time migration. It changes kubelet's configuration and takes effect only when the node reboots, so Pixee is unavailable while the host comes back, and it needs root on the node rather than cluster-admin.

Ask Pixee support for the pod log relocation runbook rather than assembling the steps yourself. It ships as a script that runs the preflight checks, makes the change, and then confirms the part that matters: that every container moved off the old directory and that log collection survived. One of its steps fails silently if taken out of order, which is the reason it is a runbook and not a list of commands.

If the host reports Enforcing, work through Checking SELinux labelling as part of it.

Checking SELinux labelling¶

Skip this if getenforce reports Permissive or Disabled. Work through it whenever it reports Enforcing, whether you relocated at install time or afterwards.

The commands below read your data directory from DATA_DIR, so set it first:

DATA_DIR=/apps/pixee/embedded-cluster   # yours, not necessarily this

Hardened images are not covered

Enforcing mode is supported on the standard SELinux policy that Enterprise Linux ships. STIG- and CIS-hardened images are neither supported nor tested, and their policies can deny access that the comparison below cannot anticipate. On one of those, check for denials after the reboot before treating the relocation as done.

Under SELinux, a new directory inherits its parent's type rather than the type kubelet's default pod log directory carries. If the container runtime is denied write access, containers start but produce no log output. Placing the directory inside --data-dir, where containerd and kubelet already read and write their own state, normally gives it a type they can use — but compare rather than assume:

# The type the new location will inherit
sudo ls -Zd "$DATA_DIR/k0s"

# The type kubelet's default location carries, for comparison
sudo ls -Zd /var/log/pods

Then, once the node has rebooted onto the new directory, check for denials:

sudo ausearch -m AVC -ts recent | grep -iE 'pod-logs|containerd|kubelet' || echo "no denials"

If denials appear, give the new tree the same type the default location carries — read it from the ls -Zd /var/log/pods output above rather than assuming a value, since it depends on your distribution's policy — and relabel:

POD_LOG_TYPE=$(sudo ls -Zd /var/log/pods | awk '{print $1}' | cut -d: -f3)
echo "relabelling to $POD_LOG_TYPE"
sudo semanage fcontext -a -t "$POD_LOG_TYPE" "$DATA_DIR/k0s/pod-logs(/.*)?"
sudo restorecon -Rv "$DATA_DIR/k0s/pod-logs"

Upgrades¶

The setting survives upgrades. Pixee ships no pod log directory of its own, so the override Embedded Cluster re-applies on each upgrade leaves your value in place, whether you set it through an install-time overrides file or afterwards.

Re-run the runbook's verification after your first upgrade anyway, and keep the Pod log directory config option in step with the path if you ever change it.

Reverting¶

The runbook has a revert command that puts podLogsDir back and reboots the node. Clear the Pod log directory setting in the admin console afterwards. Bear in mind this restores the eviction gap described at the top of this page.

Remove the setting from your install-time overrides file too

If you relocated at install time, podLogsDir also lives in the --overrides file you passed the installer, and Embedded Cluster re-applies that file on every upgrade. Undoing the change in the cluster configuration alone means the next upgrade puts it back — with the Pod log directory config option now cleared, so the collector has no access to the path kubelet returns to using, and log collection stops silently. Edit the overrides file as part of reverting, and keep it for future upgrades.