Alerting¶

Pixee Enterprise Server can evaluate threshold rules against the metrics VictoriaMetrics already stores and notify you when one fires. Alerting adds no new scrape jobs, since the data is already being collected. The only cost is the two small components that evaluate and route it, vmalert and Alertmanager.

Alerting requires local metrics, which are its data source. With metrics off there is nothing to evaluate, and the option is hidden.

Enabling Alerting¶

Embedded ClusterHelm Deployment

Navigate to the admin console
Select the Config tab
Go to the Advanced Settings section
Ensure Enable Local Metrics is checked
Check the Local alerting option
Save and redeploy the application

Alerting lives in the pixee-enterprise-server-observability chart. Add the following to that chart's values.yaml:

victoriametrics:
  enabled: true
vmalert:
  enabled: true

The main chart needs to know alerting is running so its support-bundle collectors wire themselves up. In the pixee-enterprise-server chart's values:

global:
  pixee:
    observability:
      alertsEnabled: true

Delivery¶

Slack and email are both optional and independent. Configure either, both, or neither.

With neither configured, alerting still works. Alertmanager groups and displays every firing alert in its web interface, and silences still function. Delivery is an addition on top of that, not a prerequisite.

SlackEmail

Supply a Slack incoming-webhook URL. A webhook is already bound to a single channel, so there is nothing else to configure. Alertmanager posts wherever the webhook points.

Requires outbound network access to Slack. On an air-gapped install, use email or the web interfaces instead.

Email reuses the SMTP server already configured for Authentik account recovery, so only the recipient address is alert-specific. The option appears once SMTP is enabled.

An SMTP encryption setting of None cannot deliver mail to a remote server, because unencrypted SMTP connections are refused. Choose STARTTLS or SSL.

Every alert carries the deployment's hostname, so a single Slack channel or mailbox fed by several installations can tell them apart.

Each notification carries three things: the summary naming the affected object, the description explaining what the condition means, and a link into the metrics interface showing the data that triggered it. That link's time window is pinned to the moment the alert fired rather than being relative to when it is opened, so it still shows the episode days later once the alert has resolved. Its query is scoped to the workload rather than to the individual pod, because the pod named in an alert has often been replaced by the time anyone follows the link. The link is omitted when no hostname is configured.

Credentials are delivered to Alertmanager as files mounted from a Secret, never written into a ConfigMap. Support bundles copy every ConfigMap, so a webhook or password placed there would travel with them.

Viewing Alerts¶

Both interfaces require the Observability web interfaces option in addition to alerting. With alerting on but web interfaces off, rules are still evaluated and notifications are still delivered; there is simply no UI.

Interface	Path	Use it for
vmalert	`/o11y/alerts`	The rules, their configured thresholds, and what is firing now
Alertmanager	`/o11y/alertmanager`	Grouping and routing, and creating silences

Both are protected by the same single sign-on as the other observability interfaces. This matters more here than for the read-only dashboards, because Alertmanager's API is unauthenticated and creating a silence is a write.

vmalert also logs every alert state transition to standard output, so a fired alert is collected like any other service log rather than existing only in Alertmanager's memory.

The Built-in Rules¶

The rule shapes are fixed. Thresholds marked Yes are exposed in the admin console; those marked Values file are adjustable under vmalert.thresholds in the observability chart's values but have no console option; those marked No are fixed.

Workload Saturation¶

Alert	Fires when	Default	Tunable
`PixeePodCpuHigh`	CPU use against the container's configured limit	above 50% for 1 hour	Yes
`PixeePodCpuHighNoLimit`	Absolute CPU cores, for containers with no limit set	above 4 cores for 1 hour	Yes
`PixeePodMemoryHigh`	Working set against the memory limit	above 90% for 15 minutes	Yes
`PixeePodCpuThrottled`	Share of scheduling periods in which the container was throttled	above 25% for 30 minutes	Values file

PixeePodCpuHighNoLimit exists because a container with no CPU limit has no denominator, so the percentage rule cannot see it at all. Several components run without limits and can consume an entire node.

Workload Health¶

Alert	Fires when	Default	Tunable
`PixeeContainerCrashLooping`	A container is in `CrashLoopBackOff`	continuously for 30 minutes	Yes
`PixeeDeploymentReplicasUnavailable`	A Deployment is short of replicas and its rollout has stopped progressing	15 minutes	No
`PixeeJobFailed`	A Kubernetes Job reports a failed pod	5 minutes	No

Both of the first two are written to stay quiet during a normal upgrade. Deployments do crash-loop briefly while dependencies start, measured at one to twelve minutes across two weeks of daily upgrades, so the crash-loop alert waits thirty minutes. That is comfortably past the churn and still well inside the shortest genuine failure observed. A rollout that is merely slow keeps advancing its updated-replica count, so only one that has genuinely stalled alerts.

PixeeJobFailed catches scheduled maintenance work failing silently, such as certificate renewal, cost refresh, and configuration metrics.

Node Capacity¶

Alert	Fires when	Default	Tunable
`PixeeNodeDiskFillingUp`	The node's own filesystem passes a share of its capacity	above 80% for 30 minutes	Yes
`PixeeVolumeFillingUp`	A persistent volume passes the same share of its capacity (only where the storage class enforces volume sizes)	above 80% for 30 minutes	Yes
`PixeeNodeDiskPressure`	A node reports disk pressure and begins evicting pods	10 minutes	Values file
`PixeeNodeDiskEvictions`	A pod was evicted to reclaim disk space	within the last hour	No
`PixeeNodeDiskStatsMissing`	Per-volume disk statistics stop being collected	2 hours	No

These are ordered by how much warning they give, and only the first gives any. A node that fills up evicts pods in quality-of-service order, so the workloads that go down are not necessarily the ones that consumed the disk, and a single full disk can take out most of a deployment.

PixeeNodeDiskFillingUp is the one to act on. Its default of 80% sits below 85%, which on an embedded installation is where the cluster reports disk pressure: it starts deleting unused container images, and evicts pods if that does not free enough. A threshold set at 85 arrives together with the pressure it was meant to warn about. It reads the node's own filesystem, which is the disk the image store grows on and the disk eviction is measured against. On an embedded installation the cluster also deletes any image left unused for seven days, so the image store does not creep up toward 85% between deploys; the Node Disk dashboard shows both kinds of cleanup (see Metrics & Dashboards).

PixeeVolumeFillingUp is its counterpart for persistent volumes, and the two are separate because on most storage classes they are separate disks: a volume can fill while the node has room to spare, and the workload writing to it gets ENOSPC while every node-level reading stays green. Both use the same threshold. On an embedded installation every volume is a directory on the node's own disk with no size limit, so a volume's figures are the node disk's; PixeeVolumeFillingUp is not installed there, and PixeeNodeDiskFillingUp covers it. A Helm installation controls this with global.pixee.storageSizeEnforced on the observability chart.

PixeeNodeDiskStatsMissing exists because a missing measurement is invisible in a way a broken one is not. If the node filesystem statistics stop arriving, PixeeNodeDiskFillingUp has nothing to evaluate and stays silent however full the disk gets — which looks exactly like a disk with room to spare. An installation that supplies its own metrics scrape configuration rather than the one shipped here needs a job collecting the kubelet's cAdvisor endpoint, or this warns continuously.

PixeeNodeDiskEvictions distinguishes the case that cannot recover on its own. A node sitting at 85% with a large store of unused images is healthy — the next cleanup reclaims them. A node at 85% whose disk is full of images still in use has nothing to reclaim, so it evicts, and keeps evicting, until space is freed by hand. The log and trace stores have configurable disk ceilings; unused container images are the other large consumer.

Platform Dependencies¶

Alert	Fires when	Default	Tunable
`PixeeCertificateExpiringSoon`	A certificate is approaching expiry	fewer than 14 days remaining	Yes, with a requested certificate
`PixeeScrapeTargetDown`	A metrics target cannot be scraped	15 minutes	No
`PixeeScrapeJobNoTargets`	A scrape job matches no workload at all	1 hour	No

PixeeScrapeTargetDown is the one that keeps the rest honest. If a target stops being scraped, every alert derived from it goes blind rather than green.

PixeeScrapeJobNoTargets covers the case that one cannot see. A target that is down reports as down; a job whose selector matches nothing produces no reading at all, so there is nothing to report as down, and it is indistinguishable from a job with nothing to do. This usually means a selector no longer matches how a workload is labelled.

There are deliberately no database rules. The deployment runs a single Postgres instance, so there is no replica to fall behind, and the metrics that would describe connection and lock contention are not currently collected.

PixeeCertificateExpiringSoon covers only a certificate this deployment requested. The expiry reading comes from cert-manager, which tracks the certificates it issues; an uploaded certificate is stored as an ordinary Secret that nothing here reads, so its expiry remains yours to monitor. The threshold option appears in the admin console only when a certificate is being requested.

Alert History¶

Alertmanager holds only what is active now. Once an alert resolves it is gone from its API and its interface. History lives in two other places, both populated automatically.

The Alerts dashboard, under Metrics → Dashboards, is the quickest view: a timeline of every alert that fired, when it cleared, which workload it was about, and which rules fire most often. It reads the ALERTS series that vmalert writes to VictoriaMetrics, so it covers the full metrics retention.

Its last row counts how many times each rule started firing, which is the measure of whether a rule is noisy: a rule that fires once and stays firing for a day is one event, while a rule that fires and clears twenty times is twenty interruptions. It deliberately counts transitions into the firing state rather than into the pending state. An alert that goes pending and clears before its for window elapses never notified anyone, and counting those would report routine deployment churn as noise.

Every panel is a count of what was firing at each moment, so the lines rise as alerts fire and fall as they clear. A cumulative measure such as minutes spent firing over a trailing day only climbs until the event ages out twenty-four hours later, which on a graph reads as a rule that never stops firing; for questions of that shape, query the series directly with the range you want.

For ad-hoc questions, query the same series directly in the metrics interface:

ALERTS{alertstate="firing"}                        # everything currently firing
ALERTS{alertname="PixeeContainerCrashLooping"}     # one rule's whole history
count_over_time(ALERTS{alertstate="firing"}[7d])   # which rules are noisiest

Switching that view to its Table tab gives a plain table of active alerts and their labels.

Alert state transitions are also logged. vmalert writes every INACTIVE → PENDING → FIRING → INACTIVE move to standard output, so the transitions are searchable alongside every other service log, with the full label set and the reason the alert cleared. Log and metric retention are configured separately and may differ, so check both before concluding that an older event was never recorded. In general the dashboard is the better place for questions of when and how often, and the logs are the better place for detail.

Neither depends on Alertmanager being up, which is the point: its own view of the world does not survive a restart.

Reducing Noise¶

A critical alert suppresses warning-level saturation alerts for the same pod, so a workload that is down reports once rather than several times.

To quiet an alert you already know about, create a silence in the Alertmanager interface rather than changing configuration. Silences are the intended tool, because they are scoped, they expire, and they record who created them and why.

The repeat interval controls how often an alert that is still firing notifies again. It is separate from how long a condition must hold before it fires the first time.

Note

A silenced alert is indistinguishable from a healthy one everywhere except Alertmanager. Support bundles collect active silences for exactly this reason. When reviewing one, check them before concluding that nothing was wrong.

Adding Your Own Rules¶

Site-specific rules can be appended without modifying the built-in set, using vmalert.extraRuleGroups in the observability chart's values. Entries follow the standard alerting rule format and are added verbatim.

Support Bundles¶

When alerting is enabled, support bundles capture the rules and their last evaluation, the alerts firing at the time of collection, and any active silences.

The bundle analysis names each alert individually rather than reporting that something fired: every rule appears as its own check, so a bundle shows exactly which alarms were active and what each one means, without anyone opening the raw collector output. The analysis also reports whether vmalert was reachable, whether any rule failed to evaluate, and whether a silence was suppressing anything.

A rule that fails to evaluate is reported as a failure rather than a warning. That rule is blind, not quiet, and an empty alert list is not evidence that nothing is wrong.

Disk exhaustion is the exception to all of this: it is checked in every bundle whether or not alerting is enabled. The analysis reads the cluster's own event record for failed image cleanup and for node disk pressure, and reports any evicted pods still present. Alerting tells you while it is happening; these tell you afterwards, from a bundle, on a deployment that never turned alerting on.