Skip to content

Traces

Pixee Enterprise Server includes VictoriaTraces for distributed trace collection. When local metrics is enabled, trace telemetry from the analysis service is automatically collected and stored, allowing you to inspect individual request spans and trace IDs.

Enabling Traces

Traces are automatically collected when local metrics is enabled. See Enabling Local Metrics for instructions.

Trace Sampling

The analysis service head-samples the traces it starts, so only a fraction of analyses produce a trace. The default is chosen to keep trace volume manageable, since the spans that record LLM prompts and completions are by far the largest.

Set the sampling ratio to 1.0 to capture every trace — useful on a quiet environment when verifying that traces arrive end to end. Lower it on a busy environment to trade trace fidelity for retention on the traces volume. Valid values run from 0.0 (no traces) to 1.0 (all traces) — a fraction, not a percentage; the analysis service rejects anything outside that range at startup. Sampling is applied when a trace starts, so it governs every destination the spans are exported to, and it takes effect whenever telemetry is enabled.

Navigate to the admin console, select the Config tab, then go to the Advanced Settings section. Set Analysis trace sampling ratio to a value between 0.0 and 1.0. Leave it blank to keep the analysis service default.

Add the following to your values.yaml:

analysis:
  tracesSampleRatio: "1.0"

Omit the value to keep the analysis service default.

Accessing Traces VMUI

You can access the VictoriaTraces web interface to search and explore traces. You can either expose it through the gateway or use port forwarding.

Option 1: Enable VMUI Web Interface (Gateway)

Enable the VMUI web interface to access the traces query interface directly through your browser.

To enable the VictoriaTraces web interface in Embedded Cluster deployments:

  1. Navigate to the admin console
  2. Select the Config tab
  3. Go to the Advanced Settings section
  4. Check the Enable Traces VMUI web interface option
  5. Save and redeploy the application

Once enabled, access the traces interface at:

https://<your-domain>/o11y/traces/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Only enable this option if your deployment is within a trusted network or you have implemented external authentication.

To enable the VictoriaTraces web interface in Helm Deployment, add the following to your values.yaml:

eg:
  enabled: true
victoriatraces:
  gateway:
    enabled: true

The route attaches to the Gateway the main chart renders, so eg.enabled and global.pixee.domain must match what you set there.

Then upgrade your deployment:

helm upgrade pixee-enterprise-server ./charts/pixee-enterprise-server \
  -f values.yaml \
  -n pixee-enterprise-server

Once enabled, access the traces interface at:

https://your-domain.com/o11y/traces/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Consider implementing external authentication or only enable this in trusted network environments.

Option 2: Port Forwarding

If you prefer not to expose the VMUI through the gateway, you can use port forwarding for temporary access.

Step 1: Create SSH tunnel from your local machine

ssh -L 10428:localhost:10428 pixee@<your-hostname>

Step 2: Set up port forwarding

In the SSH session, run:

sudo ./pixee shell
kubectl -n kotsadm port-forward pixee-enterprise-server-traces-server-0 10428:10428

Step 3: Access the traces interface

Open your browser and navigate to:

http://localhost:10428/o11y/traces/select/vmui/

Step 1: Port forward to VictoriaTraces

kubectl port-forward svc/pixee-enterprise-server-traces-server 10428:10428 -n pixee-enterprise-server

Step 2: Access the traces interface

Open your browser and navigate to:

http://localhost:10428/o11y/traces/select/vmui/

Querying Traces

VictoriaTraces provides a Jaeger-compatible query API for searching traces. In the VMUI, you can:

  • Search by service name — Filter traces from specific services (e.g., pixee-analysis-service)
  • Search by trace ID — Look up a specific trace using its trace ID
  • Filter by duration — Find slow requests by setting minimum/maximum duration
  • Filter by tags — Search for traces with specific span attributes
  • Filter by deployment — Every span carries deployment.environment.name, set to this installation's external host, including a non-standard port when one is configured, or to the error-reporting environment name when that is set. The analysis and platform services report the same value, so it also identifies spans that belong together when correlating a trace with error reports.

Querying Traces Programmatically

Beyond the VMUI, you can query traces over the Jaeger-compatible HTTP API with the Pixee CLI, which authenticates you through your identity provider and supplies a per-user bearer token:

TOKEN=$(pixee auth token --server https://pixee.example.com)
# List services, then search recent traces for one
curl -s -H "Authorization: Bearer $TOKEN" \
  https://pixee.example.com/o11y/traces/select/jaeger/api/services
curl -sG -H "Authorization: Bearer $TOKEN" \
  https://pixee.example.com/o11y/traces/select/jaeger/api/traces \
  --data-urlencode 'service=pixee-analysis-service' --data-urlencode 'limit=20'

See Programmatic Access with the Pixee CLI for the one-time pixee auth login step.

Trace Retention and Disk Usage

The trace store is bounded the same way the log store is, and by whichever limit it reaches first: the retention period, and a disk ceiling that drops the oldest per-day partitions. On an embedded cluster the volume is a directory on the node's root disk with no quota, so the ceiling is the bound that actually holds.

Traces maximum disk usage in the admin console is the ceiling. It defaults to 80% of the volume request (8Gi for the default 10Gi volume) and can be changed at any time. The volume request behind it is fixed at install time and not shown in the console — see Log Retention and Disk Usage for why.

Traces are far smaller than logs in practice: a busy internal environment holds under 0.5 GiB of them.

Both values are yours to set, and on a real storage class the ceiling must stay below the volume — the retention preflight fails when it does not:

victoriatraces:
  server:
    retentionDiskSpaceUsage: "8GiB"   # ~80% of the volume below
    persistentVolume:
      size: "10Gi"

Growing the volume on an existing release takes the PVC-expansion procedure described under logs; the ceiling changes with an ordinary upgrade.

Write the ceiling in GiB (8GiB) or as a bare number of GiB (8). 8Gi is a Kubernetes unit that VictoriaTraces does not accept, and the preflight rejects it.

VictoriaTraces Resources