Skip to content

Traces

Pixee Enterprise Server includes VictoriaTraces for distributed trace collection. When local metrics is enabled, trace telemetry from the analysis service is automatically collected and stored, allowing you to inspect individual request spans and trace IDs.

Enabling Traces

Traces are automatically collected when local metrics is enabled. See Enabling Local Metrics for instructions.

Trace Sampling

The analysis service head-samples the traces it starts, so only a fraction of analyses produce a trace. The default is chosen to keep trace volume manageable, since the spans that record LLM prompts and completions are by far the largest.

Set the sampling ratio to 1.0 to capture every trace — useful on a quiet environment when verifying that traces arrive end to end. Lower it on a busy environment to trade trace fidelity for retention on the traces volume. Valid values run from 0.0 (no traces) to 1.0 (all traces) — a fraction, not a percentage; the analysis service rejects anything outside that range at startup. Sampling is applied when a trace starts, so it governs every destination the spans are exported to, and it takes effect whenever telemetry is enabled.

Navigate to the admin console, select the Config tab, then go to the Advanced Settings section. Set Analysis trace sampling ratio to a value between 0.0 and 1.0. Leave it blank to keep the analysis service default.

Add the following to your values.yaml:

analysis:
  tracesSampleRatio: "1.0"

Omit the value to keep the analysis service default.

Accessing Traces VMUI

You can access the VictoriaTraces web interface to search and explore traces. You can either enable web access via ingress or use port forwarding.

Option 1: Enable VMUI Web Interface (Ingress)

Enable the VMUI web interface to access the traces query interface directly through your browser.

To enable the VictoriaTraces web interface in Embedded Cluster deployments:

  1. Navigate to the admin console
  2. Select the Config tab
  3. Go to the Advanced Settings section
  4. Check the Enable Traces VMUI web interface option
  5. Save and redeploy the application

Once enabled, access the traces interface at:

https://<your-domain>/traces/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Only enable this option if your deployment is within a trusted network or you have implemented external authentication.

To enable the VictoriaTraces web interface in Helm Deployment, add the following to your values.yaml:

victoriatraces:
  server:
    ingress:
      enabled: true
      ingressClassName: "nginx"  # Use your ingress class
      hosts:
        - name: "your-domain.com"
          path:
            - /traces
          port: http

Then upgrade your deployment:

helm upgrade pixee-enterprise-server ./charts/pixee-enterprise-server \
  -f values.yaml \
  -n pixee-enterprise-server

Once enabled, access the traces interface at:

https://your-domain.com/traces/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Consider implementing external authentication or only enable this in trusted network environments.

Option 2: Port Forwarding

If you prefer not to expose the VMUI via ingress, you can use port forwarding for temporary access.

Step 1: Create SSH tunnel from your local machine

ssh -L 10428:localhost:10428 pixee@<your-hostname>

Step 2: Set up port forwarding

In the SSH session, run:

sudo ./pixee shell
kubectl -n kotsadm port-forward pixee-enterprise-server-traces-server-0 10428:10428

Step 3: Access the traces interface

Open your browser and navigate to:

http://localhost:10428/traces/select/vmui/

Step 1: Port forward to VictoriaTraces

kubectl port-forward svc/pixee-enterprise-server-traces-server 10428:10428 -n pixee-enterprise-server

Step 2: Access the traces interface

Open your browser and navigate to:

http://localhost:10428/traces/select/vmui/

Querying Traces

VictoriaTraces provides a Jaeger-compatible query API for searching traces. In the VMUI, you can:

  • Search by service name — Filter traces from specific services (e.g., pixee-analysis-service)
  • Search by trace ID — Look up a specific trace using its trace ID
  • Filter by duration — Find slow requests by setting minimum/maximum duration
  • Filter by tags — Search for traces with specific span attributes

Querying Traces Programmatically

Beyond the VMUI, you can query traces over the Jaeger-compatible HTTP API with the Pixee CLI, which authenticates you through your identity provider and supplies a per-user bearer token:

TOKEN=$(pixee auth token --server https://pixee.example.com)
# List services, then search recent traces for one
curl -s -H "Authorization: Bearer $TOKEN" \
  https://pixee.example.com/o11y/traces/select/jaeger/api/services
curl -sG -H "Authorization: Bearer $TOKEN" \
  https://pixee.example.com/o11y/traces/select/jaeger/api/traces \
  --data-urlencode 'service=pixee-analysis-service' --data-urlencode 'limit=20'

See Programmatic Access with the Pixee CLI for the one-time pixee auth login step.

VictoriaTraces Resources