Logs & Debugging¶

Access pod status, view application logs, and query centralized logs for your Pixee Enterprise Server deployment.

VictoriaLogs (Centralized Logging)¶

When local metrics is enabled, Pixee Enterprise Server also deploys VictoriaLogs for centralized log collection. VictoriaLogs automatically collects and indexes logs from all Pixee components, making it easy to search and analyze logs across your entire deployment.

Enabling VictoriaLogs¶

VictoriaLogs is automatically enabled when you enable local metrics. See Enabling Local Metrics for instructions.

Accessing VictoriaLogs VMUI¶

You can access the VictoriaLogs web interface to query and explore logs. You can either expose it through the gateway or use port forwarding.

Option 1: Enable VMUI Web Interface (Gateway)¶

Enable the VMUI web interface to access the logs query interface directly through your browser.

Embedded ClusterHelm Deployment

To enable the VictoriaLogs web interface in Embedded Cluster deployments:

Navigate to the admin console
Select the Config tab
Go to the Advanced Settings section
Check the Enable VMUI web interface option (this enables both metrics and logs interfaces)
Save and redeploy the application

Once enabled, access the logs interface at:

https://<your-domain>/o11y/logs/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Only enable this option if your deployment is within a trusted network or you have implemented external authentication.

To enable the VictoriaLogs web interface in Helm Deployment, add the following to your values.yaml:

eg:
  enabled: true
victorialogs:
  gateway:
    enabled: true

The route attaches to the Gateway the main chart renders, so eg.enabled and global.pixee.domain must match what you set there.

Then upgrade your deployment:

helm upgrade pixee-enterprise-server ./charts/pixee-enterprise-server \
  -f values.yaml \
  -n pixee-enterprise-server

Once enabled, access the logs interface at:

https://your-domain.com/o11y/logs/select/vmui/

Unauthenticated Access

The VMUI web interface endpoints are not authenticated. Consider implementing external authentication or only enable this in trusted network environments.

Option 2: Port Forwarding¶

If you prefer not to expose the VMUI through the gateway, you can use port forwarding for temporary access.

Embedded ClusterHelm Deployment

Step 1: Create SSH tunnel from your local machine

ssh -L 9428:localhost:9428 pixee@<your-hostname>

Step 2: Set up port forwarding

In the SSH session, run:

sudo ./pixee shell
kubectl -n kotsadm port-forward pixee-enterprise-server-logs-server-0 9428:9428

Step 3: Access the logs interface

Open your browser and navigate to:

http://localhost:9428/o11y/logs/select/vmui/

Step 1: Port forward to VictoriaLogs

kubectl port-forward svc/pixee-enterprise-server-victoria-logs-single-server 9428:9428 -n pixee-enterprise-server

Step 2: Access the logs interface

Open your browser and navigate to:

http://localhost:9428/o11y/logs/select/vmui/

LogsQL Query Examples¶

VictoriaLogs uses LogsQL, a powerful query language for searching and filtering logs. Here are some useful queries:

# Search for errors across all logs
error OR ERROR

# Filter logs by pod name (pod names carry a generated suffix, so match a prefix)
_stream:{kubernetes.pod_name=~"pixee-enterprise-server-platform.*"}

# Search for specific text in platform logs
_stream:{kubernetes.pod_name=~"pixee-enterprise-server-platform.*"} AND "webhook"

# Find logs with a specific log level
_stream:{kubernetes.pod_name=~"pixee-enterprise-server-.*"} AND level:"ERROR"

# Search within a time range (use the time picker in VMUI)
_stream:{kubernetes.pod_namespace="kotsadm"} AND "analysis"

Log Retention and Disk Usage¶

Two limits bound the log store, and whichever is reached first applies: the retention period deletes logs older than it, and a disk ceiling drops the oldest logs once the store reaches it. The ceiling is what protects the node. On an embedded cluster the log volume is a directory on the node's root disk with no quota, so the volume request reserves nothing and stops nothing — without a ceiling, a burst of debug logging fills the node itself.

Embedded ClusterHelm Deployment

Logs maximum disk usage in the admin console is the ceiling. It defaults to 80% of the volume request, which is 8Gi for the default 10Gi volume. Change it under Config → the observability section and redeploy; it takes effect on the next rollout. Lower it if logs are consuming the node's disk.

For scale: a busy internal environment holds a week of logs in under 1 GiB, while a deployment left on LOGGING_LEVEL_ROOT=DEBUG has been measured at 12.6 GiB per day. The ceiling is what keeps the second case from reaching the node's disk.

The volume request behind the store is fixed at install time and is not shown in the console. Kubernetes does not allow a StatefulSet's volume claim template to be changed after it is created, and on openebs-hostpath the size is not enforced in the first place, so the ceiling is the only lever — and the one that matters.

Both values are yours to set, and on a real storage class the volume genuinely is a bound: the filesystem is the size you requested and the store hits ENOSPC when it fills. Keep the ceiling below it — about 80% leaves room for merges — and the chart's retention preflight fails when it is not smaller:

victoria-logs-single:
  server:
    retentionPeriod: "7d"
    retentionDiskSpaceUsage: "8GiB"   # ~80% of the volume below
    persistentVolume:
      size: "10Gi"

Raising persistentVolume.size on an existing release fails: a StatefulSet's volume claim template is immutable. To grow the volume, expand the PVC in place (kubectl patch pvc <name> -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}', which requires allowVolumeExpansion: true on the storage class), then kubectl delete statefulset <name> --cascade=orphan and upgrade so the template matches. The ceiling itself is a program flag and changes with an ordinary upgrade.

Write the ceiling in GiB (8GiB) or as a bare number of GiB (8). 8Gi is a Kubernetes unit that VictoriaLogs does not accept, and the preflight rejects it.

If your cluster's default storage class is a local-path provisioner, its size is not enforced either — set global.pixee.storageSizeEnforced: false and the ceiling-against-volume preflight is skipped.

VictoriaLogs Resources¶

VictoriaLogs Documentation
LogsQL Query Language - Complete LogsQL reference
VictoriaLogs VMUI - Web UI guide for logs

Viewing Pods¶

To view the pods in your deployment, use the namespace you installed Pixee into:

Embedded Cluster: kotsadm
Helm Deployment: pixee-enterprise-server, default, or your selected namespace

List Running Pods¶

kubectl get pods -n <namespace> --field-selector status.phase=Running

View All Pods with Status¶

kubectl get pods -n <namespace>

Viewing Logs¶

Basic Log Viewing¶

To view logs for a specific service, use the kubectl logs command. For example, to view logs for the platform service:

Embedded ClusterHelm Deployment

kubectl logs deployment.apps/pixee-enterprise-server-platform -n kotsadm

kubectl logs deployment.apps/pixee-enterprise-server-platform -n pixee-enterprise-server

Viewing Recent Logs¶

To narrow down to the last 25 lines with timestamps:

Embedded ClusterHelm Deployment

kubectl logs deployment.apps/pixee-enterprise-server-platform -n kotsadm --tail 25 --timestamps

kubectl logs deployment.apps/pixee-enterprise-server-platform -n pixee-enterprise-server --tail 25 --timestamps

Following Logs in Real-Time¶

To continuously stream new logs as they are generated, use the --follow flag:

Embedded ClusterHelm Deployment

kubectl logs deployment.apps/pixee-enterprise-server-platform -n kotsadm --follow --tail 25 --timestamps

kubectl logs deployment.apps/pixee-enterprise-server-platform -n pixee-enterprise-server --follow --tail 25 --timestamps

Common Deployments to Monitor¶

Here are the main Pixee Enterprise Server deployments you may need to view logs for:

Deployment	Purpose
`pixee-enterprise-server-platform`	Main platform service
`pixee-enterprise-server-analysis`	Analysis service

Example: Viewing Analysis Service Logs¶

Embedded ClusterHelm Deployment

kubectl logs deployment.apps/pixee-enterprise-server-analysis -n kotsadm --tail 50 --timestamps

kubectl logs deployment.apps/pixee-enterprise-server-analysis -n pixee-enterprise-server --tail 50 --timestamps

Viewing Logs for Specific Pods¶

If you need to view logs for a specific pod (rather than a service), first get the pod name:

kubectl get pods -n <namespace>

Then view the logs:

kubectl logs <pod-name> -n <namespace>

Multiple Containers in a Pod¶

If a pod has multiple containers, specify the container name:

kubectl logs <pod-name> -c <container-name> -n <namespace>

Troubleshooting Common Issues¶

No Logs Appearing¶

If no logs are appearing:

Verify the pod is running:
```
kubectl get pods -n <namespace>
```

Check pod status and events:

kubectl describe pod <pod-name> -n <namespace>

Check if the service has active endpoints:
```
kubectl get endpoints -n <namespace>
```

Pod Keeps Restarting¶

If a pod is repeatedly restarting, check the previous container's logs:

kubectl logs <pod-name> -n <namespace> --previous

Debugging an Outbound Integration¶

When the platform cannot reach an integration — Azure DevOps, Bitbucket, GitLab, Sonar, Checkmarx, Veracode, AppScan and the rest — and the ordinary logs say only that the call failed, the admin console's Include HTTP wire logs option (Support Bundle settings) raises the platform's Apache HttpClient header and wire loggers to DEBUG, so the request and response detail is recorded.

These loggers apply no redaction

The header logger prints every header verbatim, so an authenticated request records its Authorization: Bearer … header in plain text; the wire logger additionally records full request and response bodies, which is where credentials submitted in a payload — an AppScan API-key login, for example — end up. Credentials captured this way have previously reached a support bundle that was then shared onward.

Enable the option only on a non-production install, turn it off as soon as you have reproduced the problem, and rotate anything it logged before sharing a bundle.

Include noisy logs is a separate option and deliberately does not raise these loggers, so turning it on never exposes credentials.

Querying Logs Programmatically¶

Beyond the VictoriaLogs VMUI, you can query logs over HTTP with the Pixee CLI, which authenticates you through your identity provider and supplies a per-user bearer token:

TOKEN=$(pixee auth token --server https://pixee.example.com)
curl -sG -H "Authorization: Bearer $TOKEN" \
  https://pixee.example.com/o11y/logs/select/logsql/query \
  --data-urlencode 'query=_time:1h kubernetes.pod_namespace="pixee-enterprise-server" | limit 100'

The query parameter uses LogsQL, VictoriaLogs' query language — see the LogsQL reference for the full syntax (time filters, stream selectors, pipes, and stats).

See Programmatic Access with the Pixee CLI for the one-time pixee auth login step.

Additional Resources¶

For more information on the kubectl logs command, see the Kubernetes documentation.