Observability

Blacki always provides local structured logs and local OpenTelemetry span files. It can additionally export spans to a remote collector through gRPC or HTTP/protobuf OTLP.

Outputs

Inside the container:

File	Contents
/app/logs/blacki-telemetry.log	JSON application log records
/app/logs/blacki-traces.log	JSON Lines OpenTelemetry spans

Compose maps /app/logs to ./logs on the host. Human-readable application logs also go to stdout and are available through:

docker compose -f compose.yaml -f compose.prod.yaml logs --follow agent

Instrumentation

At startup Blacki:

1. sets OTEL_RESOURCE_ATTRIBUTES;
2. instruments Google ADK with GoogleADKInstrumentor unless a private-tool mode is enabled;
3. configures stdout and JSON file logging; and
4. registers a TracerProvider with JSONFileSpanExporter; and
5. adds the selected OTLP exporter when a validated trace endpoint is configured.

The resource contains:

Attribute	Source
service.name	AGENT_NAME
service.namespace	TELEMETRY_NAMESPACE, default local
service.version	K_REVISION, default local
service.instance.id	Process ID plus generated UUID

Message content

Both OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false and ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false are required to disable current and legacy ADK content capture. Capturing prompts and responses can expose personal data, credentials, and tool results.

When ZEPTO_MCP_ENABLED=true or KOKORO_TTS_BASE_URL is configured, Blacki forces both values to false and disables the OpenInference Google ADK instrumentor. Zepto additionally prevents MCP body-debug logging. The ADK session database still retains tool calls and results so interrupted turns and confirmations can resume correctly.

Remote OTLP export

Local-only mode is the default and never creates a network exporter. To add remote gRPC span export, set:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example.com:4317
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=replace-me

For HTTP/protobuf, set:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example.com:4318/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer%20replace-me

You can instead use the global OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL, and OTEL_EXPORTER_OTLP_HEADERS fallbacks. Trace-specific values take precedence. The protocol defaults to grpc when omitted.

The endpoint must use http or https, identify a host, and contain no embedded username, password, query, or fragment. gRPC endpoints cannot contain a path. A trace-specific HTTP/protobuf endpoint is used exactly as configured; a global HTTP/protobuf endpoint receives /v1/traces after any existing base path. Headers use comma-separated, URL-encoded name=value pairs. Use https whenever headers contain credentials. Plain http sends headers without transport encryption and is appropriate only for a trusted local collector.

Protocol or header settings without an endpoint fail startup with a secret-free configuration error. The application logs only local, local+otlp-grpc, or local+otlp-http-protobuf once; it never logs the endpoint or headers.

Retention

The application files append and have no built-in size or time rotation. Docker's stdout log-driver rotation does not rotate ./logs/*.log.

Monitor them:

du -sh logs

Configure host-side retention before long-running use. Archive or clear files only during a maintenance window after preserving anything needed for an incident.

Failure behavior

If the application cannot create its log directory or file handler, it reports the error and continues with stdout logging. Treat that as degraded observability, not a successful persistence setup.