Observability

Abstract

PYNE’s production surface today optimizes for simple, hostable signals rather than a full OpenTelemetry mesh: HTTP health checks, development debug logging (PYNE_DEBUG), process-wide runtime counters (GET /stats), process managers (gunicorn), cloud platform logs, CI artifacts (coverage, Playwright reports), and application-level API key usage counters. Treat this page as the map of what exists in-repo, not a claim of enterprise APM parity.

Conceptual model

Interface surface

Health

backend/app.py exposes:

GET /
→ { "status": "healthy", "service": "pynescript-pro-api", … }

Used by:

• Dockerfile target api HEALTHCHECK (curl -fsS http://127.0.0.1:8080/)
• docker-compose.yml api healthcheck
• Load balancers / Cloud Run startup-ish probes (platform-dependent)

Runtime statistics

GET /stats        # public, no API key
→ { counters, latencies, gauges, compile_cache, sessions, debug }

Counters, latency percentiles (avg/min/max/p50/p95), and gauges for requests, runs, interpreter, compile, prewarm, live-session deltas, sessions, and datafeed. Process-local — resets on worker restart. Full reference: GET /stats.

curl -s http://127.0.0.1:5002/stats | python -m json.tool

Debug logging (PYNE_DEBUG)

A single development-only flag turns on detailed logging across the execution engine and the datafeed:

PYNE_DEBUG=1 python -m backend.app
PYNE_DEBUG=1 PYNE_DEBUG_SCOPE=pynescript.runtime,pynescript.compiler python -m backend.app

What it traces:

PYNE_DEBUG=1 POST /run on a fresh process, for example:

DEBUG pynescript.runtime.host:  auto: eligibility src=2de7f37e3736 eligible=True reason= (cached_failure=False)
DEBUG pynescript.compiler.engine: prewarm: warming shared Numba builtins (first touch, JIT)
DEBUG pynescript.compiler.engine: prewarm: builtins warm in 7497.6ms
DEBUG pynescript.runtime.host:  compile: MISS src=2de7f37e3736 … in 8973.67ms host_cache_entries=1
DEBUG pynescript.runtime.host:  run finished mode=compile bars=40 total=8974.63ms parse=8973.67ms eval=0.06ms

A warm second run shows the cache doing its job:

DEBUG pynescript.runtime.host:  compile: HIT src=2de7f37e3736 … in 0.00ms host_cache_entries=1
DEBUG pynescript.runtime.host:  run finished mode=compile bars=40 total=0.36ms parse=0.00ms eval=0.04ms

And a live session shows the O(delta) property directly:

DEBUG pynescript.runtime.host:  append_bars run_id=250985e340… delta=2 start=20 new_total=22
DEBUG pynescript.runtime.host:  bar-loop start run_id=250985e340… start=20 end=22 bars=2 total_bars=22
DEBUG pynescript.runtime.host:  update_last_bar run_id=250985e340… idx=21 bar_time=… n_bars=22

Implementation is three small pieces:

The toggle is read at run / batch / request boundaries, never inside a per-bar loop, and log payloads are only built when the flag is on.

Process model (production)

gunicorn --bind :8080 --workers 2 --threads 4 --timeout 120 backend.app:app
# docker/entrypoint-api.sh: GUNICORN_TIMEOUT default 120
# docker-compose.prod.yml: GUNICORN_TIMEOUT default 60

Logs

Application modules use standard library / Flask logging patterns; there is no mandatory structured-log schema enforced repo-wide. �SLOT0� if a future branch introduces OpenTelemetry exporters.

CI / quality signals

Usage metering (app-layer)

backend/middleware/auth.py tracks per-key:

• calls_used / calls_limit / tier
• last_used timestamps

These are business metrics for rate limits, not Prometheus time series — export them if you need dashboards.

Internals

Invariants & edge cases

1. Health ≠ readiness of evaluator warm caches. A 200 on / does not mean the first /run will be fast (cold import / JIT / numba). Check compile.host_prewarmed / call POST /compile/prewarm.
2. /stats is per-process, not fleet-wide. Behind a load balancer each worker counts only its own slice. Aggregate per worker, or run single-worker (which live sessions require anyway).
3. Timeouts double-bind. A platform request limit shorter than gunicorn's default 120s wins unless GUNICORN_TIMEOUT is set to that limit.
4. Scale-to-zero hides idle metrics. No traffic ⇒ no samples; use synthetic uptime checks if SLOs matter.
5. Coverage artifacts expire (7–14 days retention on Actions uploads).
6. Do not scrape secrets from logs. API keys and admin tokens must never be logged at info level. PYNE_DEBUG logs whether credentials were supplied, never their values.
7. Latency percentiles are recent-window. p50_ms / p95_ms come from the last 512 samples; max_ms is all-time.

Worked examples

Local health + stats loop

make run
curl -s http://127.0.0.1:5002/    | python -m json.tool
curl -s http://127.0.0.1:5002/stats | python -m json.tool

Explain a slow run

PYNE_DEBUG=1 PYNE_DEBUG_SCOPE=pynescript.runtime,pynescript.compiler make run
# in another shell:
curl -s -X POST http://127.0.0.1:5002/run -H 'Content-Type: application/json' \
  -d '{"script":"//@version=6\nindicator(\"x\")\nplot(ta.sma(close,20))","data":[...]}'

Read the trace bottom-up: eligibility → cache HIT/MISS → JIT warm duration → run finished total=…. A first run paying ~9 s in prewarm: builtins warm and compile: MISS is expected; the second identical request should be a compile: HIT in ~0 ms.

Inspect live-session delta stats

curl -s http://127.0.0.1:5002/stats \
  | jq '{delta: .counters["delta.bars_appended"], appends: .counters["delta.append_calls"],
         max: .gauges["delta.bars_per_append_max"], loop: .latencies["interpret.bar_loop"]}'

A large delta.bars_per_append_max with a small interpret.bar_loop.p95_ms means the delta path is behaving as intended.

Docker health

docker inspect --format='{{json .State.Health}}' CONTAINER

Failure modes

See also

• Docker
• Security
• API lifecycle