[Monitoring Operations]
Health probes, queue backlogs, live edge logs, and emergency kill-switch controls via the CLI.
Algorithmic trading demands high-integrity, real-time observability. Because Hoox microservices are distributed across Cloudflare's global edge network, tracking health, logs, and message queues requires a consolidated management plane.
This guide outlines how to use the Hoox CLI to audit system status, monitor queue backlogs, stream live edge logs, and manage the emergency kill switch.
1. Probing Worker Health Status
Every Hoox worker is configured with a secure /health endpoint that validates local CPU capacity, binding access, and D1 database connection integrity.
To probe and audit all active endpoints simultaneously:
hoox check health
hoox monitor statuswas removed in v0.8.0. The single source of truth for health checks is nowhoox check health.
Expected Output
The CLI runs asynchronous concurrent health probes, displaying green indicators and public route locations:
┌─────────────────────────────────────────────────────────────┐
│ Hoox Microservice Health Probes │
├─────────────────────────────────────────────────────────────┤
│ hoox (Gateway) ... OK https://hoox.alpha.workers.dev│
│ trade-worker ... OK (Internal Binding Verified) │
│ d1-worker ... OK (Internal Binding Verified) │
│ telegram-worker ... OK (Internal Binding Verified) │
│ agent-worker ... OK (Internal Binding Verified) │
│ web3-wallet-worker ... OK (Internal Binding Verified) │
│ email-worker ... OK (Internal Binding Verified) │
│ report-worker ... OK (Internal Binding Verified) │
│ analytics-worker ... OK (Internal Binding Verified) │
│ │
│ System Status: HEALTHY (0 warnings, 0 errors) │
└─────────────────────────────────────────────────────────────┘
Interpreting Health States
| Indicator | Meaning | Action Required |
|---|---|---|
| OK | All bindings active, D1 responsive | None — operating normally |
| DEGRADED | Some bindings unavailable | Check hoox logs tail <worker> for errors |
| FAILED | Worker unreachable or D1 disconnected | Run hoox repair check immediately |
| STARTING | Worker is still booting after deploy | Wait 30–60 seconds and retry |
2. Governing the Global Kill Switch
The Global Kill Switch is your emergency brake. Stored inside the sub-millisecond CONFIG_KV namespace, flipping this parameter instantly blocks all incoming trade signals globally in under 10 seconds, without requiring code updates or worker redeployments.
hoox monitor kill-switch show
# B. Emergency HALT - disable all edge trade execution immediately
hoox monitor kill-switch on
# C. Resume normal operations
hoox monitor kill-switch off
Kill Switch Behavior
When activated, the following sequence occurs automatically:
hooxgateway rejects all incoming/webhookPOST requests with503 Service Unavailable.A
KILL_SWITCH_ACTIVEevent is logged to D1 for audit trail.agent-workerhalts its 5-minute cron execution loop.trade-workerqueues any in-flight payloads for retry (not abandoned).- Telegram notification is sent to alert you.
Warning
When the Kill Switch is turned ON, the hoox gateway router immediately
rejects all incoming TradingView alerts and API webhook requests with a 503 Service Unavailable error and logs a KILL_SWITCH_ACTIVE warning. Active
positions must be flattened manually or via hoox monitor trades close-all.
3. Auditing Recent Transactions
To review execution history directly from your terminal:
# Print the 10 most recent trades executed across all exchanges
hoox monitor trades
# Print the 50 most recent trades
hoox monitor trades 50
# Filter by specific exchange
hoox monitor trades --exchange bybit
# Filter by date range (ISO format)
hoox monitor trades --from 2026-05-15 --to 2026-05-20
This aggregates entries from your production D1 database and prints a formatted terminal table outlining: TIMESTAMP | REQUEST_ID | EXCHANGE | SYMBOL | ACTION | QUANTITY | PRICE | STATUS
Trade Status Codes
| Status | Meaning |
|---|---|
Filled | Order fully executed on the exchange |
Partial | Only portion filled; remainder queued |
Rejected | Exchange returned a business logic error |
Failed | Max queue retries exhausted; logged for review |
Enqueued | Submitted to Cloudflare Queues for async retry |
4. Inspecting Queue Depth (Guaranteed Delivery Backlog)
If network volatility causes exchange API dropouts, Hoox offloads trade execution payloads to Cloudflare Queues. To check if there is an active execution backlog:
hoox monitor queue-depth
This displays the number of pending messages in the queue:
0: System is running normally (all trades executing on the fast-path direct service binding).> 0: Exchange API is experiencing rate-limits or downtime. Trade-worker is retrying transactions asynchronously in the background.
Queue Health Indicators
| Queue Depth | Risk Level | Recommended Action |
|---|---|---|
| 0 | Normal | No action needed |
| 1–5 | Watchful | Monitor exchange health pages |
| 6–20 | Elevated | Check exchange maintenance schedule; consider hoox config kv set webhooks:queue_mode direct_only |
| 20+ | Critical | Verify exchange API keys are valid; check rate limit headers; scale manually if needed |
5. Real-Time Log Streaming & Telemetry
When debugging custom strategies, you can stream verbose console logs directly from Cloudflare's global edge nodes to your local terminal using Wrangler bindings:
# A. Stream live console logs for a specific worker
hoox logs tail trade-worker
# B. Stream gateway traffic in real-time
hoox logs tail hoox
# C. Stream all workers simultaneously (diagnostic mode)
hoox logs tail --all
# D. Filter logs by severity level
hoox logs tail trade-worker --level error
Downloading Logs for Off-Line Analysis
To download historical logs offloaded to your R2 storage bucket for compliance audits or tax reports:
hoox logs download hoox --output logs/gateway-backup.log
# Download trade-worker logs for the last 24 hours
hoox logs download trade-worker --output logs/trade-execution.log --since 24h
6. System Metrics Overview
Hoox tracks key performance indicators via Cloudflare Analytics Engine. Access summary metrics via:
# View current system metrics snapshot
hoox monitor metrics
# Or use the Dashboard Command Center for visual charts
hoox dashboard
Key Metrics Tracked
| Metric | Source | Description |
|---|---|---|
| Orders/Min | analytics-worker | Trade execution throughput |
| Avg Latency (ms) | analytics-worker | Signal-to-fill round-trip time |
| Error Rate (%) | analytics-worker | Failed orders / total attempts |
| Queue Depth | Cloudflare Queues | Pending execution backlog |
| Daily P&L | D1 Database | Realized profit/loss calculation |
| Drawdown % | agent-worker | Current daily asset drawdown |
Tip
If the system status check returns a red error indicator (e.g. d1-worker ... FAILED), immediately proceed to the Self-Healing & Repair
Guide to run automated diagnostics and restore service bindings!
Next Steps
- Self-Healing & Repair — Diagnose connection drops, recreate broken bindings, and rebuild environments.
- Terminal UI Operations — Monitor health status, tail logs, and edit configurations interactively in a full-screen GUI cockpit.
- Infrastructure Management — Manage KV config namespaces, Queue parameters, and R2 storage buckets.