Deployment & Operations

Octopool runs as a Cloudflare Worker (octopool) plus Durable Object classes (PoolCoordinator, PolicyCoordinator, and BackendAdmission) and a D1 database. The OpenClaw deployment serves the authoritative data plane at octopool.openclaw.ai; a thin octopool.dev Worker in the domain's Cloudflare account forwards into it so both hosts share one cache. Self-hosters can ignore that proxy and deploy only the main Worker.

The Go CLI is a separate, statically-linked binary released via GoReleaser.

Source: wrangler.jsonc, wrangler.public-proxy.jsonc, migrations/, package.json, test/e2e.sh.

#Self-host on Cloudflare

Prerequisites:

• Cloudflare account on the Workers Paid plan (Durable Objects + D1 require it).
• A domain you can route at a Worker — managed by Cloudflare, or with a CNAME you can point at the Worker.
• A GitHub org you can verify membership against (ALLOWED_GITHUB_ORG).
• At least one GitHub identity to pool — a PAT, or a GitHub App installation on the repos you want to serve.

#1. Clone and configure

git clone https://github.com/openclaw/octopool.git
cd octopool
pnpm install

Edit wrangler.jsonc:

• account_id — your Cloudflare account id.
• vars.ALLOWED_GITHUB_ORG — the GitHub org whose members may mint caller tokens.
• vars.DEFAULT_ALLOWED_OWNERS — comma-separated GitHub owners (orgs/users) with scoped identity routing. Other public repositories are allowed by the public-repo guard.
• vars.GITHUB_OAUTH_CLIENT_ID — OAuth client id of your GitHub App, used for browser sign-in. Pair with the GITHUB_OAUTH_CLIENT_SECRET secret below.
• vars.GITHUB_OAUTH_CALLBACK_ORIGIN — optional HTTPS origin registered as the GitHub OAuth callback when browser sign-in starts on a different host.
• routes[] — the custom domain you want octopool served on.

If you only need one host, ignore wrangler.public-proxy.jsonc. OpenClaw uses it to forward octopool.dev to the authoritative octopool.openclaw.ai data plane. Both Workers run in the OpenClaw Services Cloudflare account.

#2. Create the data plane

# pick the location hint nearest your callers (e.g. wnam, enam, weur);
# D1 location is fixed at creation and cannot be changed later
wrangler d1 create octopool --location wnam
# copy the printed database_id into wrangler.jsonc d1_databases[].database_id

wrangler d1 migrations apply DB --remote

The pool coordinator's location hint in src/pool-coordinator.ts (poolCoordinatorStub) should match the D1 location so per-request coordinator calls do not cross regions.

The PoolCoordinator, PolicyCoordinator, and BackendAdmission Durable Object classes are provisioned by migration tags v1, v2, and v3, respectively, in wrangler.jsonc on wrangler deploy. See the policy coordinator upgrade for freshness and rollback behavior. The policy object's initial location hint also targets the D1 primary; keep its fixed string-rewrite-policy name unchanged across deployments.

#3. Add secrets

Every credential lives in Cloudflare's secret store. Nothing here ever goes into wrangler.jsonc, D1, or logs.

wrangler secret put OCTOPOOL_ADMIN_TOKEN              # bearer for /v1/admin/* and octopool admin ...
wrangler secret put GITHUB_OAUTH_CLIENT_SECRET        # browser GitHub login (with GITHUB_OAUTH_CLIENT_ID)
wrangler secret put OCTOPOOL_PROXY_SECRET             # only if you run a second proxy Worker
wrangler secret put OCTOPOOL_GITHUB_ORG_TOKEN         # background org-membership + public-repo proof
wrangler secret put OCTOPOOL_GITHUB_APP_ID            # only if you use GitHub App identities
# one secret per pooled identity, referenced by name from D1:
wrangler secret put OCTOPOOL_PAT_ALICE                # raw PAT value
wrangler secret put OCTOPOOL_GITHUB_APP_PRIVATE_KEY   # PKCS#8 (BEGIN PRIVATE KEY) PEM

Keep copies in a real secret manager (1Password, etc.) — Cloudflare's UI does not show the values back after you set them.

#4. Deploy the Worker

wrangler deploy

For Cloudflare-managed domains, the routes[] entry registers the custom domain automatically. For external DNS, CNAME the host at the Worker once.

#5. Register at least one identity

export OCTOPOOL_ADMIN_TOKEN=...                              # the value you set above
export OCTOPOOL_URL=https://octopool.your-org.dev

octopool admin identity \
  --pool maintainers \
  --id pat_alice --login alice \
  --secret-ref OCTOPOOL_PAT_ALICE \
  --scope your-org

# Optional broad public-repo PAT identity:
octopool admin identity \
  --pool maintainers \
  --id pat_public --login alice \
  --secret-ref OCTOPOOL_PAT_ALICE \
  --scope '*'

The first reference to a pool by name (here, maintainers) creates it with the default policy. Verified org members can now octopool login https://octopool.your-org.dev and use the relay; the registered identities serve misses that cache and token-free transports cannot satisfy. See Admin & provisioning for --scope, --kind github_app, and --installation-id shapes.

#6. Verify

curl https://octopool.your-org.dev/.well-known/octopool
# {"service":"octopool","version":1,"api_base":"...","default_pool":"maintainers", ...}

OCTOPOOL_E2E_HOST=octopool.your-org.dev pnpm e2e
octopool stats

#Cloudflare resources

The hosted OpenClaw deployment keeps both Workers in the OpenClaw Services account (91b59577e757131d68d55a471fe32aca):

• Authoritative Worker octopool — config wrangler.jsonc, entry src/index.ts, nodejs_compat, observability on, custom domain octopool.openclaw.ai.
• Public-host Worker octopool-public-proxy — config wrangler.public-proxy.jsonc, entry src/openclaw-proxy.ts, custom-domain proxy for octopool.dev.
• Durable Object PoolCoordinator (binding POOL_COORDINATOR, SQLite-backed, migration tag v1).
• Durable Object PolicyCoordinator (binding POLICY_COORDINATOR, SQLite-backed class, migration tag v2; D1 remains the durable policy store).
• Durable Object BackendAdmission (binding BACKEND_ADMISSION, SQLite-backed, migration tag v3), with one independent admission object per pool.
• D1 database octopool-wnam (binding DB).

Both hosted Worker deploys use the Molty item OpenClaw Services Cloudflare API Token; the proxy does not use a personal-account Cloudflare credential. The shared OCTOPOOL_PROXY_SECRET still has to exist on both Workers. Self-hosted deployments use whatever account and routes they configure.

#Configuration

Plain vars (in wrangler.jsonc):

• ALLOWED_GITHUB_ORG — the only GitHub org that can mint caller tokens.
• DEFAULT_ALLOWED_OWNERS — comma-separated owners served by scoped identity routing.
• MAX_RESPONSE_BYTES — single response-body cap for every route (2 MiB default; the hosted deployment sets 4 MiB).
• REQUEST_TIMEOUT_MS — 15s default.
• CLIENT_BACKEND_CONCURRENCY — 16 default; concurrent backend-work requests per authenticated caller/client in each pool (positive integer).
• ORG_VERIFY_TTL_SECONDS — 24h default; how long an org-membership verification stays fresh before octopool re-checks at request time.
• GITHUB_OAUTH_CLIENT_ID — GitHub App OAuth client id used for browser sign-in.

Optional vars (set as needed): PUBLIC_REPO_TTL_SECONDS (default 30; the hosted deployment sets 900 — see the public-repo guard notes in cache.md for the trade), PUBLIC_REPO_NEGATIVE_TTL_SECONDS (default 3600), DEFAULT_LOGIN_POOL (default maintainers).

Secrets (via wrangler secret put, never in D1/KV/logs):

• OCTOPOOL_ADMIN_TOKEN — admin API auth.
• GITHUB_OAUTH_CLIENT_SECRET — website GitHub login.
• OCTOPOOL_PROXY_SECRET — shared secret on both Workers so only the public-host proxy can assert the original octopool.dev host.
• OCTOPOOL_GITHUB_ORG_TOKEN — background org-membership verifier and public-repo proof fetcher.
• OCTOPOOL_GITHUB_APP_ID — GitHub App id (for App identities).
• One secret per identity secret_ref — PAT value, or the App private key as PKCS#8 (BEGIN PRIVATE KEY) PEM. Keep a copy in 1Password.

#Migrations

#Relay phase timing

Apply 0025_audit_timings.sql before running the timing-enabled Worker. It adds nullable audit_events.timings_json TEXT; existing rows remain NULL, older Workers can still insert rows, and no backfill is needed.

wrangler.jsonc enables Worker traces at a head sampling rate of 0.2, leaving log settings unchanged. The authoritative Worker runs relay authentication, cache, admission, and publication; the public proxy forwards requests, responses, and Server-Timing without config changes. Timings exclude the proxy hop and caller/network latency. Use native traces for D1 serving regions, Worker-to-DO transport, and DO-internal D1/SQLite.

Every relay HTTP response has an outer Server-Timing header such as response_ready;dur=42, auth;dur=5, rewrite_policy;dur=2, ..., separate from forwarded GitHub headers and the JSON envelope. Audited paths persist the same snapshot as version 1 JSON. Requests rejected before an audit context exists get the header without an audit row.

response_ready_ms measures elapsed time from authoritative Worker relay entry through response encoding, security headers, and foreground cleanup, not the sum of phases. duration_ms retains its earlier capture point. The audit INSERT runs in waitUntil after the snapshot freezes. Timings exclude audit completion, successful quota feedback, detached admission release, deferred log proofs, and stale-while-revalidate refreshes. Normal permit release is awaited for up to one second and included in admission. Deferred log audits retain the original response snapshot when their proof completes later.

phases_ms contains every key below (zero for skipped or sub-clock phases). Durations are inclusive: auth includes rewrite-policy and membership work; public_guard includes probes and proof publication; acquisition can include admission and cache rechecks; publication includes renewal, storage, edge warming, and completion. On pooled identity misses, the public-proof owner can overlap public_guard with identity selection, token acquisition, and pooled_fetch (including github_headers/github_body); proof followers wait for its verdict. Both must complete before the response can be used; their durations in timings_json and Server-Timing must not be added to estimate response latency. Different phases can overlap. Repeated calls accumulate elapsed time; overlapping or nested calls to the same phase count once. phase_calls separately counts entries, including unsuccessful attempts and early returns.

The guard checks fresh negative proofs before allowing its owner to start the pooled read. This check remains inside public_guard; no separate preflight repeats the edge or D1 lookup.

d1_ops counts Worker-issued D1 executions (first, all, run, raw, or batch), including failures and retries, but excludes preparing/binding SQL and DO-internal D1 operations. Use native traces for the latter. do_calls counts Worker-issued RPC/fetch calls, excluding stub acquisition. github_fetches counts all outbound GitHub attempts, including visibility probes, membership refreshes, redirects, and token exchanges. Counts stop at response readiness; detached background work has no request timer.

Identity-candidate D1 fallbacks and validator reads count one d1_ops per batch. Identity passes count one cache_lookup per edge probe and one per D1 batch; validators count one per batch. Later passes and post-acquisition rereads count separately.

config_cache_hits counts settled hits for caller, membership, policy, and identities; config_cache_pending_hits counts reuse of in-request pending loads. A positive count indicates a hit. cache_hits counts successful fresh edge or shared (D1) body lookups and r2 log lookups, including rechecks, even if the entry is not served. No token hashes, SQL, URLs, repository keys, or credential values are retained.

Instrumentation adds synchronous clock/counter work and promise completion observers, with no foreground awaits or I/O. Date.now() advances on I/O in Workers, so CPU-only phases can report zero. These measure latency, not CPU; sub-clock differences are meaningless.

Query the JSON with SQLite JSON functions, filtering by version and cache outcome:

SELECT request_id, cache_status, duration_ms,
       json_extract(timings_json, '$.response_ready_ms') AS ready_ms,
       json_extract(timings_json, '$.phases_ms.pooled_fetch') AS pooled_ms,
       json_extract(timings_json, '$.phases_ms.body_publish') AS publish_ms,
       json_extract(timings_json, '$.d1_ops') AS worker_d1_ops,
       json_extract(timings_json, '$.do_calls') AS do_calls,
       json_extract(timings_json, '$.github_fetches') AS github_attempts
FROM audit_events
WHERE json_extract(timings_json, '$.version') = 1
ORDER BY created_at DESC LIMIT 100;

SELECT cache_status, count(*) AS requests,
       avg(json_extract(timings_json, '$.response_ready_ms')) AS avg_ready_ms,
       avg(json_extract(timings_json, '$.phases_ms.public_guard')) AS avg_guard_ms,
       sum(json_extract(timings_json, '$.config_cache_hits.caller') > 0) AS warm_callers
FROM audit_events
WHERE created_at >= datetime('now', '-1 hour')
  AND json_extract(timings_json, '$.version') = 1
GROUP BY cache_status;

#Cache publication upgrade and restore

Apply additive 0020_cache_publication.sql after 0019 and before routing traffic to the publication-fenced Worker. Verify the owner table's AUTOINCREMENT primary key, safe-integer CHECK, unique epoch/resource index and expiry index, the body receipt columns, and the new github_public_repo_proofs table. The migration does not copy legacy positive proofs, purge bodies, alter identity feedback, or apply itself remotely.

The server-owned publication-v1 epoch composes with every existing representation generation. New readers refuse old bodies/proofs, including stale and 304 candidates. Warm-up therefore costs fresh upstream observations; no global purge is necessary. Old Workers still in flight can call the retained legacy cache-fill RPCs and publish only to their old key/proof namespaces. Each new acquisition also drains at most 16 expired cache_fills rows in its contacted DO using a local expiry index; live fills, identity leases, cooldowns and rate feedback remain intact. Dormant old DOs retain their rows until contacted. Remove the legacy RPC/table surface only after old Worker routing and in-flight requests are demonstrably drained. Rolling back to old readers restores the old publication defect, regardless of the new schema.

Never reset, drop/recreate, or restore cache_publication_owners.sqlite_sequence behind issued capabilities within the same epoch. Backup/export must retain its high-water mark. If restore/import/rebuild can lower it, deploy a new isolated server epoch before resuming new-reader publication, covering body keys, proof keys and edge namespaces. Do not reuse an epoch name or copy old positives into it. The final safe ID is 9007199254740991; exhaustion fails closed, rather than rounding, wrapping, or recycling. There is no dynamic epoch registry: retired code may still write its own old namespace. Drain its routing before claiming every request is repaired.

Owner completion deletes immediately. Each D1 acquisition is one SQL statement and one binding call; expired ownership for the same resource is replaced atomically without GC. Hourly maintenance deletes at most 10,000 expired owners, separately from proof/body/audit pruning. Inline GC was removed after 778,116 statements deleted only 4,813 rows in the three-day September 23, 2026 sample. The eight-second expiry, bursts, outages and cron health determine transient occupancy. Monitor expired count and oldest expiry rather than claiming a fixed cardinality or treating a bounded batch as a bounded backlog. DELETE makes pages reusable and does not guarantee that the allocated SQLite file shrinks.

The following operator diagnostics are aggregate-only; they do not expose capability tokens. Run them only through the operator's normal authorized database workflow:

SELECT count(*) AS owners,
       sum(lease_until_ms <= unixepoch('now') * 1000) AS expired_owners,
       min(CASE WHEN lease_until_ms <= unixepoch('now') * 1000
                THEN lease_until_ms END) AS oldest_expired_ms
FROM cache_publication_owners;
SELECT seq, 9007199254740991 - seq AS remaining_safe_ids
FROM sqlite_sequence WHERE name = 'cache_publication_owners';

These approximate monitoring timestamps have second resolution; publication and renewal use the millisecond D1 execution clock. Local native D1 tests establish generated-query behavior, retained sequence, commit receipts, expiry/GC, DO eviction, and Cache API lifecycle. They do not establish hosted retry timing, globally ordered edge puts, D1 billing, or an atomic D1/DO/Cache API commit. An accepted delayed edge put can remain visible until its original immutable expiry; hot hits add no new authority lookup.

#Migration inventory

D1 schema lives in migrations/:

• 0001_init.sql — pools, callers, callerpools, identities, identityscopes, audit_events.
• 0002_github_cache.sql — github_user_id column + production caller backfill, and github_cache_entries.
• 0003_github_app_public_cache.sql — installation_id column and github_public_repos.
• 0004_web_dashboard_sessions.sql — dashboard role, OAuth states, and hashed website sessions.
• 0005_audit_cache_metrics.sql — per-request cache status and cacheability columns, plus stats indexes for route and hit-rate aggregates.
• 0006_pr_state_proofs.sql — short-lived validated PR state discriminators for state-aware PR subresource cache keys.
• 0007_audit_cache_stale.sql — stale-cache audit status and cacheability metrics.
• 0008_public_api_rates.sql — anonymous GitHub API quota snapshots recorded from API responses.
• 0009_audit_outcomes.sql — local-fallback reasons and coalesced-fill telemetry.
• 0010_audit_retention.sql — audit timestamp index for bounded retention cleanup.
• 0011_cache_stale_retention.sql — per-entry stale deadlines and indexed cache cleanup.
• 0012_caller_clients.sql — concurrent per-client caller tokens and audit attribution.
• 0013_audit_backends.sql — bounded upstream-source attribution for route-level stats.
• 0014_public_repo_negative_proofs.sql — cached definitive private-repository proofs.
• 0015_drop_oauth_states.sql — remove obsolete database-backed OAuth state.
• 0016_string_rewrites.sql — authoritative deployment-wide singleton string policy; seeds explicit empty rules at revision 1.
• 0017_org_identity_verification.sql — add a separate nullable identity-bound proof timestamp without backfill or changes to old data; apply before the rolling Worker upgrade. Old timestamp writes cannot authorize new Workers.
• 0018_active_caller_enrollment.sql — unique active immutable GitHub ID plus case-insensitive org; refuses ambiguous duplicates without changing caller history. Apply before the updated enrollment Worker.
• 0019_client_name_guards.sql — schema guards for normalized client-name aliases.
• 0020_cache_publication.sql — committed publication ownership, body receipts, and isolated repository proofs.
• 0021_audit_requested_max_age.sql — nullable caller max-age in seconds on every audited relay request.
• 0022_terminal_job_proof.sql — index terminal-job completion proof lookups by pool and path.
• 0023_audit_cache_miss_reason.sql — nullable bounded cache-miss reason for cache-accepting relay reads.

Apply with wrangler d1 migrations apply DB (add --remote for production).

#Cache miss reason audit upgrade

Apply 0023_audit_cache_miss_reason.sql before or with the Worker deployment, completing it before the new Worker serves traffic: audit inserts require audit_events.cache_miss_reason. Existing rows remain NULL, and older Workers can continue inserting without the column. Keep this additive migration on rollback. No CLI upgrade, cache purge, or re-login is required. Settled CI collection TTLs change as entries are refilled. See cache miss reasons for the values and their precedence.

SELECT route_kind, cache_miss_reason, count(*) AS requests
FROM audit_events
WHERE created_at >= datetime('now', '-24 hours')
  AND (requested_max_age IS NULL OR requested_max_age > 0)
  AND cache_miss_reason IS NOT NULL
GROUP BY route_kind, cache_miss_reason
ORDER BY requests DESC;

#Requested max-age audit upgrade

Apply migration 0021_audit_requested_max_age.sql before or with the updated Worker deployment, completing it before the new Worker serves traffic. Audit inserts require the new column. Existing rows remain NULL; older Workers can keep inserting rows without the column, so retain the additive migration on rollback. No CLI upgrade, cache purge, or re-login is required. Age-scaled TTLs apply as entries are refilled.

#Atomic enrollment upgrade

Before applying 0018_active_caller_enrollment.sql, run this aggregate-only, read-only preflight on the intended database. Rerun it at deployment: an earlier clean observation is not a lock. Do not include caller names, IDs, hashes, or credentials in public proof.

WITH enrollment_groups AS (
  SELECT github_user_id, org_login COLLATE NOCASE AS org,
         COUNT(*) AS n, SUM(status = 'active') AS active_n
  FROM callers
  WHERE github_user_id IS NOT NULL
  GROUP BY github_user_id, org_login COLLATE NOCASE
)
SELECT
  (SELECT COUNT(*) FROM enrollment_groups WHERE active_n > 1)
    AS blocking_active_duplicate_groups,
  (SELECT COALESCE(SUM(active_n - 1), 0) FROM enrollment_groups WHERE active_n > 1)
    AS excess_active_rows,
  (SELECT COUNT(*) FROM enrollment_groups WHERE n > 1 AND active_n <= 1)
    AS nonblocking_historical_groups,
  (SELECT COUNT(*) FROM callers WHERE github_user_id IS NULL) AS null_id_rows,
  (SELECT COUNT(*) FROM callers WHERE github_user_id IS NULL AND status = 'active')
    AS active_null_id_rows,
  (SELECT COUNT(*) FROM callers WHERE github_user_id IS NOT NULL AND
    (typeof(github_user_id) <> 'integer' OR github_user_id <= 0
     OR github_user_id > 9007199254740991)) AS invalid_nonnull_id_rows;

A nonzero active-duplicate or malformed-ID count blocks deployment for operator review. Historical/NULL counts are inventory, not evidence of ownership or permission to merge. The index covers only active nonnull immutable IDs and uses COLLATE NOCASE for orgs; malformed-ID review is a preflight gate, not a new schema constraint.

Apply schema first, including 0017_org_identity_verification.sql, then verify the migration record and the actual idx_callers_active_github_org definition before deploying the Worker. Index creation is the authoritative uniqueness gate and fails unchanged if a new duplicate appeared after preflight. Never automatically choose an oldest/newest row, union grants, promote roles, or move tokens, sessions, or audit history. An operator must explicitly decide which ambiguous rows to retire in place before retrying. Retired rows must not be re-enabled as a recovery shortcut; that can revive attached credentials.

Existing singletons, disabled history, and NULL-ID records survive unchanged. Migration 0012's named legacy tokens are preserved without replaying that migration or adding a bootstrap-hash authentication fallback. Same-client rotation retains token row IDs and audit links. Ordinary cap pruning keeps the audit event, caller ID, and recorded client name while the deleted token's audit FK becomes NULL. Browser sessions remain attached to their original caller. Historical noncanonical client names (for example host.local versus host) are not changed by migration 0018; see the client-name upgrade below for verified singleton reconciliation and the remaining ambiguity decision.

During rollout, old code against the new index can reject a competing initial login; new code without the index fails enrollment closed. Coordinate an enrollment quiet window if transient errors matter. Keep the index on a Worker rollback. A failed migration leaves the old enrollment race unresolved and must block rollout, not trigger a fallback. The normal 30-second cross-isolate auth-cache window still applies after rotation or row revocation. The deployment operator owns production preflight and deployment.

#Client-name upgrade

Apply 0019_client_name_guards.sql before deploying the client-name Worker fix. Verify the migration record and both caller_tokens_canonical_insert and caller_tokens_canonical_update trigger definitions. Keep these guards on rollback. This schema-only migration never rewrites existing token names, hashes, audit labels or foreign keys, and deliberately permits existing ambiguous families to remain authenticated. It adds no per-relay write or history backfill.

The policy removes every trailing .local suffix, case-insensitively, preserving base case. It treats compound spellings as aliases, not evidence of their original intent or physical host identity. Verified login resolves the immutable caller inside the existing atomic batch, renames an unambiguous singleton in place, rotates its token, then applies the normal cap. The row ID, creation time and audit links survive; unrelated clients, roles, grants and browser sessions are preserved. A family with multiple stored rows refuses with 409 client_name_ambiguous; no new credential, profile refresh, grant or pruning commits. Existing credentials keep working.

An operator must review each refused family and explicitly decide which credentials, if any, to retire. Do not choose by age, recent use or display label, union privileges, reparent audit history, or globally backfill names. Deleting a token clears its audit token FK under the existing schema; retaining every retired token identity would require a separately approved retirement design. The deployment operator owns production inventory and that unresolved decision.

The triggers reject new noncanonical names and canonical inserts conflicting with a historical alias. Old Workers may therefore reject compound-name or historical-alias login during rollout; they cannot reintroduce a raw alias after corrected rotation. Old canonical writes still rotate an already reconciled singleton. Coordinate an enrollment quiet window if transient old-Worker errors matter. Deploy schema first and drain old Workers; new code without the guards is not a supported deployment. Upgrade CLIs and re-login to refresh saved labels. The 30-second cross-isolate auth-cache window and issuing-isolate invalidation remain unchanged.

Login adds one indexed, caller-scoped reconciliation statement and bounded trigger checks over that caller's tokens (normally at most 16). Stats filters compare canonical aliases within the existing pool/time window; client groups project distinct recorded labels for the caller/window before aggregating and limiting results. Audit writes and stored history remain unchanged. Stats grouping cannot resolve credential ambiguity or establish that two labels were one machine.

#Activating string protection

Apply migration 0016 before deploying the updated Worker. The seeded empty policy preserves existing behavior, but the new Worker requires the row to exist even when no rules are configured. Install an updated CLI for local publication protection, then import a private UTF-8 rules file with octopool admin string-rewrites set --file <path>. See Admin for the exact file/API contract, portable regex subset, limits, and revision conflict handling. This is a deployment-wide setting, not a per-pool override.

Each protected CLI operation downloads a fresh policy; the Worker separately obtains the authoritative snapshot from PolicyCoordinator for each relay request after authentication and normalization, before classification, repository probes, caches, upstream calls, or audit metadata. An admin API change through the coordinator affects the next request even in a warm isolate or on a cache hit. An in-flight operation uses the snapshot checked before its dispatch; changing a policy does not recall an already dispatched request. Older CLIs retain their old behavior; writes by older Workers follow the 60-second visibility bound below.

Each policy download and relay policy load costs one call to the dedicated global object. It holds a validated, compiled snapshot in memory, loads the D1 primary when cold or expired, and serializes D1 revision-CAS writes with snapshot installation before acknowledging success. Admin API writes through the coordinator are immediately visible to subsequent GETs and relay policy loads. Any other write, including older Workers, manual D1 edits, and restores, is visible within 60 seconds. The maximum snapshot age is measured from the start of its D1 load or write, so a slow response cannot extend that bound. The first read at expiry reloads from the primary under the existing gate; concurrent reads share that reload and never receive expired rules. Unexpired reads issue no policy SQL and share no pool/publication coordinator critical section. Relay requests still compile their request-scoped rules because compiled RE2 instances cannot cross the object boundary. No policy is cached outside the object. Even empty policies require an authoritative object read; there is no expired-policy fallback, replica shortcut, or offline allowance. Eviction reloads the primary. Failed writes invalidate the snapshot before any possibly committed D1 change; subsequent reads fail closed until primary reconciliation succeeds. Expired-snapshot reload failures likewise return 503 unavailable, never the old policy. Measure DO request/CPU/duration cost, response bytes, queue time, cold-load frequency and latency with representative rules before claiming cost or tail-latency savings.

For reads the Worker inspects the normalized path and its segments, query keys and all values, and allowed forwarded header names/values. The JSON envelope is decoded once with strict UTF-8 and duplicate-key validation. The guard checks literal and once percent-decoded text, including decoded segment boundaries and query + as space. Residual %HH after decoding (such as %2569), malformed percent/UTF-8 encoding, and embedded backslash escape layers are rejected while rules are active. This conservative boundary may refuse legitimate literal percent/backslash searches; it prevents common double encoding from revealing an unchecked name downstream. Read inspection is bounded in aggregate, and the raw relay envelope is capped at 1 MiB. Inputs are never silently rewritten into different repository or search semantics.

Matching reads return 403 string_rewrite_denied with no content in the error. Policy load/corruption/overload returns 503 string_rewrite_policy_unavailable, not a 424 fallback_local instruction. These failures stop before request audit/cache writes and do not log patterns, matched text, or backend exception messages. API responses, including errors, are no-store. Protect policy GET access and D1 backups: rules can contain private names even though error responses and normal import output do not.

CLI policy-load failures return only bounded metadata on the existing stderr error line; they do not change protection, retry policy, or stdout. The CLI diagnostic contract defines every class, timestamp, elapsed scope, HTTP status, and validated CF-Ray field. Use the correlation procedure below to investigate the exact failed attempt without exposing policy material or changing authentication.

If policy storage is missing or corrupt, restore the migration/schema and last reviewed valid singleton row through the controlled recovery process. PUT cannot repair an unreadable current policy, and a deleted row is not a supported way to clear rules. For a healthy policy, deliberate deactivation is an authenticated revision-checked PUT with rules: []. Do not remove the guard, suppress load failures, or treat a D1 outage as an empty policy. No live rollout or credentials are required for the unit and local Workerd integration tests; the ordinary Worker suite applies migration 0016 explicitly through the migration runner and exercises the same production checks.

#Policy coordinator upgrade

This Worker adds SQLite Durable Object migration v2 and the POLICY_COORDINATOR binding; there is no new D1 schema migration or CLI change. All policy access uses the single fixed string-rewrite-policy object. Never rename it, split it by pool, or send traffic to two independent policy namespaces backed by the same D1 row.

No policy write fence or exclusion of mixed Worker versions is required. Admin API writes mediated by the coordinator are immediately visible. During a mixed-version rollout, writes by an older Worker bypass the coordinator and become visible within 60 seconds, just like manual D1 edits and restores. Reads require a successful primary reload after snapshot expiry; failures fail closed with 503 string_rewrite_policy_unavailable instead of expired rules. During the authorized rollout, verify authenticated admin/caller GETs through each ingress observe the reviewed policy within that bound. Do not log rule contents in rollout evidence.

Keep migration v2 and its binding/class in rollback builds; roll back application behavior with a forward deploy if the platform cannot roll back across the DO migration. Returning to coordinator reads does not require an object restart: direct-D1 changes are picked up at the next snapshot expiry, within 60 seconds. Keep the existing object name and namespace.

For emergency direct D1 repair, restore the reviewed singleton row and verify authenticated policy GETs match the primary within 60 seconds. A restart is not required. Prefer the revision-checked admin API for normal edits because it publishes immediately; missing or corrupt rows still require storage repair, and failed reloads never authorize fallback.

The Worker remains GET-only. Local GitHub writes use the caller's own credentials and need the updated CLI's supported content preparation. Direct real gh, browsers, Git pushes, old local-write clients, unsupported obfuscation, and other programs are outside this boundary. Previously published content and existing stored cache/audit records are not retroactively purged by importing a policy.

#Build, test, deploy

pnpm install
pnpm check        # format:check + lint + vitest + build + go test + go vet
pnpm test         # vitest only
pnpm test:e2e:cli-worker # compiled CLI → local Workerd/D1/DO → public GitHub
pnpm run deploy                 # authoritative Worker, then public proxy
pnpm run deploy:authoritative   # authoritative Worker only
pnpm run deploy:public-proxy    # octopool.dev proxy Worker only
pnpm e2e                        # smoke-test the live deployment

pnpm check is the deterministic TypeScript + Go gate. The networked release gate pnpm test:e2e:cli-worker additionally crosses the compiled CLI, local Workerd, migrated D1, a real Durable Object, and public GitHub. The Go CLI also builds/tests with go build ./cmd/octopool and go test ./....

The hosted deployment runs both commands with CLOUDFLARE_API_TOKEN loaded from the Molty item OpenClaw Services Cloudflare API Token; both Worker configs are pinned to the same Services account. Use the config-qualified deploy:public-proxy command when proving proxy deploy access so Wrangler cannot target the authoritative Worker by accident. Self-hosters who don't operate the public-host proxy can run deploy:authoritative only. Pushing to main does not auto-deploy the Workers — only the docs site has a GitHub Pages workflow. Run pnpm run deploy whenever Worker code or landing-page CSS needs to ship to production.

#Worker integration test isolation

pnpm test:e2e runs real Workerd storage with a test-owned lifecycle in test/e2e/setup.ts. Each isolated file applies the actual migrations once in beforeAll, verifies that D1 was pristine, and captures an immutable migrated baseline. The original 10-second hook deadline, 15-second test deadline and default file parallelism remain in effect; migration initialization still has to finish within its hook deadline.

Before each test, D1 restoration discovers the current application schema and foreign-key dependencies, removes views/triggers and child tables before parent tables, then replays the baseline in one atomic batch. This covers renamed or dropped tables and unexpected schema objects as well as rows. The ordinary path uses three D1 requests: schema discovery, batched foreign-key discovery, and the atomic teardown/replay. SQLite/Cloudflare internals are preserved; the export restores application AUTOINCREMENT sequences, migration records and the original string-policy seed timestamp. Unsupported foreign-key cycles (including self references), virtual tables and SQLite statistics tables fail explicitly before destructive restoration.

test/e2e/d1-baseline.ts isolates a version-specific local runtime contract: Miniflare 5.20260815.0-alpha in the test pool implements PRAGMA miniflare_d1_export(?,?,?);, also used by Wrangler 4.135.0 with Miniflare 5.20260918.0-alpha. The D1 binding accepts numeric flags bind(0, 0) for schema plus data with no table filter. Its single result row is an array of complete SQL statements, which the adapter validates and submits individually through D1.batch. It must never be joined and fed to newline-splitting D1.exec, replaced by D1.dump, or used as a production API. Dependency updates must revalidate this export/replay contract.

Storage cleanup owns the configured POOL_COORDINATOR, ACTIONS_LOGS and native Cache API writes. It enumerates persisted coordinator IDs, including dormant objects, deletes their alarms and storage inside each real object, then evicts each instance so the constructor recreates SQL and discards memory/timers/waiters. R2 cleanup lists every page and deletes at most 1,000 keys per call. The Cache API ledger forwards default/named cache operations to the native implementations, waits for owned puts, and deletes their recorded keys. It only owns writes made through that ledger; it cannot purge arbitrary untracked cache entries. Tests that explicitly provide mock caches retain their own cache semantics.

The harness tracks requests and scheduled calls, draining their execution contexts even when handlers throw. callWorker retains its cold-config-cache behavior; callWarmWorker explicitly preserves a warm isolate. Concurrency tests register gate releases with ownedWork and also release/drain in finally. After each test, gates are released and requests, contexts and cache writes drain before mocks/globals are restored. A failed cleanup or timed-out hook/body poisons the file lifecycle: subsequent test bodies fail rather than racing the old promise. The deadline watchdog does not cancel Workerd operations or extend Vitest's timeouts. Test code must register owned async work and release gates; arbitrary untracked timers/writes are outside this lifecycle's ownership.

The lifecycle never calls the native global storage reset(). The regression uses the official global abortAllDurableObjects() to reproduce a dormant internal D1 actor; natural 11-second idle independently reproduces the same retained-storage failure in the pinned runtime. evictAllDurableObjects() only reaches the current Worker's explicit Durable Object bindings and does not establish D1 dormancy through its service binding. Per-stub eviction remains appropriate for testing the coordinator's constructor and storage cleanup.

#Backend-work admission

Each pool has a separate BackendAdmission Durable Object. It admits up to CLIENT_BACKEND_CONCURRENCY requests per authenticated (caller_id, client_name); the default is 16, while the CLI's default machine-wide relay-read limit (OCTOPOOL_RELAY_CONCURRENCY) is 8. These limits cover different lifetimes: a CLI slot ends when its HTTP attempt finishes, but server permits also cover detached stale-while-revalidate refreshes and can survive a timed-out or cancelled client. Equal limits therefore cause avoidable overloads even when every CLI queues locally. Background refresh admission uses the same live count with a threshold of half the server cap, at most 8, leaving 8 default slots for a full foreground wave.

The server cap remains a backstop for older CLIs, disabled or failed-open local limiters, and accumulated abandoned permits. Set customized server limits to at least twice the client limit to preserve this headroom. Local queues wait up to 30 seconds and cannot guarantee admission after repeated cancellations or cleanup failures. The client name comes from the stored token row, never a request header. Tokens rotated for the same client keep the same allowance. Different clients have independent allowances, including two clients belonging to the same caller. This is a concurrency cap, not a fair queue or a pool-wide ceiling.

Fresh reusable shared and identity-cache responses do not acquire permits. Misses, forced-fresh reads, conditional revalidations, and cache-bypass reads acquire one permit before publication coordination or backend fetching. It covers token-free and pooled transports, visibility probes, identity retries, pagination, and cache publication. A cached body that still needs a live visibility, PR-state, or log-completion probe does backend work and needs a permit. Authentication, membership verification, protection policy loads, and cache eligibility reads precede this gate; it does not replace bounded client-side launch concurrency. Native-only handoffs do not acquire permits.

Admission performs only synchronous local SQLite decisions, independently of the pool coordinator's publication block and D1 round trips. A permit lasts 30 seconds, renews every 10 seconds, and has a fixed 60-second overall admitted-work deadline. Pending acquisition and renewal promises belong only to the current Worker request. Renewal failure, expiry, the overall deadline, and detectable caller cancellation abort active GitHub requests and prevent new backend work. Completion releases the exact permit, waiting up to one second for acknowledgement on normal completion so the next CLI attempt does not race its predecessor's cleanup. Already-cancelled work returns immediately; slow release continues in waitUntil, and failed cleanup or a terminated request loses its slot at durable expiry. Live counts survive object restart. Expired rows do not count and drain in batches of 64 on admission; an idle object may retain expired rows until its next request. Already submitted storage operations cannot be recalled and may finish after cancellation under the existing publication fencing rules.

Normal response completion also aborts the request's backend-egress signal. Cache publication is awaited before the response; background audit inserts, D1-to-edge warming, and rate snapshots use storage operations without that signal or the active-work guard, so their waitUntil work can finish after the response.

Denials return 424 fallback_local with reason relay_overloaded. The CLI's default budget retries three times with jittered backoff (at most eight seconds of total sleep), then attempts guarded native fallback; OCTOPOOL_NO_FALLBACK=1 disables that handoff. Audit records include the authenticated caller, token ID, client name, and fallback_reason=relay_overloaded. Inspect octopool stats --client <name> for that client's fallbacks and the fallback-reason aggregate for overloads. Infrastructure overload shares this reason. Permits store only a random ID, caller/client key, expiry, and deadline, with no request contents.

Upgrades must include the new BACKEND_ADMISSION binding and v3 SQLite-class migration from wrangler.jsonc. No D1 migration or CLI update is needed. During a rolling upgrade, old Worker invocations do not participate in admission.

#Coordinator expiry retention

Every PoolCoordinator.selectIdentity invocation performs two synchronous, indexed deletions: at most 16 leases and 16 cooldowns whose expiry is at or before one captured coordinator clock. Each statement selects and deletes its batch atomically, so a prior lease renewal or cooldown extension that is still live survives. Sticky and unavailable selections also clean up. Feedback and snapshots do not trigger cleanup, and rate history is retained. Distinct contents paths can create distinct route keys; route normalization and cooldown scope remain unchanged.

The DO constructor adds leases_expiry (expires_at, route_key) and cooldowns_expiry (expires_at, identity_id, route_key) idempotently. These covering indexes bound expiry batch selection without scanning or sorting the whole history; snapshots also use expiry indexes. This is additive DO schema initialization, not a D1 migration. The first activation builds each index over retained rows, and subsequent writes maintain it. Existing rows and legacy RPC/table contracts survive reconstruction. A Worker rollback can leave the harmless indexes in place; older code simply stops this cleanup.

The per-call budget does not bound absolute storage during arbitrary bursts or guarantee when idle rows are physically deleted. No timer, alarm, new RPC, or global object sweep is added. Deletions make SQLite space reusable, without promising file-size shrinkage or hosted billing savings. Hourly publication-owner cleanup retain their separate authority and budgets described in Cache.

#SQL catalog

Runtime SQL lives in sql/queries/*.sql with sqlc annotations. sqlc.yaml points sqlc at the D1 migrations plus the Durable Object SQLite schema. pnpm sql:compile validates the catalog without generating an unused Go package. pnpm sql:generate updates src/generated/sql.ts, the D1/Durable Object query constants used by the Worker.

Run pnpm sql:generate after changing query files. pnpm check runs pnpm sql:check first and fails if generated SQL artifacts are stale.

#Smoke test

test/e2e.sh resolves octopool.dev by default, then asserts:

• GET / returns the landing page. On octopool.dev it must show the Homebrew install command, not the GitHub login CTA.
• GET /dashboard redirects to the authoritative app host on octopool.dev, and to GitHub login on octopool.openclaw.ai.
• GET / with Accept: application/json returns the JSON health body ("ok":true, "service":"octopool").
• GET /.well-known/octopool returns discovery metadata for CLI self-host login.
• GET /v1/pools/maintainers/health without a token returns 401 missing_auth.
• POST /v1/github/request without a token returns 401 missing_auth.

Override the host/resolver with OCTOPOOL_E2E_HOST / OCTOPOOL_E2E_RESOLVER.

#Observability

Both authoritative and public-proxy Worker source configurations enable observability at full sampling (head_sampling_rate: 1). Every validated relay request from an authenticated caller to an existing pool writes an audit_events row (caller, client, pool, route key/kind, identity, status, error/fallback classification, duration, cache hit/miss/bypass status, coalesced-fill marker, and parsed requested_max_age in seconds); parse, authentication, string-protection, and pool-lookup failures occur before that boundary. Secrets and request bodies are never recorded.

requested_max_age = 0 identifies callers requiring upstream validation; positive values bound cache age, and NULL means no parsed maximum age. Older rows are also NULL, so compare post-upgrade windows when measuring hit rates by caller freshness. For example, group requests by route, requested_max_age, and cache_status to separate live reads from requests accepting cached bodies.

#Correlating policy-load failures

Preserve the exact safe failure line, its attempt_utc and elapsed_ms, and the binary version/revision in the external incident record. The CLI does not persist them. The timestamp uses the client's clock; allow for known clock skew when bounding the search window from start through elapsed time. setup and request failures do not establish any HTTP response. A local/merge failure with http_status=200 describes a successful GET followed by a failing client check, not an HTTP outage.

In authorized Cloudflare edge/Worker observability, filter by method GET, the expected /v1/pools/<pool>/string-rewrites path, that UTC window, and observed status. Add cf_ray when present and supported by the deployed log fields; it is only a strictly validated, unauthenticated correlation hint. Inspect each relevant layer separately: the public proxy's edge/security events, the public-proxy Worker invocation/upstream outcome, the authoritative Worker invocation/authentication/policy outcome, and authorized D1 health or query-failure telemetry. Direct authoritative URLs skip the public-proxy hop. A 403 alone does not identify which layer refused the request; a 503 alone does not identify D1 as the cause. Correlate observed events before assigning a source.

Full-sampling settings in source do not prove deployed log availability or reliable Ray-ID joins across proxy and authoritative hops. Verify those deployed fields and joins before using them for attribution. Missing logs or IDs are not proof of where a failure arose: an edge rejection can occur before Worker execution, expected policy errors do not emit the unexpected-exception console event, and retention/access/field coverage may differ. Policy GETs and authentication/protection failures bypass relay audit_events and stats; their absence there is expected, not evidence that no request occurred.

whoami reads saved metadata, health uses another caller endpoint, and admin string-rewrites status uses separate admin authentication and does not merge local rules. They test different boundaries. A later unchanged guarded invocation can succeed because each attempt reloads policy and failures are not cached; that recovery does not establish the historical failure's cause. A separate Cloudflare-shaped 403 likewise cannot attribute an earlier generic failure. These diagnostics improve evidence, not resolve an outage. Never dump policy files, error response bodies, arbitrary headers, tokens, or debug traces, and never reconfigure authentication as a diagnostic shortcut.

#Relay retention and stats

The main Worker has an hourly cron trigger at minute 17. It deletes bounded batches of cache entries after each entry's route-specific stale deadline and audit rows older than 30 days; this retains every configured stale window and the full supported stats window.

Each table emits an octopool.maintenance log with deleted rows, completed batches, duration, failure status, and budget_exhausted. Each run allows 20 batches of 500 rows per table. A full final batch means capacity was exhausted; check expired-row counts and oldest timestamps to confirm a backlog. For repeated exhaustion, compare arrival and cleanup rates. A failed table stops the run before later tables.

GET /v1/pools/<pool>/stats?since=24h returns pool-, caller-, and client-specific cache stats, plus bounded backend-by-route and local-fallback-reason aggregates. Anonymous API conditional replacements and 304 verifications are attributed to github_api prospectively. Existing misattributed history is not backfilled and expires under normal audit retention. This correction requires no schema or cache-generation change. During a rolling upgrade, old Workers may still emit the former attribution and serve cached statistics 202; new readers reject pending entries, including late writes, without purging valid ready data. The CLI wraps this as octopool stats. The browser dashboard at /dashboard exposes the same data plus identity health, live leases, seven-day normalized request patterns and outcome causes, and per-caller/client usage — see Dashboard.