Last reviewed: 2026-07-31

Direct answer

A reliable design for CometAPI rate limits for parallel coding agents starts with one shared admission controller in front of every agent process using the same gateway. Each model call should enter a queue with its route, model ID, task deadline, and retry state. The controller, rather than an individual agent, decides when the call may leave the application.

The CometAPI rate-limit and concurrency guide recommends controlling concurrency before requests leave the app. It also recommends retrying 429 responses with exponential backoff and jitter, lowering burst traffic when repeated retries occur, and recording model- and route-level metrics. Those controls work together: a semaphore limits simultaneous calls, a queue absorbs short bursts, and bounded jittered retries stop failed workers from immediately colliding again.

Do not copy an example concurrency value into production and treat it as an account limit. The CometAPI guide demonstrates a semaphore value of five and a capped retry delay, but it does not present those example values as universal service guarantees. Start conservatively, run a controlled fixture set, and tune from observed queue time, latency, status, model, and route evidence.

Use one retry owner. If the SDK, agent worker, queue consumer, and workflow orchestrator all retry independently, a single rejected call can become several uncoordinated calls. Put the attempt budget in the admission layer, disable overlapping retry loops where possible, and return a terminal result when that budget or the task deadline is exhausted.

Who this is for

This guide is for teams running two or more coding agents through a shared CometAPI integration, including parallel test repair, repository analysis, review, migration, or documentation tasks. It is especially useful when separate workers appear healthy in isolation but produce 429 responses when they start together.

You should already know which agent processes send model calls and where their gateway client is constructed. You do not need a distributed scheduler on day one: a process-wide queue is enough for one runtime, while multiple hosts need a shared coordinator or another mechanism that enforces the same aggregate policy.

Key takeaways

  • Put a shared queue or worker pool before the gateway client; per-agent semaphores do not enforce a shared ceiling.
  • Partition admission and metrics by route and model ID so one busy workload does not hide another.
  • Treat configured concurrency as a tested operating value, not a number inferred from sample code.
  • Retry 429 and approved temporary server failures only; fail request-shape and other nonretryable errors without a loop.
  • Use capped exponential backoff with jitter and one retry owner.
  • Bound queue age, attempt count, and end-to-end task time so stale calls cannot complete after an agent has moved on.
  • Keep prompts, repository contents, model output, request headers, and credential material out of rate-limit logs.
  • Reduce admissions when 429 responses repeat; adding more retries is not a capacity fix.

Sources checked

  • The first-party CometAPI concurrency documentation supports pre-request concurrency control, jittered backoff for 429 responses, lower burst traffic after repeated failures, and per-model and per-route monitoring.
  • The Anthropic rate-limit reference shows why model traffic can be constrained along more than one dimension: its API documents request, input-token, and output-token limits, short-interval burst enforcement, acceleration limits, and a retry hint for 429 responses. These are provider-specific facts, not a claim that CometAPI exposes identical quotas or headers.
  • GitHub REST API best practices provide a relevant coding-agent analogy: queue concurrent calls, pause when a retry hint is present, increase wait time after repeated rate-limit errors, and stop after a defined number of retries. GitHub rules should not be copied as CometAPI limits.
  • The supplied AWS retry and jitter article currently resolves to AWS Builder Center. The accessible page text confirms that destination but does not expose enough detail to set a queue value or retry rule here, so no operational setting below depends on it.

Contract details to verify

Before enabling parallel workers, write down the runtime contract. Verify it against your current CometAPI account behavior, client version, selected models, and controlled tests. At minimum, record:

  • Which processes, hosts, and scheduled jobs share the same admission budget.
  • The exact route and model ID attached to each queue partition.
  • The initial concurrency per partition and the total global ceiling.
  • Maximum queue age and the end-to-end deadline inherited from the agent task.
  • Which response classes are retryable, who owns the retry, and how many attempts are permitted.
  • Base delay, maximum delay, jitter method, and cooldown behavior after consecutive 429 responses.
  • Whether the response actually exposes a reliable retry hint. Anthropic and GitHub document such hints for their own APIs; the refetched CometAPI guide used here does not document one, so verify rather than assume.
  • Which sanitized fields are emitted for every queued, dispatched, completed, retried, and abandoned call.
  • The operator threshold for lowering concurrency, pausing a partition, or escalating.

An initial configuration can be deliberately conservative. The following values are a test hypothesis, not published CometAPI limits:

queue_scope: shared-gateway
partition:
  - route
  - model_id
initial_concurrency_per_partition: 2
global_concurrency: 4
max_attempts: 4
base_delay_ms: 1000
max_delay_ms: 30000
jitter: full
cooldown_after_consecutive_429: 3
max_queue_age_ms: 120000

Keep timeout policy separate from admission policy. Concurrency limits how many calls are in flight; a timeout limits how long one call can occupy a slot. Use the companion guide to set explicit timeout rules before increasing parallelism.

Happy-path operator workflow

  1. Assign every agent task a short internal run ID and a deadline before it can enqueue model work.
  2. Add the request to the shared queue with route, model ID, priority, enqueue time, and attempt number. Do not store the full prompt in the scheduler record.
  3. Let the admission controller dispatch only when both the global ceiling and the matching route-model partition have capacity.
  4. Record queue wait time at dispatch. On completion, record status, latency, attempt count, and a small outcome category.
  5. Release the slot exactly once, including when the client raises an exception or the task is cancelled.
  6. Mark the agent step complete only if the returned result still belongs to the active run and has not passed its deadline.
  7. Increase concurrency one step at a time only after a representative fixture batch completes without repeated 429 responses and with acceptable latency and queue age.

A healthy run should show bounded queue growth, mostly first-attempt completions, and no cluster of retries at the same instant. Fast completion alone is not sufficient if one model partition is quietly accumulating a queue.

Error-path operator workflow

  1. When a call returns 429, stop immediate admission for the affected partition. Do not freeze unrelated partitions unless the evidence shows a shared limit.
  2. Emit a sanitized failure event and increment consecutive-429 and retry counters for that route-model pair.
  3. If the live CometAPI response contract exposes a verified retry hint, wait at least that long. Otherwise, apply the configured capped exponential delay plus jitter.
  4. Requeue the same logical operation under the existing task deadline. Do not create a fresh agent task or reset the attempt counter.
  5. If another 429 arrives, reduce the affected partition’s admission rate or extend its cooldown. Repeated rejection is evidence that the current burst is too high.
  6. When the attempt budget or deadline is exhausted, return a terminal rate-limit outcome. Let the agent stop, defer, or escalate according to its task policy rather than looping indefinitely.
  7. For a nonretryable response, fail immediately with a sanitized category. Retry a temporary server failure only when the operation is safe to repeat and the approved policy includes it.

This flow separates recovery from denial. One delayed retry may recover from a short burst; a continuing stream of 429 responses should change admissions, not merely create a longer retry queue.

Sanitized logging fields

A useful event can remain small:

event: model_call_completed
run_id: run-042
agent_id: worker-03
route: chat-completions
model_id: configured-model
partition_id: chat-model-a
attempt: 2
http_status: 429
latency_ms: 812
queue_wait_ms: 1540
retry_delay_ms: 2718
rate_limit_signal: response-status
outcome: retry-scheduled

Add timestamps, deployment version, queue depth, and worker location when they help correlate events. Do not log prompts, source files, generated patches, full responses, request headers, or credential material merely to diagnose throttling. If response text varies, map it to a small reviewed error category instead of copying it wholesale.

Use the same fields on successful calls. Otherwise, operators can count failures but cannot compare them with normal route-model latency or determine whether lower concurrency improved the outcome. The guide on how to review coding-agent telemetry logs can help keep that comparison reviewable.

Failure modes

Every worker has its own semaphore

Four agents with a local limit of four can still produce sixteen simultaneous calls. The workers comply locally while violating the intended aggregate policy. Move admission to a shared layer or allocate explicit sub-budgets whose sum cannot exceed the tested global ceiling.

Sample code becomes an undocumented service contract

A team copies the semaphore value from a documentation example, launches that many workers per host, and assumes the number is guaranteed. Examples demonstrate a mechanism; they do not establish the limit for every route, model, account, or traffic shape.

One global queue causes head-of-line blocking

A long-running model partition occupies every slot while short review calls wait behind it. Keep a global safety ceiling, but add route-model partitions and fair scheduling. Partitioning also makes repeated 429 responses diagnosable without claiming that each partition has an independent provider quota.

Retries multiply across layers

The client retries, the queue retries the client, and the agent retries the queue. Attempt counts become misleading and load rises during failure. Select one retry owner and propagate the terminal result upward.

Every error is treated as temporary

Request-shape mistakes and other nonretryable failures return unchanged on the next attempt. Retrying them wastes queue capacity and can obscure the real defect. CometAPI’s guide limits retries to 429 and temporary server failures; classify everything else before scheduling another call.

Jitter exists, but admissions never fall

Randomized delays spread retries, yet new work continues entering at the original rate. The queue grows and old tasks consume every recovered slot. After repeated 429 responses, lower burst traffic, pause the affected partition, and reject or defer work that exceeds its deadline.

Rate-limit logs leak task content

Operators copy prompts, repository snippets, or full model responses into logs to explain a 429. Those payloads do not establish whether concurrency was excessive. Keep the operational fields and remove the content.

Rate control is mistaken for cost control

A run can remain below its concurrency ceiling and still consume more tokens or money than intended. Track rate, latency, and queue health here, then trace CometAPI usage by run as a separate control.

FAQ

What concurrency value should we start with?

There is no universal value in the refetched evidence. The CometAPI documentation shows a semaphore as an implementation example, not an account-wide guarantee. Start low enough to establish a clean baseline, run a representative fixed workload, then raise one partition by one step while watching 429 frequency, latency, queue wait, and task deadlines.

Should every 429 response be retried?

CometAPI recommends retrying 429 responses with exponential backoff and jitter, but the retry still needs an attempt cap and task deadline. If the task is already obsolete, the queue is beyond its safe age, or the attempt budget is exhausted, return a terminal outcome instead of retrying.

Should the scheduler honor a retry hint?

Honor it when the actual response contract exposes one and your integration has verified its meaning. Anthropic and GitHub document retry hints for their own APIs, but that does not prove the same header is available through every CometAPI route. Keep a bounded fallback delay policy for cases where no verified hint is present.

Why partition by model if we do not know the provider’s exact quota?

Partitioning improves fairness and evidence. CometAPI recommends model- and route-level monitoring, while Anthropic’s documentation demonstrates that an upstream model API can apply model-specific request and token dimensions. A partition lets you identify which workload is saturated without asserting that CometAPI mirrors Anthropic’s limits.

Can each coding agent manage its own retries?

Only if a higher layer can still enforce the aggregate attempt and concurrency budgets. In most shared-gateway deployments, centralized retry ownership is easier to reason about because the scheduler can see queue depth, recent 429 responses, deadlines, and total in-flight work.

Does a concurrency limiter replace call timeouts?

No. A limiter controls admission; a timeout releases capacity when a call takes too long. You need both, plus cancellation handling, so an abandoned agent run cannot retain an in-flight slot.

When should the operator escalate?

Escalate when 429 responses continue at the lowest practical concurrency, response semantics differ from the verified contract, cooldowns do not restore normal calls, or important tasks repeatedly expire in the queue. Record the route, model ID, sanitized statuses, timings, queue depth, and attempted mitigations. Use the task stop and escalation guide to keep the handoff bounded.

Reader next step

Build the smallest shared admission test before increasing production parallelism. Route a fixed set of secret-free fixtures through one worker, save the sanitized baseline, and repeat with two workers. Then use a local mock or controlled test harness to return a synthetic 429 and verify that only the affected partition pauses, retries are jittered and bounded, stale work expires, and no task content enters the logs. Do not create failure evidence by deliberately overwhelming a live service.

Review the resulting config with the people who own agent deadlines, gateway behavior, telemetry, and cost controls. Write the tested concurrency and retry policy next to its evidence and review date so later workers do not treat it as a permanent provider guarantee.

When you are ready to connect the shared queue to a multi-model gateway, Start with CometAPI .