Last reviewed: 2026-08-13

Direct answer

A reliable CometAPI screenshot input is a small, intentional bug fixture rather than an unexplained screen capture. Start with one redacted screenshot, preserve enough surrounding interface to locate the problem, and pair it with the route or screen name, viewport, reproduction steps, actual behavior, and expected behavior. Then send the text and image together using the multimodal envelope supported by the selected endpoint and model.

CometAPI documents an image_url content item for Chat Completions and a detail choice of low, high, or auto. Use high when the task depends on small text, alignment, controls, or dense interface state. Use low for a coarse layout question when fine detail is unnecessary. Do not assume that every model accepts the same image form simply because it is reachable through one gateway. CometAPI explicitly notes that fields and behavior can vary among the providers it routes to.

Treat transport acceptance, visual understanding, and bug diagnosis as three separate checks. A successful response only proves that a request completed. It does not prove that the model read the decisive pixels or identified the cause. Require the agent to separate visible observations from inferences, name missing evidence, and propose a verification step before it edits code. A screenshot-input preflight complements the site’s visual-regression testing workflow ; it does not replace an automated before-and-after comparison.

Who this is for

This guide is for developers and operators who send visual bug reports to coding agents through CometAPI, especially teams that may switch among OpenAI, Claude, and Gemini model families. It is useful when screenshots come from issue trackers, browser test runs, support reports, or manual reproduction sessions.

The workflow assumes that a human still controls repository access, approves the task scope, and reviews the resulting patch. It is designed for diagnosis and evidence gathering before an agent receives permission to make changes.

Key takeaways

  • Package a screenshot with explicit expected and actual behavior, not a vague request to fix the page.
  • Redact private data before upload, while preserving labels and layout needed for diagnosis.
  • Lock the endpoint, model ID, content envelope, image-reference method, media type, and detail setting into a tested contract.
  • Test one known-good image and at least one controlled error case before adopting a new model route.
  • Log dimensions, byte count, media type, detail, model, status, finish reason, and a short image hash prefix. Do not log raw image bytes or the full remote reference.
  • Treat an accepted request with an unsupported, vague, or ungrounded answer as a semantic failure.
  • Recheck the contract whenever the model or endpoint changes.

Sources checked

The CometAPI Chat Completions documentation is the primary gateway contract. It shows multimodal content as an array containing text plus an image_url object, explains the low, high, and auto detail settings, lists relevant finish reasons, and warns that request and response support varies across providers. It also advises using the Responses endpoint for Codex and some newer reasoning model families.

The OpenAI images and vision guide confirms that image analysis is available through both Responses and Chat Completions for supported models. Its model-dependent framing is important: endpoint compatibility does not establish that an arbitrary model accepts images.

The Gemini image-understanding guide documents URL references, inline image data, and uploaded files in Gemini’s native interface. It also states that inline data shares a 20 MB request limit with the text and instructions. Those native forms and limits are useful test inputs, but they should not be copied into a CometAPI request without confirming the gateway mapping.

The Claude vision guide documents URL, embedded, and uploaded-file sources in Claude’s native API, plus image formats, resolution behavior, and payload limits. It also explains that embedded image bytes remain in the resent history of a multi-turn agent workflow, which can increase request size and latency. That makes repeated-image behavior a contract item, not an implementation detail.

Contract details to verify

1. Lock the endpoint and model together. Do not maintain a generic setting called vision enabled. Record the exact endpoint family and model ID that passed the fixture. CometAPI’s Chat Completions example is not a universal template for every Codex or reasoning model; its documentation directs some of those families to Responses. A model change therefore requires an endpoint check as well as a model-name change.

2. Build the smallest useful fixture. Capture the failing state at the viewport where it occurs. Keep the component, nearby labels, and enough page structure to establish location. Remove unrelated browser chrome when it adds no evidence. Redact names, messages, account details, internal hostnames, and identifiers before the image leaves the approved environment. After redaction, inspect the exported file itself rather than trusting a temporary overlay in the capture tool.

Pair the image with a compact report containing:

  • screen or route name;
  • viewport dimensions and device scale when relevant;
  • deterministic reproduction steps;
  • actual visible behavior;
  • expected visible behavior;
  • the suspected component only when it is clearly labeled as a hypothesis;
  • the output requested from the agent, such as observations, likely code surfaces, and a verification plan.

3. Use the documented multimodal envelope. A Chat Completions fixture can follow this shape after VISION_MODEL_ID and SCREENSHOT_REFERENCE are supplied by approved runtime configuration:

{
  "model": "VISION_MODEL_ID",
  "messages": [
    {
      "role": "developer",
      "content": "Separate visible evidence from inference. Do not edit code until the reproduction is stated."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "At 1280x720, the checkout button overlaps the total. Expected: a 16px gap. Identify visible evidence, likely code surfaces, and a verification plan."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "SCREENSHOT_REFERENCE",
            "detail": "high"
          }
        }
      ]
    }
  ],
  "max_completion_tokens": 800
}

Keep this fixture separate from the production agent prompt. Its purpose is to establish whether the selected route accepts the content structure and produces grounded visual observations. If another endpoint uses a different envelope, maintain a separate fixture instead of silently translating fields in application code.

4. Verify the image transport. A remote reference must be retrievable by the service, return actual image bytes, and remain valid for the duration of the request. A browser being able to display an image from an authenticated session does not prove that the model service can retrieve it. Do not put private access material in a query string. Prefer an approved storage mechanism with narrowly scoped access and a lifecycle that matches the task.

Native provider documentation shows that embedded data and uploaded-file references are also possible in some interfaces. That does not prove that the same native field is supported through CometAPI’s compatibility layer. Test the exact gateway request you intend to operate.

5. Choose detail and resolution intentionally. Start with high for small text, subtle spacing, dense tables, or a control-state bug. Try low for a broad layout classification where speed and lower token use matter more than fine text. Avoid sending an enormous desktop capture when a smaller crop preserves the necessary pixels. Provider documentation shows that large images may be resized and consume visual tokens differently, so retaining more pixels is not automatically more useful.

6. Define semantic acceptance. A passing answer should identify at least two facts visible in the fixture, distinguish those facts from likely causes, state what cannot be determined from the screenshot, and propose a reproducible verification. Reject an answer that invents file names, selectors, stack traces, or hidden application state. Only after this check should the screenshot be attached to a scoped coding-agent task brief .

Happy-path operator workflow

  1. Reproduce the issue and record the viewport, route name, expected result, and actual result.
  2. Capture one screenshot, apply permanent redactions, and inspect the exported file.
  3. Record its media type, dimensions, byte count, and a short digest prefix.
  4. Confirm the chosen endpoint and model are expected to accept images.
  5. Send the known-good fixture with one image and a bounded response length.
  6. Confirm a successful transport status, a normal completion reason, and an answer grounded in visible details.
  7. Save only sanitized request metadata and the acceptance result.
  8. Ask the agent for a reproduction and verification plan before allowing repository edits.

Error-path operator workflow

  1. For a client or invalid-request error, stop retrying the unchanged payload. Check the model ID, required messages, content-array shape, image-reference accessibility, and endpoint family.
  2. For an authentication failure, stop the run and repair the execution environment without copying authentication material into logs or issue comments.
  3. For a rate-limit response, use bounded exponential backoff with jitter and record the attempt count. Do not let a visual diagnosis loop retry indefinitely.
  4. For a server error, retry only the diagnosis request, with a small bound. Do not automatically repeat any later tool action or code mutation.
  5. For a successful status with no specific visual evidence, classify the run as a semantic failure. Verify modality support, try a more useful crop or higher detail, and compare against the known-good control.
  6. If the finish reason indicates a length or filtering stop, do not treat the partial response as authorization to patch code. Adjust the response bound or escalate for review.

Sanitized logging fields

A useful event contains contract evidence without retaining the screenshot or full prompt:

{
  "event": "coding_agent_image_request",
  "request_id": "req_1042",
  "model": "VISION_MODEL_ID",
  "endpoint_family": "chat_completions",
  "image_reference_kind": "url",
  "image_sha256_prefix": "8f31c2a4c977",
  "mime_type": "image/png",
  "width_px": 1280,
  "height_px": 720,
  "byte_count": 284112,
  "image_count": 1,
  "detail": "high",
  "prompt_template_version": "bug-triage-v3",
  "attempt": 1,
  "redaction_applied": true,
  "http_status": 200,
  "finish_reason": "stop",
  "usage_total_tokens": 1430,
  "status": "accepted",
  "error_type": null,
  "error_code": null
}

Do not log raw image bytes, the full image reference, request headers, cookies, unredacted user text, or the model’s complete answer by default. Keep the image only in the approved bug-evidence system, with its own retention and access rules. Use the metadata to compare models, investigate failures, and connect image use to the site’s CometAPI cost-tracing workflow .

Failure modes

  • The selected model does not accept images. The gateway is reachable, but the model rejects the content item or ignores it. A per-model known-good fixture catches this before a real bug report is routed.
  • The wrong endpoint is used. An envelope valid for one endpoint is sent to a model family that expects another. Lock endpoint and model in the same compatibility record.
  • The image reference is inaccessible. A developer’s browser can display the screenshot, but the remote service cannot retrieve it. Check retrieval from the operating path without exposing private access material.
  • The media type and bytes disagree. Renaming a file does not convert it. Inspect the exported image and record its actual media type.
  • Redaction removes the evidence. A large opaque block hides the alignment or control state needed for diagnosis. Reframe the capture or substitute safe labels instead of exposing private data.
  • The crop is too tight. The agent sees the overlap but cannot identify the page region, neighboring component, or responsive breakpoint.
  • The screenshot is too large or too compressed. Large captures can increase payload and visual-token cost, while aggressive compression can make text unreadable. Preserve the smallest image that still contains the deciding evidence.
  • Multiple images are unlabeled. Before and after states become ambiguous. Label each image in text and state the comparison requested.
  • Embedded images accumulate in history. Provider documentation shows that repeated image bytes can enlarge multi-turn requests. Keep diagnosis short and test the route’s reuse behavior.
  • A successful response is mistaken for a correct diagnosis. The model may describe the interface fluently while inventing an underlying cause. Require observation, uncertainty, and a verification plan as separate fields in the review.
  • Retries expand into duplicate work. A retry that begins as image analysis can repeat later agent actions if the stages are not separated. Keep screenshot diagnosis read-only and make every mutation a separately approved step.

FAQ

Should every screenshot use high detail?

No. CometAPI describes high as more detailed and more token intensive, while low is faster and uses fewer tokens. Use high when the bug depends on small text, borders, alignment, or dense controls. Use low for coarse layout questions, then escalate only when the control fixture shows that more detail is necessary. auto is convenient, but it makes an operator’s cost and fidelity choice less explicit.

Can I send more than one screenshot?

Provider documentation supports multiple images in several interfaces, but the usable count and payload behavior are model and route dependent. Start with one image. For a before-and-after comparison, label each screenshot in the text and ask one precise question. Add a compatibility test before increasing the count in an automated workflow.

Can I embed the screenshot instead of using an image reference?

Google and Anthropic document embedded image data in their native APIs, while the supplied CometAPI Chat Completions example uses image_url. Native support does not establish compatibility through the gateway. Use the documented CometAPI form unless the exact endpoint and model have passed a separate embedded-data fixture. Embedded bytes can also make multi-turn histories much larger.

What is the safest universal image size?

There is no universal value in the checked sources because providers impose different native request, dimension, and token rules. Choose a crop that preserves readable evidence, keep its dimensions and byte count in the fixture record, and test it against every model route you approve. Avoid relying on a provider’s maximum as an operating target.

Does a screenshot replace reproduction steps?

No. A screenshot records one visible state. It does not establish the event sequence, data conditions, viewport change, or code path that produced it. Include deterministic steps and ask the agent to restate them before proposing a fix.

How can I tell whether the model actually used the image?

Use a known-good control containing two harmless, specific visual facts that are absent from the text prompt. Require the answer to report those facts. This does not prove every later diagnosis is correct, but it detects routes that reject, ignore, or fail to understand the image.

Which errors should be retried?

Do not retry malformed requests or unsupported-model errors unchanged. Apply bounded backoff to rate limits, and use only a small retry bound for transient server failures. Keep diagnosis separate from tool execution so a transport retry cannot repeat a repository mutation.

Reader next step

Create two fixtures for the exact model route you plan to use. The happy-path fixture should contain one permanently redacted screenshot, explicit expected and actual behavior, and two visible control facts. The error fixture should use a deliberately unavailable image reference or unsupported media declaration in a non-production environment. Confirm that the first produces grounded observations and that the second produces a classified, non-retried contract failure.

Save the sanitized event fields, add the fixture to the model-change checklist, and rerun it whenever the endpoint or model changes. Once the route passes, turn the real report into a scoped task brief and require a human-reviewed verification step before merge.

Start with CometAPI after the fixture, logging policy, and error bounds are ready.