Last reviewed: 2026-09-14

Direct answer

Unicode testing for coding agent patches should start with an explicit text contract, not a handful of familiar strings. Define which normalization form applies at each boundary, what the application counts as a character, which identifier scripts are allowed, how confusable identifiers are handled, and which Unicode version the test data represents. Then exercise both accepted and rejected inputs with deterministic fixtures.

This matters because visually equivalent text can have different underlying sequences. The Unicode Consortium’s Unicode Normalization Forms defines canonical and compatibility equivalence plus NFC, NFD, NFKC, and NFKD. For example, a precomposed C with cedilla and a C followed by a combining cedilla are canonically equivalent. Normalizing both strings to the contract’s chosen canonical form makes a binary comparison meaningful. Compatibility normalization is a separate decision: it can remove distinctions such as circled forms, width variants, or superscripts, so it must not be applied indiscriminately to arbitrary text.

Counting and slicing need their own contract. The Unicode Text Segmentation defines default boundaries for grapheme clusters, words, and sentences. A grapheme cluster is intended to approximate a user-perceived character. Code that truncates a label, moves a cursor, or enforces a visible-length limit should therefore be tested against the intended segmentation boundary rather than assuming every stored unit is a complete character.

Protected identifiers need an additional policy. Unicode Security Mechanisms specifies identifier profiles, mixed-script detection, restriction levels, mixed-number detection, and confusable-detection mechanisms. A confusable result is a policy signal, not proof of malicious intent. The application must state whether to reject, warn, or route the identifier for review.

Use a compact fixture manifest before asking an agent to change implementation code:

case                     contract                          expected result
canonical-pair           NFC comparison                   equal normalized bytes
combining-mark-order     NFC comparison                   one stable result
grapheme-with-mark       extended grapheme boundaries     one segment
compatibility-variant    preserve display distinction      remains distinct
mixed-script-name        protected identifier profile     block or review

Happy-path operator workflow

  1. Locate every changed text boundary: input validation, comparison, lookup, storage, truncation, display, and export.
  2. Write down the normalization, segmentation, and identifier rules for each boundary. Keep policy decisions outside the agent’s discretion.
  3. Record Unicode 17.0.0 as the fixture source version because all three checked specifications identify that version. Verify separately that the runtime and libraries under test use compatible data.
  4. Add canonical-pair, combining-mark, segmentation, compatibility, and identifier-policy fixtures. Keep each fixture focused on one contract rule.
  5. Run the narrow test target first. Confirm accepted inputs normalize or segment exactly once, retain required distinctions, and resolve to the expected stored or compared form.
  6. Run the full suite to detect effects on persistence, indexes, search, validation, and rendering.
  7. Review the patch and its test output together. A passing assertion is useful only when it proves the written contract.

Error-path operator workflow

  1. Stop the merge when a fixture produces the wrong normalized value, boundary count, or identifier-policy result.
  2. Classify the first divergence as normalization choice, segmentation profile, security profile, Unicode-version mismatch, or unrelated regression.
  3. Preserve the failing fixture and expected rule. Do not silently update the expectation to match the patch.
  4. Reduce the input to the smallest sequence that still fails and report code-point counts rather than relying on appearance alone.
  5. Correct either the implementation or the documented contract, then rerun the focused case and the full suite.
  6. If the intended behavior remains unclear, return the decision to the product or security owner instead of letting the agent choose a compatibility or rejection policy.

Logs should expose enough structure to diagnose the result without copying raw submitted text into broad-access output. A sanitized record can use fields like these:

{
  "event": "unicode_contract_test",
  "case_id": "mixed-script-name-01",
  "unicode_version": "17.0.0",
  "normalization_form": "NFC",
  "input_code_point_count": 8,
  "output_code_point_count": 8,
  "grapheme_count": 8,
  "script_set": ["Latin", "Other"],
  "confusable_flag": true,
  "result": "blocked",
  "failure_class": "identifier_policy"
}

Who this is for

This workflow is for engineers reviewing agent-written changes to validators, account names, filenames, search keys, editors, parsers, import pipelines, databases, or user-interface length limits. It is also useful for operators responsible for CI gates when a patch changes string comparison or text storage.

You do not need to be a Unicode specialist. You do need authority to state the application’s behavior. The agent can implement fixtures and transformations, but it should not invent whether compatibility distinctions are meaningful or which writing systems a protected identifier may contain.

Key takeaways

  • Normalization, segmentation, and identifier security are separate contracts. Passing one does not prove the others.
  • Select NFC, NFD, NFKC, or NFKD deliberately. Compatibility forms can erase distinctions that an application needs to preserve.
  • Test user-perceived character boundaries for visible-length limits, cursor movement, truncation, and slicing.
  • Define a documented identifier profile before enabling mixed-script or confusable checks.
  • Pin the Unicode source version in fixtures so a data update becomes an observable change.
  • Keep both accepted and rejected fixtures. The error path is part of the product behavior.
  • Log counts, versions, policy outcomes, and failure classes; avoid treating rendered appearance as diagnostic evidence.

Sources checked

  • Unicode Normalization Forms is Unicode Standard Annex #15, revision 57 for Unicode 17.0.0. It defines canonical and compatibility equivalence, the four normalization forms, normalization behavior, and conformance testing. It also warns that compatibility normalization can remove meaningful distinctions.
  • Unicode Text Segmentation is Unicode Standard Annex #29, revision 47 for Unicode 17.0.0. It specifies default grapheme-cluster, word, and sentence boundaries and explains how an implementation can declare a tailored profile.
  • Unicode Security Mechanisms is Unicode Technical Standard #39, revision 32 for Unicode 17.0.0. It defines identifier security profiles and mechanisms for confusable, mixed-script, restriction-level, and mixed-number detection.

These sources answer different parts of the test contract. Normalization establishes equivalence, segmentation establishes boundaries, and the security standard supplies identifier-oriented detection mechanisms.

Contract details to verify

Before accepting a patch, verify the following details in writing:

  • Comparison domain: State whether the rule applies to a display label, search key, filename, database key, or protected identifier. Different domains may need different behavior.
  • Normalization point: Specify whether normalization occurs at input, before comparison, before persistence, or during a controlled migration. Repeated normalization should produce the same result under the selected form.
  • Normalization form: Choose a canonical form when the goal is canonical equivalence. Use a compatibility form only when folding compatibility distinctions is explicitly intended.
  • Original-text handling: Decide whether the original input must remain available for display or round-trip behavior while a normalized derivative is used for lookup.
  • Segmentation boundary: Name grapheme clusters, words, or sentences rather than using the vague word character. If behavior differs from the default rules, document the profile precisely.
  • Identifier repertoire: List allowed or restricted scripts and characters for the protected namespace. Keep ordinary prose separate from identifier restrictions.
  • Confusable action: Define whether a match blocks creation, emits a warning, or requires review. Include collision behavior against existing identifiers.
  • Version: Record the Unicode version behind fixtures, runtime data, and generated tables. Treat a version change as a reviewed dependency change.
  • Failure response: Specify the stable error class and whether input may be corrected, rejected, or preserved unchanged.
  • Observability: Log the test case, version, selected form, counts, script set, policy outcome, and failure class without relying on raw text.

A small machine-readable contract makes agent scope easier to inspect:

normalization: NFC
comparison_domain: account_name
segmentation: extended_grapheme_default
segmentation_profile: none
identifier_profile: documented_application_profile
confusable_action: review
unicode_version: 17.0.0

The values are examples, not universal defaults. In particular, an application that preserves mathematical or stylistic distinctions may reject compatibility folding even when another application uses it for search.

Failure modes

  • Comparing before normalization: Canonically equivalent sequences reach a uniqueness check in different binary forms, producing duplicate logical names or missed matches.
  • Applying compatibility normalization globally: A patch uses NFKC or NFKD across arbitrary content and removes a distinction the product intended to retain. The Unicode normalization specification explicitly calls for care because compatibility forms do not preserve every visual or semantic distinction.
  • Normalizing only new records: Existing and new keys follow different rules. Lookups, uniqueness checks, or indexes can then disagree unless the change includes a deliberate migration and collision review.
  • Counting stored units as visible characters: Truncation or cursor logic cuts inside a grapheme cluster. The output may still be valid encoded text while violating the application’s user-perceived-character contract.
  • Using an undeclared segmentation customization: Tests pass against a local rule set, but reviewers cannot determine how it differs from the default boundary specification.
  • Treating a confusable flag as a verdict: The patch rejects every flagged string without a documented namespace policy. Detection should feed the application’s reject, warn, or review decision.
  • Ignoring script-policy error cases: The happy path accepts ordinary identifiers, but no fixture proves what happens for mixed scripts, restricted characters, or a collision with an existing confusable identifier.
  • Allowing Unicode-version drift: A runtime or generated data table changes while golden expectations remain unversioned. A later failure then looks nondeterministic even though the underlying data changed.
  • Updating snapshots without examining meaning: An agent refreshes expected output after a failing test and records the regression as the new baseline. Review the first semantic difference before accepting generated fixtures.
  • Logging raw submitted text: Diagnostic output reproduces sensitive or adversarial input unnecessarily. Structured counts and policy results usually provide a safer first diagnostic layer.

FAQ

Is NFC always the correct storage form?

No. NFC is often useful when canonical equivalence is the goal, but the correct choice belongs to the application’s data contract. Some systems preserve original text and store a normalized comparison value separately. The important requirement is that the selected behavior is explicit and tested.

Should an identifier test use NFKC?

Only when the identifier profile deliberately calls for compatibility folding. The normalization specification explains that NFKC and NFKD remove compatibility distinctions and should not be applied blindly to arbitrary text. Test the exact profile and its collision behavior.

Is one code point the same as one character?

Not for every application operation. The segmentation standard defines grapheme clusters to approximate user-perceived characters. Visible-length checks, cursor movement, and truncation should test the boundary model they actually promise.

Should every mixed-script identifier be rejected?

Not automatically. The security standard supplies mixed-script and restriction-level mechanisms, while the application supplies the permitted profile and response. A multilingual product may need a documented allowance that a narrower administrative namespace does not.

Are confusable identifiers canonically equivalent?

Not necessarily. Canonical equivalence and visual confusability solve different problems. Keep normalization assertions separate from confusable-detection assertions so a failure identifies the correct layer.

What should happen when the Unicode version changes?

Update the recorded version, regenerate or refresh the affected fixtures, inspect every changed expectation, and rerun both accepted and rejected cases. Do not accept a bulk golden-file change without reviewing its policy impact.

How broad should the agent’s patch be?

Limit it to the named text boundaries and contract. If the work reveals an undocumented storage or identifier policy, pause implementation and obtain that decision before changing unrelated strings throughout the repository.

Reader next step

Choose one text boundary changed by the patch and write a five-line contract covering domain, normalization, segmentation, identifier policy, and Unicode version. Turn that contract into one accepted fixture and one rejected fixture, then run both before expanding the suite.

For a bounded implementation request, use the process in task briefs that produce reviewable changes . To generate more sequences without losing the explicit contract, extend the fixture set with property-based testing for coding-agent patches . The patch is ready for review when the happy path, error path, sanitized records, and versioned contract all agree.