Hidden Text Characters: Detection and Removal

You paste a value into a form, and the comparison fails. A regular expression misses a match. An AI assistant follows an instruction nobody can see. The text looks normal because your screen renders the visible glyphs, not every code point in the underlying string.

Hidden text characters are Unicode characters with no visible width, unusual formatting behavior, or a control effect on text direction and layout. They aren't necessarily malicious. Unicode added the zero-width space, zero-width non-joiner, and zero-width joiner in Unicode 1.1 in 1993, while the word joiner arrived in Unicode 3.2 in 2002. These characters support typography and script shaping, especially in languages that don't use visible spaces between words (Unicode history and invisible characters).

The engineering problem is context. A character that helps a browser render Thai correctly can also defeat an exact-match filter. A directionality mark that resolves mixed Arabic and Latin text can also make source code appear to say something different from what a compiler reads.

Table of Contents

• The Invisible Problem in Plain Sight
• Zero-Width Spaces and Directionality Controls
• Security Risks and AI Prompt Injection
  • Why Unicode tags matter
• How to Detect Hidden Text Characters
  • A focused inspection routine
• Safe Removal and Multilingual Pitfalls
• Building a Resilient Text Sanitization Workflow

The Invisible Problem in Plain Sight

A production bug involving invisible text usually starts with a reasonable assumption: if two strings look identical, they must contain the same characters. That assumption is false. One value may contain an ordinary space, while the other contains a zero-width space. A copied identifier may include a byte-order mark or a word joiner. A prompt may contain instructions embedded in characters that the editor never displays.

The first useful distinction is between rendering and storage. Rendering turns code points into glyphs and layout decisions. Storage preserves the character stream. A zero-width character can disappear during rendering while remaining available to a parser, tokenizer, search routine, database comparison, or language model.

Three groups cause most operational confusion:

• Zero-width spacing and joining characters affect word boundaries, line-break opportunities, and how scripts form connected glyphs. U+200B ZERO WIDTH SPACE has no intrinsic width, but Unicode defines it as a possible word or line break. U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER influence joining behavior.
• Directionality controls influence bidirectional display. U+200E LEFT-TO-RIGHT MARK, U+200F RIGHT-TO-LEFT MARK, and U+061C ARABIC LETTER MARK can resolve ambiguous text containing Arabic, Hebrew, numbers, and Latin characters.
• Formatting and tag characters carry invisible metadata or presentation instructions. U+FEFF ZERO WIDTH NO-BREAK SPACE has historical uses, U+2060 WORD JOINER prevents certain breaks, and the Unicode Tags block, U+E0000 through U+E007F, can carry invisible tag data.

Practical rule: Never debug invisible text through appearance alone. Inspect the raw code points before changing application logic.

The standards history matters because it prevents the wrong fix. These characters weren't created as tricks or loopholes. They address real problems in multilingual publishing, browser layout, editor behavior, and script shaping. Treating every invisible character as garbage will fix one class of bugs while creating another.

Zero-Width Spaces and Directionality Controls

Zero-width characters create trouble because they alter processing without adding visible width. U+200B ZERO WIDTH SPACE can mark a word boundary or line-break opportunity. U+200C and U+200D control joining behavior, and U+2060 WORD JOINER blocks a break without displaying a space. U+FEFF may appear as a byte-order mark at the start of text, but an unexpected occurrence inside content deserves investigation.

The exact code point matters more than the label “invisible Unicode.” A sanitizer that removes U+200B may be appropriate for a copied English identifier, but unsafe for content written in Thai or Khmer. A filter that removes every format character may damage Arabic, Hebrew, or Indic text. Your policy should be based on the input's purpose and language, not on whether a character has visible width.

Bidirectional controls are a separate risk. U+202A through U+202E include embedding, override, and directional-isolate controls. They can change how a mixed-script string is displayed while leaving the logical character order intact. NIST documents CVE-2021-42574, commonly called Trojan Source, as an issue where source code can render differently to human reviewers than to compilers and interpreters (NIST's CVE record for Unicode bidirectional controls).

A practical inspection policy should flag directionality controls in source files, filenames, prompts, and configuration data unless the workflow explicitly requires them. The Unicode Standard describes these marks as legitimate tools for resolving bidirectional text, but the same behavior can visually reorder or conceal content (Unicode Standard, Chapter 23).

For teams comparing text cleanup with statistical watermark handling, keep the problems separate. Unicode sanitation works at the character and encoding layer, while statistical watermarking can exist in token selection even when no unusual code point is present. This distinction is covered in hidden Unicode versus statistical watermarks.

Security Risks and AI Prompt Injection

The assumption that hidden characters are merely copy-paste noise no longer holds. They can form a covert instruction channel inside text that appears harmless to a person but remains available to an AI system or downstream parser.

Cloud Security Alliance research reports that steganographic jailbreaks using invisible characters reached a 92% average attack success rate while evading external safety detectors. The same note reports 64–67% success for encoding-based attacks against open-source LLMs and 42–58% for homoglyph substitution attacks (Cloud Security Alliance research note on Unicode instruction injection). These figures don't mean every hidden character is an attack. They show that the technique is operationally relevant.

An attacker can place concealed instructions in a document, email, web page, agent skill file, or MCP metadata. The user sees an ordinary paragraph. The model receives the full character stream and may interpret the hidden payload as an instruction. External detectors that inspect rendered text can miss what the model sees.

Why Unicode tags matter

The attack surface extends beyond U+200B, U+200C, and U+200D. The Unicode Tags block, U+E0000 through U+E007F, can encode invisible characters that applications handle inconsistently. Bidirectional overrides add another path to visual deception. A zero-width character encoding can even represent each ASCII character as 8 binary digits, using U+200B as 0 and U+200C as 1, creating a machine-readable payload inside visible text (the Reverse CAPTCHA research).

Code review has the same weakness. A reviewer sees one ordering, while a compiler or interpreter consumes another. Security guidance specifically recommends scanning U+202A through U+202E and related format-character ranges because visual inspection isn't a reliable control (Unicode spoofing risks and scanning guidance).

Security boundary: Normalize and inspect text before it reaches an agent, tokenizer, policy filter, or code-review system. Checking only the final rendered output is too late.

AI teams should treat hidden-character handling as part of broader AI support agent security, alongside tool permissions, untrusted-content isolation, and confirmation requirements for destructive actions. Sanitization reduces one input channel. It doesn't replace authorization or prompt-injection defenses.

How to Detect Hidden Text Characters

Start with the raw input, not the UI. Copy the suspicious value into an environment where code points are visible, then inspect each character's hexadecimal value, Unicode name, and position. A browser developer console can expose the string through code-point iteration, while an editor with “render whitespace” or Unicode highlighting can reveal some, but not all, format characters.

For files, use byte-level inspection. hexdump or an equivalent binary viewer helps identify the UTF-8 byte sequence, while a Unicode-aware script tells you which code point those bytes represent. The byte view and code-point view answer different questions. A byte dump shows what was stored, and code-point inspection shows what the application will interpret.

A focused inspection routine

1. Record the original input. Preserve the exact string before normalization so you can reproduce the bug and compare output.
2. Enumerate code points. Print each code point as U+XXXX, along with its Unicode name and character index.
3. Flag high-risk ranges. Check U+200B–U+200D, U+FEFF, U+2060, U+061C, U+200E, U+200F, and U+202A–U+202E. For AI ingestion, also inspect U+E0000–U+E007F.
4. Check malformed encoding. Reject invalid UTF-8 and unexpected surrogate values before processing. A sanitizer shouldn't silently repair ambiguous input.
5. Compare normalized forms. Unicode normalization can expose compatibility differences, but it doesn't automatically remove every control character or hidden payload.

A regular expression can help with known ranges, but don't mistake a regex match for a complete Unicode security policy. Regex engines differ in Unicode support, and a pattern aimed only at zero-width spaces won't catch bidi controls or tag characters.

Configure CI and editor checks to flag unexpected format characters in source files. For user content, log the code-point category and location without storing sensitive text unnecessarily. In high-risk workflows, reject the input and ask for a clean copy rather than altering evidence or instructions.

Safe Removal and Multilingual Pitfalls

Blanket stripping feels attractive because it produces a visibly clean string. It also destroys legitimate text. Zero-width spaces can support word segmentation in Thai, Khmer, Myanmar, and Japanese, where visible spaces don't play the same role as they do in English. Removing them can change tokenization, search behavior, line wrapping, or readability.

The same caution applies to joining behavior. Arabic, Persian, and Urdu can require zero-width joiner or non-joiner behavior for correct letter forms. Hebrew and Arabic text mixed with numbers can depend on direction markers. Devanagari and Bengali may use invisible joining behavior for conjuncts. A cleanup routine that “removes all invisible characters” has no way to distinguish malicious obfuscation from required script formatting unless it understands the document's language and purpose.

Use a policy matrix instead of a universal delete rule:

Input context	Safer default
English identifiers, usernames, filenames	Reject or remove unexpected zero-width and bidi controls
Source code and configuration	Reject format controls unless explicitly approved
AI prompts from external sources	Scan broadly, remove or quarantine hidden payloads, then apply model-level defenses
Thai, Khmer, Myanmar, Lao, Arabic, Hebrew, or Indic publishing	Preserve script-required formatting and validate with language-aware tests
Plain-text exports	Normalize deliberately, then verify that language rendering and search still work

A tool such as an invisible character remover can help locate code points by name, count, and position before you decide what to remove. That inspection step is more valuable than an opaque “clean” button because it leaves the decision with the workflow owner.

The correct question isn't “Is this character invisible?” It's “Is this code point expected in this field, language, and trust boundary?”

Strip high-risk controls from fields that require strict identifiers. Preserve characters required for known scripts. For AI pipelines, treat Unicode Tags and unexpected format controls as untrusted input, but don't assume their removal clears statistical watermark signals carried by ordinary word choices. Those are different layers and need different tests.

Building a Resilient Text Sanitization Workflow

A reliable workflow makes text policy explicit before the system processes content:

1. Define the field. Decide whether the input is a username, source file, multilingual prose, prompt, email, or metadata.
2. Capture the original. Keep a controlled diagnostic representation for incident analysis.
3. Validate encoding. Reject malformed UTF-8, invalid sequences, and unexpected surrogate values.
4. Inspect code points. Scan zero-width marks, bidi controls, formatting characters, and the Unicode Tags block.
5. Apply a field-specific policy. Remove, reject, or preserve characters based on language and use.
6. Normalize safely. Use Unicode normalization where appropriate, then verify that required script shaping remains intact.
7. Separate content from instructions. Treat documents, emails, and retrieved text as untrusted data before sending them to an AI agent.
8. Test the output. Compare code points, visible rendering, search behavior, tokenization, and downstream model handling.

Don't combine Unicode cleanup with watermark claims. Google DeepMind's SynthID-Text changes token selection during generation, and independent research reported that paraphrasing eliminated detection in 98.3% of initially detected SynthID cases (Nature research on SynthID-Text). A character sanitizer can't remove a signal that lives in ordinary word choices. For workflows that need both layers addressed, an AI watermark remover represents a separate rewriting approach rather than a substitute for input security.

Document every exception. Add multilingual regression tests, security test strings, and code-review checks for directionality controls. The strongest pipeline is not the one that deletes the most characters. It's the one that knows which characters belong, which require review, and which should never cross a trust boundary.

---

Simple Unmark cleans pasted text by removing hidden Unicode artifacts and rewriting wording to reduce statistical watermark signals while preserving meaning, facts, numbers, proper nouns, tone, and intent. Visit Simple Unmark to inspect suspicious text and produce a cleaner copy for editorial, development, or AI workflows.