OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
OPENVIST RECIPES

Locate Unicode replacement characters in text lines with Rust

Rust

Find already-present U+FFFD markers without mistaking valid UTF-8 decoding for evidence of original data quality.

Rust code excerpt: Locate Unicode replacement characters in text lines with Rust. Version 0.1.0. Read the full recipe for code, sources and recorded test scope.

Find already-present U+FFFD markers without mistaking valid UTF-8 decoding for evidence of original data quality. Work only with supplied input; this does not contact external services or modify input.

Before you start

  • Save as recipe.rs and compile with Rust 1.99.0 using: rustc --edition=2021 recipe.rs -o recipe
  • Run ./recipe with input supplied through stdin. No command-line arguments are accepted, including help flags.
  • Reads at most 65537 bytes and rejects input exceeding 65536 bytes. Requires strict UTF-8, rejects a leading UTF-8 BOM, and rejects every CR character, including CRLF.
  • Splits only at LF and discards exactly empty segments. Retains whitespace-only segments without trimming or normalization. Before classification, validates at most 100 retained lines and at most 256 UTF-8 bytes per retained line.
  • Successful output is exactly one JSON object followed by a newline, with the form {"positions":[1,3]}. Positions are ascending, one-based indices among retained nonempty lines. Each matching line appears once, regardless of its number of U+FFFD characters. Empty input or no matching lines produces {"positions":[]}.
  • All input validation finishes before successful stdout is written. Success exits 0 without stderr. Invalid arguments, invalid input, or stdin-read failures exit 2 with no stdout and exactly error: invalid input followed by a newline on stderr, subject to the failure limitations below.

Build and check

  1. Save the example as recipe.rs.
  2. Compile with Rust 1.99.0 using rustc --edition=2021 recipe.rs -o recipe.
  3. Prepare UTF-8 input.txt following the documented operation-specific conventions.
  4. Run ./recipe < input.txt.
  5. Inspect the JSON result. Exit 2 with exactly error: invalid input followed by a newline on stderr and no stdout identifies rejected input or stdin-read failure. Output-write failures may also exit nonzero, without that diagnostic and potentially after partial stdout; these are outside the input-error guarantee.

Limits to understand

  • Memory use is bounded independently of input length: a 65537-byte stack buffer, a small output string and temporary index strings, plus standard-library I/O buffers and runtime overhead. Lines are borrowed from the input buffer rather than copied.
  • Stdin may block while waiting for data or EOF. Input of exactly 65536 bytes requires a subsequent read to establish EOF; observing a 65537th byte immediately establishes oversize input.
  • Only already-present U+FFFD markers are located. Valid UTF-8 decoding neither establishes original data quality nor reveals whether a marker was intentional or introduced earlier. Malformed UTF-8 is rejected, never decoded with replacement.
  • Successful output reveals matching retained-line positions and may itself be sensitive. Error diagnostics do not disclose input values.
  • Memory-allocation failures and output-write failures, including stderr failures, are outside the input-error guarantee. A stdout-write failure attempts exit status 2 and may leave partial output.
  • A replacement marker may have been intentionally supplied. This does not recover or diagnose the original encoding.

Run it

rustc --edition=2021 recipe.rs -o recipe
./recipe < input.txt

Example input

ok
�
x�y

Expected output

{"positions":[2,3]}
INSPECT THE SCRIPT

Inspect, copy and run the documented checks.

This browser preview builds code locally. It does not contact an API or run the code.

Default example ready.

Rust example
use std::io::{self, ErrorKind, Read, Write};

fn prepare_output() -> Result<String, ()> {
    if std::env::args_os().skip(1).next().is_some() {
        return Err(());
    }

    // Read at most 65537 bytes, using the extra byte to detect oversize input.
    let mut buffer = [0u8; 65537];
    let mut used = 0usize;
    {
        let stdin = io::stdin();
        let mut input = stdin.lock();
        loop {
            if used == buffer.len() {
                return Err(());
            }
            match input.read(&mut buffer[used..]) {
                Ok(0) => break,
                Ok(n) => used += n,
                Err(error) if error.kind() == ErrorKind::Interrupted => continue,
                Err(_) => return Err(()),
            }
        }
    }

    let text = std::str::from_utf8(&buffer[..used]).map_err(|_| ())?;
    if text.starts_with('\u{FEFF}') || text.contains('\r') {
        return Err(());
    }

    // Complete all line validation before inspecting replacement markers.
    let mut count = 0usize;
    for line in text.split('\n') {
        if line.is_empty() {
            continue;
        }
        count += 1;
        if count > 100 || line.len() > 256 {
            return Err(());
        }
    }

    let mut output = String::from("{\"positions\":[");
    let mut retained_index = 0usize;
    let mut first = true;
    for line in text.split('\n') {
        if line.is_empty() {
            continue;
        }
        retained_index += 1;
        if line.contains('\u{FFFD}') {
            if !first {
                output.push(',');
            }
            first = false;
            output.push_str(&retained_index.to_string());
        }
    }
    output.push_str("]}\n");
    Ok(output)
}

fn main() {
    match prepare_output() {
        Ok(output) => {
            let stdout = io::stdout();
            if stdout.lock().write_all(output.as_bytes()).is_err() {
                std::process::exit(2);
            }
        }
        Err(()) => {
            let stderr = io::stderr();
            let _ = stderr.lock().write_all(b"error: invalid input\n");
            std::process::exit(2);
        }
    }
}
What to verify after running
  • The example must produce the documented JSON result.
  • Empty stdin must produce the documented empty result.
  • Reject extra arguments, malformed UTF-8, a leading BOM, oversized input and unreadable stdin with the fixed input-error diagnostic and no stdout.

Request or format preview

Illustrative preview. No provider response, price or balance is fetched.

Change a setting to inspect the preview.

Official sources and related profiles

    Code license: MIT. Keep the license notice with your copy.

    Example version 0.1.0 · Documentation reviewed 2026-10-14.

    Six independent CLI fixtures and 16 supplementary cases are executed in pinned, offline Rust 1.99.0 containers. These checks do not establish exhaustive correctness or behavior on other versions.

    Related browser tools

    Inspect related data locally in your browser. These tools have their own input rules and limits; they do not execute this recipe.

    Base64 Encoder & Decoder