OpenViSTAI · Agents · Blockchain · Crypto
Submit a project
Menu
OPENVIST RECIPES

Measure UTF-8 byte and scalar widths of text lines with Rust

Rust

Report storage length and Unicode scalar count per nonempty line without confusing either measure with display width.

Rust code excerpt: Measure UTF-8 byte and scalar widths of text lines with Rust. Version 0.1.0. Read the full recipe for code, sources and recorded test scope.

Report storage length and Unicode scalar count per nonempty line without confusing either measure with display width. Work only with supplied input; this does not contact external services or modify input.

Before you start

  • Save as recipe.rs and compile with Rust 1.99.0 using: rustc --edition=2021 recipe.rs -o recipe
  • Run recipe with text supplied on stdin. No CLI arguments are accepted, including help flags.
  • Reads at most 65537 bytes from stdin and rejects input exceeding 65536 bytes. Requires strict UTF-8, rejects a leading UTF-8 BOM, and rejects every CR character.
  • Splits only on LF and discards exactly empty segments. Preserves all other text, order, whitespace, and duplicates without trimming or normalization.
  • Validates at most 100 nonempty lines and at most 256 UTF-8 bytes per retained line before computing scalar counts.
  • Successful execution exits 0, emits exactly one JSON object followed by a newline, and emits no stderr. Empty input or input containing only LF characters produces {"widths":[]}.
  • Invalid arguments, invalid input, and stdin-read failures exit 2 with no stdout and exactly error: invalid input followed by a newline on stderr, subject to the output-failure limitation below. Diagnostics never contain input values.
  • Uses only the standard library, explicit std::io imports, and permitted fully qualified APIs. No Cargo, external crates, filesystem access, networking, subprocesses, or dynamic loading are used.

Build and check

  1. Save the example as recipe.rs.
  2. Compile with Rust 1.99.0 using rustc --edition=2021 recipe.rs -o recipe.
  3. Prepare UTF-8 input.txt following the documented operation-specific conventions.
  4. Run ./recipe < input.txt.
  5. Inspect the JSON result. Exit 2 with exactly error: invalid input followed by a newline on stderr and no stdout identifies rejected input or stdin-read failure. Output-write failures may also exit nonzero, without that diagnostic and potentially after partial stdout; these are outside the input-error guarantee.

Limits to understand

  • The bytes field measures UTF-8 storage bytes excluding LF separators. The scalars field counts Rust .chars(): Unicode scalar values, not grapheme clusters or terminal display columns. Combining marks and components of multi-scalar emoji count separately.
  • Only a BOM at the beginning of the entire input is rejected as a BOM. U+FEFF elsewhere is retained and counted. Unicode line separators other than LF remain part of a line.
  • Stdin may block until EOF or until the 65537-byte read bound is reached. No timeout is imposed.
  • Memory holds up to 65537 input bytes, at most 100 borrowed line references, the serialized JSON, and small temporary numeric strings. Allocator, buffering, and runtime overhead are additional; this is not a strict total-memory cap.
  • Successful output reveals line order, the number of nonempty lines, and each line's byte and scalar lengths. These measurements can be sensitive even though text contents are not printed.
  • Memory-allocation failures and output-write failures are outside the input-error guarantee. A detected successful-output write or flush failure exits 1; stdout may already be partial. Failed stderr writes can prevent delivery of the prescribed error diagnostic.

Run it

rustc --edition=2021 recipe.rs -o recipe
./recipe < input.txt

Example input

café
💻
é

Expected output

{"widths":[{"bytes":5,"scalars":4},{"bytes":4,"scalars":1},{"bytes":3,"scalars":2}]}
INSPECT THE SCRIPT

Inspect, copy and run the documented checks.

This browser preview builds code locally. It does not contact an API or run the code.

Default example ready.

Rust example
use std::io::{self, Read, Write};

fn invalid() -> ! {
    let stderr = io::stderr();
    let mut err = stderr.lock();
    let _ = err.write_all(b"error: invalid input\n");
    let _ = err.flush();
    std::process::exit(2);
}

fn main() {
    if std::env::args_os().skip(1).next().is_some() {
        invalid();
    }

    let mut input = Vec::with_capacity(65_537);
    {
        let stdin = io::stdin();
        let mut reader = stdin.lock().take(65_537);
        if reader.read_to_end(&mut input).is_err() {
            invalid();
        }
    }
    if input.len() > 65_536 {
        invalid();
    }

    let text = match std::str::from_utf8(&input) {
        Ok(text) => text,
        Err(_) => invalid(),
    };
    if text.starts_with('\u{feff}') || text.contains('\r') {
        invalid();
    }

    let mut lines: Vec<&str> = Vec::with_capacity(100);
    for line in text.split('\n') {
        if line.is_empty() {
            continue;
        }
        if line.len() > 256 || lines.len() == 100 {
            invalid();
        }
        lines.push(line);
    }

    // All input validation is complete before constructing successful output.
    let mut output = String::from("{\"widths\":[");
    for (index, line) in lines.iter().enumerate() {
        if index != 0 {
            output.push(',');
        }
        output.push_str("{\"bytes\":");
        output.push_str(&line.len().to_string());
        output.push_str(",\"scalars\":");
        output.push_str(&line.chars().count().to_string());
        output.push('}');
    }
    output.push_str("]}\n");

    let stdout = io::stdout();
    let mut out = stdout.lock();
    if out.write_all(output.as_bytes()).is_err() || out.flush().is_err() {
        std::process::exit(1);
    }
}
What to verify after running
  • The example must produce the documented JSON result.
  • Empty stdin must produce the documented empty result.
  • Reject extra arguments, malformed UTF-8, a leading BOM, oversized input and unreadable stdin with the fixed input-error diagnostic and no stdout.

Request or format preview

Illustrative preview. No provider response, price or balance is fetched.

Change a setting to inspect the preview.

Official sources and related profiles

    Code license: MIT. Keep the license notice with your copy.

    Example version 0.1.0 · Documentation reviewed 2026-10-14.

    Six independent CLI fixtures and 16 supplementary cases are executed in pinned, offline Rust 1.99.0 containers. These checks do not establish exhaustive correctness or behavior on other versions.