# Tokenese (archived)

This repository was archived on 2026-07-25. Tokenese was an attempt at a token-native interlingua for LLM-to-LLM communication: plain text on the wire, a tokenizer-audited lexicon, a designed grammar. The designed-interlingua premise was measured and falsified: counted on the certified tokenizers, the flagship Tokenese example was 1.3x larger than the English prose it replaced, and the fair baseline (terse English) matched or beat every designed form. The full reasoning, the measurement tables, the 70-year lineage of the idea, and the pre-registered revival conditions are at https://tokenese.org/post-mortem.html. Everything below is a historical record of the spec as it stood at the final release (grammar v0.3, tools v0.3.9, 2026-06-23), not a live project.

## Historical record: what Tokenese was

Tokenese was an open specification, not a product. It defined a wire grammar, a closed function vocabulary of sigils and operators, an in-band symbol table, a capability handshake, and a self-repair protocol. The bet, later falsified by the project's own audit, was that natural language sits far enough from the efficient frontier that a designed language could be both more compressed and more accurate at once.

## Historical record: key concepts

- Token-space only: plain text on the wire, each party tokenizes independently. No embeddings, KV-cache sharing, or latent channels (security and cross-vendor portability required this).
- Tokenizer-audited lexicon: function-vocabulary symbols cost 1 token worst case (bare and space-prefixed) in every certified tokenizer; content words admitted on tokens-per-meaning advantage. The final audit covered seven columns: OpenAI o200k_base, Anthropic count-tokens (claude-haiku-4-5), Gemini (gemini-2.5-flash, API-gated), Qwen2.5-7B, DeepSeek-V3, Llama 3 8B, and Gemma 4 E4B. Six are offline-reproducible; Gemini is API-gated.
- Compression from structure, not glyphs: savings were to come from eliminating function-word syntax, anaphora, and repeated referents. Measurement showed the designed structure cost more than it saved.
- Report-only framesets: typed slot signatures for common ops as structural telemetry only.
- Self-repairing: the `??` misparse signal and a plain-English escape hatch were mandatory.
- Human-auditable: a competent human with the one-page audit card can follow any conforming transcript.

## Final status

Archived 2026-07-25. Final release v0.3.9 (2026-06-23): seven-column tokenizer audit complete; reference translator, deterministic checker, CLI, MCP server, compression/hypothesis evals, N2 static package report, and report-only frameset validator preserved in the repo (163 tests passing at archive). The validating live A/B experiment (N2) was designed but never run: the compression claim was falsified by direct measurement first, and demand evidence (zero external adopters over six weeks public) answered the remaining question. What survives: the reproducible cross-tokenizer worst-case cost audit methodology, and the published negative result itself.

## Key files (historical)

- Post-mortem (authoritative): https://tokenese.org/post-mortem.html
- Specification (foundational wire grammar): https://github.com/snapsynapse/tokenese/blob/main/spec.md
- Grammar v0.3 (final): https://github.com/snapsynapse/tokenese/blob/main/GRAMMAR-v0.3.md
- Design invariants and intent: https://github.com/snapsynapse/tokenese/blob/main/INTENT.md
- Conformance claim classes: https://github.com/snapsynapse/tokenese/blob/main/CONFORMANCE.md
- Reference toolchain (translator, checker, CLI, MCP server): https://github.com/snapsynapse/tokenese/tree/main/tools/translator
- Changelog: https://github.com/snapsynapse/tokenese/blob/main/CHANGELOG.md

## Links

- Canonical home: https://tokenese.org/
- Repository (archived): https://github.com/snapsynapse/tokenese
- Sibling specs: https://turnfile.work/ , https://gracefulboundaries.dev/ , https://hardguard25.com/

## License

Specification text: CC BY 4.0. Code (audit scripts): MIT.