AI

A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space: What the Units of Voynichese Are Not

Researchers Liudmila Rozanova and Alexander Temerev analyzed the Voynich manuscript using various assumptions about its units. They tested three common assumptions: that glyphs are letters, strings between blanks are words, and every blank is a word space. The results showed that none of these assumptions hold true. Instead, the order in Voynichese sits at the edges of tokens and at graded boundaries between them. The study found that glyph regularity is too strong for one-to
Researchers Liudmila Rozanova and Alexander Temerev analyzed the Voynich manuscript using various assumptions about its units. They tested three common assumptions: that glyphs are letters, strings between blanks are words, and every blank is a word space. The results showed that none of these assumptions hold true. Instead, the order in Voynichese sits at the edges of tokens and at graded boundaries between them. The study found that glyph regularity is too strong for one-to-one substitution with any tested plaintext, and resolves onto a scale of recurrent multi-symbol units. Tokens form a plausible vocabulary, but their identity does not predict the next token's identity. Blanks were also found to behave differently than expected. --- Why it matters: This research matters to AI engineers because it highlights the importance of carefully examining assumptions when analyzing complex data. The Voynich manuscript is a challenging example of a mysterious text that has been studied for centuries, and this study provides new insights into its structure and meaning. Source: https://arxiv.org/abs/2608.17096

This article was originally published at: https://arxiv.org/abs/2608.17096