How good is our probability model, really? A day behind the scenes

11.11. Five processes start. Each builds its own edition of the text model with a fifth of the stones held out, and then fills 698 hidden gaps inside words and as many whole hidden words with the app's own settings. It is a test we have never run before, though we should have: the figures the app shows were measured with a simpler setting than the one the app itself uses. None of the processes writes anything until it is done. That is a flaw in the test script, not in the model, and it will be fixed.

The text model is Vitki's way of reading runes with numbers. You meet it on every stone page under the tab Läsning och luckor (reading and gaps), where it divides the rune sequence into words, gives every word a probability and proposes what may have stood in damaged stretches. You meet it in Forskningsluckor under Luckor i texten, where it fills gaps the editors left open, in the stone report, and in our articles on the Malt stone and on the runes that are missing. It also dates a text by its language, with a figure that says how uncertain the dating is. Everything else in the app – the grooves, the carvers, the statistics – manages without it; but when it comes to what the runes say, it is the one that speaks.

That is what a day looks like when we work on the model, and there have been many such days. The model has been rebuilt in nineteen versions since mid-October: the spelling model, the word order, the Edda's vocabulary, the calibration, the carver's own spelling, names built from name elements, the traces from 3D scans, the tolerance for unusual spellings. Every version has been run through the same tests with stones held out, and several attempts have been discarded because they did not hold – a variant quota last week, a joint dating, an imputation of undated stones. This article is an attempt to show how that works, and whether the model can be made better.

What the model is

It is not a large language model. It is a statistical model built on 3,662 Viking-age runic inscriptions in the Scandinavian Runic-text Data Base, with a spelling model learned from 26,337 pairs of carved rune sequence and normalised word, a vocabulary of 4,567 words plus the Edda's, and a model of which words follow which. It reads a runestone by trying every way of dividing the runes into words and weighing them against each other. Every word gets a probability, and the probability is calibrated: when the model says 80 per cent, it should be right four times in five. That detail is the whole point. We have written about how it read the Malt stone and about the gaps it fills.

Three rules always apply. The model never trains on the stone it is to read. Every claim comes with its probability. And every improvement is tested against a plan locked before we see the result, so that we cannot pick the figures that look best.

10.32, the same morning. Before the question came, we had already let the model read the Rök stone, built without Rök in its training. It took five minutes. Of Rundata's 147 comparable words it delimited 79 correctly and read 50 as the same word. It managed the memorial formula – standa runaR þaR, faði faðiR, kunungaR tvæiR – and failed on every word that exists only on Rök: mogminni, valraufaR, Þioðrik, Siolundi, the ciphers. The model knows only words it has seen, and the whole singularity of the Rök stone is the words no other stone has. But it was honest: it put 95 per cent or more on only 8 words, and 6 of them were right. The dating came out as the tenth century at 78 per cent, against Rundata's ninth, for a simple reason: the model has learned from 19 stones of the ninth century and 440 of the tenth.

How we measure

The figures come from tests where whole stones are held out. We hide runes in texts we know, let the model fill them, and count. Inside a word, where the neighbouring runes help, 76 per cent come out right. When the whole word is gone, 59 per cent. On the 361 gaps where the runes are known from older drawings but lost today – the gaps the app actually meets – 70 per cent. With traces from a 3D scan, where half the rune is visible, whole words rise to 84 per cent.

And the calibration: of the answers the model gives at 95 per cent or more, 95 per cent are right. In a test on 808 inscriptions the calibration error was under one percentage point, and for rare words, where models tend to boast, it said 47 per cent on average and was right 47 per cent of the time. That is why the Rök reading is not a failure. A model that reads wrongly and knows it is useful. One that reads wrongly with confidence is dangerous.

11.52. Step one done. With the app's own settings it came to 77 per cent inside words and 59 per cent whole words: the same as the published figures. But the app is more cautious: 342 confident answers with 15 errors, against 409 with 19 in the simpler test. That was the day's most important news, and it was not exciting, which is as it should be. Step two – whether the model improves by judging candidates on sequences of three words – starts at once.

Compared with others

As far as we know there is no other model that reads Viking-age runic texts with probabilities tested on held-out stones. Rundata and Runor are databases. The large language models can reproduce the interpretation of a known stone because they have read it, but on an unknown text they guess without knowing how sure they are, and no one has tested them on stones they have not seen. When we let an image AI guess the style group of 57 stones, it was right 35 per cent of the time.

The closest comparison is Ithaca, DeepMind's model for Greek inscriptions, published in Nature in 2022. It is a neural network trained on 78,608 inscriptions. It restores damaged text correctly in 62 per cent of cases, dates with a mean error of about 30 years and places an inscription geographically in 71 per cent. Our 59 and 76 per cent are in the same region, on a corpus twenty times smaller, and our dating is worse: the right epoch in 59 per cent of cases, the right or the neighbouring one in 95. The comparison limps – different languages, tasks and measures – but it says roughly where we stand.

The most important thing in the Ithaca study was not the model's figure. Historians alone restored the texts correctly in 25 per cent of cases, the model in 62, and historians with the model's help in 72. That is the working order Vitki is built for: the model proposes and says how sure it is, the runologist decides.

What is missing

The model has three weaknesses we know of. Its vocabulary is Rundata and the Edda, so a word that stands on one stone only is invisible. Its context is two words in a row. And the corpus is small, which no one can change: there are no more Viking-age runic texts. The first weakness can be addressed.

14.00. The next question: can the vocabulary be extended, and with what? We took stock of what exists.

The sources are not equal, and the order matters:

• The medieval runic inscriptions are already in Rundata but unused by the model: 2,748 inscriptions, mostly Norwegian from Bryggen in Bergen, Gotlandic and Danish. It is the same script, the same way of spelling with runes, only a couple of centuries later. It is the closest source there is.
• Heimskringla and the legendary sagas are normalised Old West Norse, some 600,000 words, fetched this spring for our article on the Vikings' world. The language is thirteenth-century Icelandic, but the vocabulary is largely the same.
• The sagas of Icelanders are the largest source, nearly 900,000 words, but in modern Icelandic spelling. They can be used only if the spelling can be brought back to Old Norse form by rules we first test on the normalised texts.
• The Old Swedish law codes, the Västgöta and Uppland laws, are what stands closest to runic Swedish in time and dialect. We do not have them yet.
• The lexica: Zoëga's dictionary of Old Icelandic is free to use, and Lena Peterson's Nordiskt runnamnslexikon has since today been a table of 1,710 entries. It is the lexicon that can teach the model to build names like Vífrøðr, which every interpreter of the Malt stone reads and which the model cannot spell.

The risk is as clear as the gain. More words mean more candidates, and more candidates mean more false hits in short gaps. That is why the plan is locked before anything is counted: each source gets a weight of its own chosen on a single test, and the lexicon is approved only if the model wins more words than it loses without the confident errors growing. Then we read Malt and Rök again and count how many unknown sequences remain. That test decides whether the Malt stone comes a step closer, or not at all.

17.48. Step two done, after six hours. The trigrams win 8 words and lose 6 inside words, but lose more whole words than they win (16 against 22), and they make the model confident about more answers – 430 instead of 342 – with more errors, 17 against 15. The plan required gains in both kinds without more confident errors. It does not hold. Nothing changes in the app, and it goes in the log as a negative result, like the variant quota last week. That is how most days with the model end: a test that shows where it does not get better. Those are the days that make the figures trustworthy.

What we will not do

We will not replace the model with a large language model. Not because they are bad, but because we cannot test them: they have read the interpretations, and we cannot hold a stone back from them. A model whose figures we cannot measure cannot sit in an app that promises to say how sure it is.

Sources

• Scandinavian Runic-text Data Base (Rundata), Department of Scandinavian Languages, Uppsala University. Open Database License.
• Assael, Y., Sommerschield, T., Shillingford, B. et al. 2022. Restoring and attributing ancient texts using deep neural networks. Nature 603, 280–283 (Ithaca).
• Peterson, Lena 2007. Nordiskt runnamnslexikon, 5th ed. Institute for Language and Folklore.
• Zoëga, Geir T. 1910. A Concise Dictionary of Old Icelandic. Oxford.
• heimskringla.no and sagadb.org for the saga texts; Vitki's method description (METHODS.md 8.1), the tests and the locked plans are open on GitHub.