The missing runes – work in progress on Vitki's probability model

Enlarge ⤢

Hardly any runestone has survived a thousand years undamaged. Stones have cracked, flakes have come off and weathering has eaten into the carving. Of the 2,633 Viking-age runestones in the Scandinavian Runic-text Data Base, 1,087 have at least one place where one to four runes have been lost.

When runologists publish a stone they often fill such gaps. The restored runes are put in square brackets, as in GæiRf[as]tr. The brackets say that the runes cannot be seen on the stone. They do not say how sure the editor was, or which other possibilities were weighed.

In Vitki we are building a model that does the same work in a different way. It proposes what stood in the gap, ranks the proposals and states how sure it is. The percentage is tested against gaps where the answer is known, so that 90 per cent should mean that the proposal is right about nine times out of ten. Here we report how far we have come, and we account for every test, so that anyone who wishes can examine how the model is built and tested before it is released.

A puzzle with too few pieces

The younger futhark has only 16 runes, but the language had many more sounds. The same rune therefore had to stand for several sounds: ᚴ could be k, g or ng, and ᛁ could be i, e, æ or j. Nor did the carvers always put dividers between the words. A sequence of runes can thus be read in several ways even when all the runes survive. When some are missing, the possibilities multiply.

Enlarge ⤢

How the model reasons

Take Sö 45 as an example. According to Södermanlands runinskrifter the stone is broken into three pieces. The first name begins with the runes kai on the lowest piece and continues with Rfastr on the upper left one. There the stave of the a is "indistinct because of flint", and of the s only "the upper stave and the starting point of the middle stroke" remain; "the rest of the rune is destroyed by a deep pit". The editors read GæiRf[as]tr. The model is shown the runes kaiRf??tr, where the two question marks are the lost runes.

Enlarge ⤢

First the model tries which words can be spelled with the surviving runes plus two more. Then each proposal is weighed by how well it fits where it stands. It is the same principle as when a phone suggests the next word: some words tend to follow certain others. Here the word stands first in the text, followed by another name, Æistfari, and the verb ræisþu, 'raised'. The name GæiRfastr occurs on several other stones, for example Sö 227, Sm 111 and U 1144, so the model knows it. We did not enter any traces of the runes here.

The answer is GæiRfastr with 95 per cent and GæiRfriðr with 1. The first proposal is the same name that the editors wrote in. The model has not seen the text of Sö 45 itself, because a stone is always read by a version of the model that was trained without it.

The building blocks

The model is not one big machine but several parts, each of which can be tested on its own. Several of them are well-tried methods from language technology. The model learns the language from every kind of Viking-age inscription, not only the runestones, since it is the same on an amulet as on a stone.

• Spelling. The model has learned how sounds were written in runes from 27,218 pairs of carved words and their normalized form, from 2,877 inscriptions. Spelling is estimated for each period and region, so that a stone from the 900s is not read like one from the 1000s.
• Word order. How often one word follows another is counted with a method from speech recognition, Kneser–Ney smoothing (Kneser & Ney 1995). It also gives rare word pairs a reasonable chance.
• Word classes. Words that occur in similar contexts are grouped automatically. The genitives hans, hennaR, sina and þæiRa end up in the same class without anyone having told the model what a genitive is. This helps it with rare words. The classes are computed with scikit-learn.
• Names. Many names occur on one stone only. The model therefore builds unknown names from first elements, second elements and endings, spelled as the carvers spelled them. The first element Ragn-, for example, is carved as rahn 17 times in Rundata.
• Neighbouring stones. Names on stones within ten kilometres weigh a little more, since the same families often raised several stones in the same district.
• Dividers. Carvers differed. Most put dividers between nearly all words, some between none at all. The model therefore estimates for each stone how often that carver marked a word boundary, and lets this decide how readily it places a boundary itself where the carver put no sign.
• Traces. If a remnant of a rune is visible in the scan or in the edition's description, it can be entered. A trace counts as likely but not certain, so that a misread remnant does not shut out the right answer.
• Calibration. The model's own percentages were too high, especially for names. They are therefore adjusted with a logistic model, the same principle as in Platt (1999), separately for formula words, known names, names the model has composed itself and other words. All the proposals in a gap are calibrated together, and "none of the proposals" gets a probability of its own; the first choice gets a deduction of its own, since it was chosen precisely because its probability was the highest. The calibration is estimated on three kinds of gaps with a known answer, described below.
• No cheating. The material is split into five parts by signum. A stone is always read by a model trained on the other four, so that the editors' restoration never leaks in. When a single stone is to be analysed, the model can also be trained on everything except that stone.

Six tests with a known answer

A proposal is only worth something if you know how often it is right. We therefore test the model in six ways, always on stones it was not trained on. Each test answers a question of its own:

• The reading test: does the model read whole texts correctly when it must decide for itself where the words begin?
• The masking test: can it fill runes that we hide in texts with a known answer?
• The hard masking test: can it do the same when the answer is never a formula word?
• The editors' gaps: does it arrive at the same word as the editors where they have filled a gap?
• The documented gaps: is it right where runes have really been lost but are known from older drawings?
• The dating test: can it tell when a text was written, from the language alone?

The reading test. The model reads 808 inscriptions where the answer is known, 10,669 words in all, without the editors' word boundaries. It chooses the same word as the editors in 84.3 per cent of cases and divides the words correctly in 94.6 per cent. When the first version was ready in October the figures were 81 and 92 per cent. Three percentage points may sound little, but it is about 300 more words read correctly, and about half of the gain came when the model learned to build names. The 808 inscriptions are 698 runestones, 84 other stones and 26 loose objects. On the runestones the model chooses the same word as the editors in 85.2 per cent of cases and divides the words correctly in 95.4 per cent. The loose objects are hardest, at 56 and 71 per cent.

The masking test. Here we hide runes in texts where the answer is known and let the model fill them in. If a few runes inside a word on a runestone are missing, it gives the right word in 76 per cent of cases. If a whole word is missing, the figure is 59 per cent, since then no runes are left to go on, only the words around it. In a simulated test on runestones, where half of each lost rune is visible, the share of whole words that are right rises to 84 per cent, and it is still 77 per cent when one trace in five is misread. That test is simulated with clean rune shapes and shows what the traces are worth at best.

The hard masking test. Formula words such as stone, raised and in memory of are easy to guess. In this test we therefore hide only words outside the 50 most common. The model then gives the right word in 47 per cent of cases when a few runes are missing inside the word, and in 9 per cent when the whole word is missing. The test shows how much the formulas help, and how hard it is to guess an unusual word without traces. What matters is that the model knows it, and it does so almost all the way: when it says at least 95 per cent it is right in 96 cases out of a hundred here too, measured by the same rule as in the editors' gaps – counting every normalization variant as wrong instead, such as hakun against hakon, gives 92 out of a hundred. The test did teach us one reservation: in the 80–95 per cent band the model is somewhat too sure about unusual words. There it claims 89 per cent on average but is right in 84 cases out of a hundred. Anyone filling a gap whose answer cannot be a formula word should read percentages in that band with that margin.

The editors' gaps. We test the model on 565 gaps on 399 runestones that the editors have filled. The model makes a proposal in 544 of them, and 85 per cent of the proposals are the same word the editors chose. Formula words are right in 96 per cent, names in 75 and other words in 65 per cent. Even so, this test is easier than reality. Editors prefer to fill gaps where the answer is given and leave the hard ones open, so the gaps they have filled are often the simple ones.

The documented gaps. A better test is found in Rundata itself. Runes within square brackets are lost on the stone but known from older drawings. They fell off for physical reasons, not because they were easy to guess, and yet the answer is documented. We turn each such run of one to four runes inside a word into a gap. On 361 such gaps on runestones the model gives the right word in 69 per cent of cases. That is lower than in the other tests, as expected: here the gaps lie wherever the damage happened to fall, and the rest of the stone is often damaged too. It is the test that most resembles the gaps the model is meant for.

The dating test. The model can also estimate when a text was written, from its spelling and vocabulary. It reads the text with each of four periods, from the 800s to the 1100s, and chooses the one in which the text is most probable. On 1,670 dated runestones it chooses the right period in 59 per cent of cases. Always guessing the most common period gives 47 per cent. It is at most one period off in 95 per cent. That is not enough to replace the style groups, but it can give a first date to the stones that have none.

When it says 90 per cent, it means 90 per cent

What matters most is not that the model is right as often as possible, but that its percentages can be trusted. A weather forecast that says 70 per cent chance of rain should see rain on about seven days out of ten. We hold the model to the same standard.

Here the documented gaps were decisive. With a calibration estimated on the editors' gaps alone, the percentages held there, but not on the documented gaps. When the model said about 62 per cent, it was right in only 33 per cent of cases. It was too sure precisely where the answer was uncertain. The calibration is now estimated on three kinds of gaps together: the editors' gaps, the masked words and the documented gaps. The app uses the level of the documented gaps, since they resemble real damage.

The calibration does more than adjust figures: the model now chooses the word with the highest calibrated percentage, not necessarily the language model's own first choice. A word's percentage can never overtake another word of the same class, but a formula word can overtake a composed name, since composed names are more often wrong. In cross-validated tests this choice raised the share of right words from 69.1 to 69.9 per cent on the documented gaps and from 85.5 to 85.7 on the editors' gaps.

The bars in each pair should be equally long, and in three of the four bands they are. In the middle range, where the gaps are few, the model is still somewhat too sure. The average distance between what the model says and how often it is right, the measure called calibration error, fell on the documented gaps from 14.4 to 3.4 percentage points. When the model says at least 95 per cent it is right in 99 per cent of cases. That applies to 165 of the documented gaps.

We also tested the calibration where it had not seen the answers. In the hard masking test, which was not used when it was estimated, the calibration error for gaps inside a word fell from 17 to 4.8 percentage points. In the editors' gaps the model arrives at the same word as the editors in 99.6 per cent of the cases where it says at least 95 per cent. And in the reading test the model's own percentages for the words differed on average by 8.8 percentage points from how often it was actually right; after calibration the difference is 0.6 percentage points.

Checked in detail

A tool meant for research must be trustworthy also in the parts one does not see. We have therefore checked the model's calculations against independent programs and against itself:

• Word order is computed exactly as in NLTK, a widely used library for language technology. The largest difference over 3,000 word pairs is 10⁻¹⁶, nothing but rounding.
• The calibration gives the same results as scikit-learn. We also tried another method, isotonic calibration, but it gave worse results and is not used.
• The faster calculation. For each proposal in a gap the model used to read the whole text again. Since a word never runs across a divider, it is enough to read the group around the gap again. The new calculation gives the same answers as the old one, with differences of at most 10⁻¹⁴, and is 4.7 times faster.
• Stones with many gaps. When the proposals for the gaps give more than 256 combinations, the model searches for the best one gap by gap. On U 512, with nine gaps, this gave the same answer as trying every combination.
• Automatic tests. 32 tests run at every change to the code.
• Everything is open. The code, the tests and the figures are on GitHub, and the method is described in METHODS.md, section 8.1.

A blind test, locked in advance

Every figure above comes from tests we built and read off ourselves as the work went along. No test stone is ever part of its own model, but whoever builds a tool can still shape it, without noticing, after their test data. We therefore finished with a blind test under a pre-registered protocol. Population, drawing rule, settings, measures and pass limits were written down and locked in the version history before a single gap was filled, and the draw's random seed is the registration's own checksum – so the selection cannot have been picked after the fact. From 197 runestones with documented gaps, where every stone that had ever appeared in our articles, scans or case studies was removed in advance, 14 stones with 21 gaps were drawn. The locked requirement: between 10 and 18 right out of 21 would be consistent with the benchmark, the test runs once, and the result is reported whatever it shows.

The model chose the editors' word in 16 of 21 gaps, 76 per cent – in the middle of the locked interval. All 13 gaps where it claimed at least 95 per cent came out right. And four of the five misses carried the model's own warning, with claims below 45 per cent: spaka where the editors read the compound malspaka, guesses at 14 per cent where the answer is the place name svanaby and the rare name liðsvaldr. The one half-confident miss deserves quoting as it stands: in Sm 121's ---ur the model proposed faður, "father", at 77 per cent – the editors read moðurbroður, "mother's brother". The protocol, the draw and every gap's outcome sit untouched in the version history on GitHub.

Sö 206: a single rune decides

Sö 206 stands at Överselö church. Rundata translates the inscription: "Here shall these stones stand, reddened with runes: Guðlaug raised (them) in memory of her sons; and Hjalmlaug in memory of her brothers." Two places are damaged.

Enlarge ⤢

In the first word, ru?niR, the rune is according to the edition "entirely gone, because a flake of the stone surface has fallen off". The model has two main proposals: Hrungnir, the name of a giant in the Edda, and roðnir, 'reddened', which is the editors' reading. The language model itself prefers the giant, with 60 per cent against 27. But in tests with a known answer such first choices often prove wrong, and after calibration 26 per cent remain for Hrungnir and 18 for roðnir – the rest is left open. The language alone thus does not settle the matter.

The difference between the proposals is a single rune. roðnir is written with þ and Hrungnir with k, rukniR. Had the scan shown a remnant of one or the other, the question would be nearly settled.

Enlarge ⤢

On Sö 206 there is no such remnant, since the flake is gone. The editors instead settled the matter by meaning. Elias Wessén writes in the edition that it is "not possible, by inserting any rune other than þ, to obtain a word that makes sense in the context". He mentions that Hallström had objected to the reading in Fornvännen in 1931, but dismisses the objections as "a scepticism carried too far". The model works with words and word order, not with meaning. What it adds is to show exactly what the decision rests on: a single rune, and the meaning.

In the second damaged word, k???auk, only the lower parts of the staves of three runes remain according to the edition, "but the reading is certain because of Sö 208". On that stone, at the same church, Guðlaug and Hjalmlaug stand together. Sö 208 is in the same test part as Sö 206, and so the ordinary model has not seen it. It gives no proposal more than 11 per cent: Guðlaug 11, Gíslaug 8 and kumbl 6 per cent. In other words, it says that it does not know. When a single stone is to be analysed for real, the model is therefore trained on everything except that stone, so that it sees the sister stones but never the stone's own text.

Every runestone with gaps

We let the model read all 1,087 Viking-age runestones that have gaps of one to four runes. 58 of them have fewer than four legible runes and cannot be read. The other 1,029, with 1,910 gaps in all, are read in five parts at once on an ordinary computer, and the run takes about 25 minutes. The first run took considerably longer; the faster calculation above has shortened it. At the same time the model read 247 inscriptions on other stones and loose objects, with 476 gaps, but we do not count them here.

In 18 per cent of the gaps the model is at least 95 per cent sure. In 56 per cent it is under 50 per cent, and there it gives a list of several proposals instead of an answer.

Most interesting is the comparison with the editors. Among the gaps on Swedish runestones that the editors have filled, the model says at least 95 per cent in 267 of 555. Among those the editors left open, it does so in none of the 70. Where the runologists could not settle the matter, the model thus never claims to know. That is how we want it to work.

In 108 gaps the words stand without dividers, and the proposal changes if the words are divided as the editors divide them. There the model shows both proposals.

Why it matters

• The certainty is visible. A square bracket shows that something is restored. The model also shows how certain it is and which other possibilities exist, with percentages tested against the answer.
• The same yardstick for every stone. All 1,910 gaps on the runestones are read by the same rules. When the model improves, or a new reading appears, everything can be recalculated.
• It shows where to look. On Sö 206 everything depends on whether a remnant of þ or k survives. With 3D scans such places can be examined to half a millimetre.
• Everything is open. The code, the tests and the figures are on GitHub, so that anyone can test them again.

Similar tools have been built for Greek inscriptions, where the model Ithaca proposes lost letters (Assael et al. 2022). Vitki's model is built for the runes' own difficulties: the ambiguous spelling, the many names and the traces in the scans. It does not replace the runologist. It gives the runologist a second opinion that can be counted on.

Before the model is released

Work remains before the model comes out in the app:

• Traces from the scans. Today the model only counts traces that someone has entered by hand, but the reader that will take them straight from the 3D scan is well under way. The branches, which tell the runes apart, cannot be trusted in the thinned groove lines – they are now read from the depth image itself instead, weighted by how deeply they are carved. In a first test on 17 scanned stones, where whole words were masked and filled by a model that had never seen the stone, the share right rose from 53.5 to 59.9 per cent when the scanner's traces were allowed to help – and with the traces randomly reshuffled among the stone's runes the whole gain vanished. So it is the stone that is being read, not an exercise in numbers. At badly weathered patches the traces can mislead, so the panel shows what the scanner sees – the branches with their carving depth and the possible runes – and the traces only count once the user has confirmed them. A rune with more deep branches than any rune holds, a possible bind rune, proposes nothing at all.
• Names. Names are right in 75 per cent of the editors' gaps, against 96 for formula words. Most errors in the documented gaps concern names. The latest version took a step right here: the name builder's proposals used to be cut from the candidate list before they could be weighed, and when they were given a place of their own (textmodel-18) the model won eight masked gaps – among them the names Guðfastr and Þormóðr – without losing any.
• More texts from Denmark and Norway, so that the yardstick is equally fair to every country.
• Inflection. The model does not know case and agreement. That concerns only about one word in a hundred in the reading test, however, mostly names in the wrong case. Most of what looks like a wrong form is instead different normalizations in Rundata, such as sin and sinn.
• The answer key. The editors' restoration is itself a judgement. The masking test and the documented gaps have a firmer answer.

When this is in place, the model will come out in the panel Reading and gaps on every inscription's page in Vitki.

The other model: how language-like is a sequence of runes?

Vitki also has a smaller model that measures how much a sequence of runes resembles language. We used it in the article Interpret, he who can to divide the uninterpreted inscriptions into futhark rows, broken language and sequences that do not resemble language. It too has been improved by tests that first went wrong. Short, broken pieces looked incomprehensible just because they were short, so each inscription is now compared with real texts cut to exactly the same lengths. Danish and Norwegian fragments were at first measured against a mostly Swedish yardstick; now they are compared with texts from the same country. The futhark row on DR EM85;470C, fuþrkhainsl, was first judged "not language" because it departs from the usual order; now futhark rows with deviations are recognised too. And Rundata's alternative readings are no longer counted as part of the text. Every change is in the code.

How we did it

• Data: the Scandinavian Runic-text Data Base (Rundata). The model learns from all 3,662 Viking-age inscriptions with transliteration and normalization and from the vocabulary of the Edda. The tests and the run are reported for the 2,633 runestones, fragments and lost stones included. Other stones and loose objects are named separately (utdata/modellartikel/runstenar.py).
• The reading test: scripts/runlm_calibrate.py, on 808 inscriptions where Rundata's words cover the carved text exactly. The material is split into five parts by signum.
• The masking tests: scripts/runlm_restore_test.py, 1396 hidden gaps on runestones, and with --hard only words outside the 50 most common. The traces in the simulated test are drawn with the typeface Apple Symbols.
• The editors' gaps: scripts/runlm_gaps.py. The same word in another inflection or normalization counts as the same word.
• The documented gaps: scripts/runlm_documented_test.py. Runes within square brackets are turned into gaps, and the answer is the documented runes and Rundata's normalization.
• The calibration: scripts/runlm_gap_calibrate.py, with all three kinds of gaps and cross-validation with whole stones held out.
• The dating test: scripts/runlm_dating.py, on 1,670 dated runestones.
• The run over every stone: scripts/textmodel_batch.py. The comparison with the editors is made in utdata/modellartikel/batch_status.py.
• Sö 206 with traces: the model's answer when only þ, or only k, is allowed in the gap (src/textmodel.py, with traces counted as likely).
• The images: Sö 45 in raking light, computed in Vitki. Sö 206 exported at high resolution from Vitki. The illustrations are drawn with utdata/modellartikel/figurer.py.
• The whole method: METHODS.md, section 8.1.

About the scans

Sö 45 and Sö 206 were scanned by Laila Kitzler Åhfeldt of the Swedish National Heritage Board within the research project Runristandets dynamik (2009–2014), with a scanner that measures the surface with structured light. Only the carved surfaces were scanned. In 2024 she published the scans of 35 runestones from Södermanland openly on Zenodo, under the licence CC BY 4.0. We use the files So 45_1_4 thin_closed holes.stl and So 206_thin03_closed holes.stl as published. The image of Sö 45 is computed in Vitki, and the image of Sö 206 was exported at high resolution from Vitki by Viktor Kvant. Her own studies of the scans are Kitzler Åhfeldt 2015 and 2017. We are grateful that the dataset has been made freely available.

Sources

• Kitzler Åhfeldt, L. 2024. 3D-data Runstenar i Södermanland [Dataset]. Swedish National Heritage Board. doi:10.5281/zenodo.11951255. CC BY 4.0.
• Kitzler Åhfeldt, L. 2015. Rune Carvers and Local Mobility. Viking and Medieval Scandinavia 11, pp. 143–182.
• Kitzler Åhfeldt, L. 2017. Runristare och mobilitet i Södermanland. Report. Stockholm: Swedish National Heritage Board.
• The Scandinavian Runic-text Data Base (Rundata), Department of Scandinavian Languages, Uppsala University. Open Database License.
• Brate, E. & Wessén, E. 1924–36. Södermanlands runinskrifter (Sveriges runinskrifter 3). Stockholm: KVHAA. Digitized by the Swedish National Heritage Board. Sö 45 on pp. 34 f., Sö 206 and Sö 208 on pp. 181 ff.
• Kneser, R. & Ney, H. 1995. Improved backing-off for M-gram language modeling. Proceedings of the 1995 International Conference on Acoustics, Speech, and Signal Processing 1, pp. 181–184.
• Bird, S., Klein, E. & Loper, E. 2009. Natural Language Processing with Python. Sebastopol: O'Reilly. (NLTK)
• Pedregosa, F. et al. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12, pp. 2825–2830.
• Platt, J. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Smola, A. et al. (eds), Advances in Large Margin Classifiers. Cambridge, MA: MIT Press, pp. 61–74.
• Assael, Y., Sommerschield, T., Shillingford, B. et al. 2022. Restoring and attributing ancient texts using deep neural networks. Nature 603, pp. 280–283.