Statistical terms in the reports

The research reports use a small number of statistical terms again and again. Here they are explained in plain words: what the term is and how to read the number. Each report also states its main result in plain words.

p-value: The probability of getting a pattern at least as strong if there were in reality no association, only the variation of chance.; How to read it: A small p means the pattern is hard to explain as chance. Below 0.05 the result is called significant. p says nothing about how large or important the association is; the effect measure, for example the odds ratio, does that.
Significant: That the p-value lies below a limit fixed in advance, in the reports 0.05.; How to read it: Significant means the pattern is probably not chance, not that it is large. Not significant means the material is not enough to decide, not that there is no difference.
Odds ratio: How much the odds of a trait differ between two groups, or change for each step in time. The odds are the share with the trait divided by the share without it.; How to read it: 1 means no difference, 2 twice as high odds and 0.5 halved odds. When the trait is rare the odds ratio is close to the ratio of the shares. An odds ratio of 0.55 per 50 years means the odds roughly halve in fifty years.
95 % interval: The range where the true value probably lies, given that the material is limited.; How to read it: A wide interval means great uncertainty. If 1 lies outside the interval for an odds ratio, or 0 outside the interval for a difference, the difference is significant.
Logistic regression: A model for a yes or no, for example whether a stone has a certain trait, in which several explanations are tested at once.; How to read it: It shows how much one explanation matters when the others are held constant, and gives the result as an odds ratio.
Held constant, adjusted for: That the comparison is made between stones that are alike in what is held constant, for example the same province and the same period.; How to read it: An association that remains with time and province held constant is not due to the groups having different ages or standing in different parts of the country.
Random effect: That the model lets each carver, or each parish, have its own level.; How to read it: A prolific carver's habit is then not counted as a general pattern. An association that holds with the carver as a random effect holds across carvers, not just for one of them.
Adjusted share, marginal effect: The share the model gives for a group when the other explanations are held equal for all groups.; How to read it: It can differ from the raw share, which also carries differences in time, place and carver.
The halves: The material is divided once and for all, parish by parish, into two halves, and a finding must hold in each half on its own.; How to read it: It is like testing the finding on new material. A pattern found in only one half may depend on a few parishes or carvers.
Holm correction: When a study makes several tests, the demand on each test is raised, so that the risk of at least one false finding in the study is kept below 5 %.; How to read it: A Holm-corrected p below 0.05 holds even when one counts that several questions were asked.
q, adjustment for all associations: All associations the project reports are adjusted together (Benjamini–Hochberg). q is the share of false findings to expect among all associations with a value that low.; How to read it: q below 0.05 means that at most one in twenty such findings is expected to be false, counting all the associations the project has tested.
Sensitivity analyses: The same test made with other models and selections: raw, adjusted, with the carver or the parish as a random effect, with the stone's location, in each half, without one province at a time and without moved stones.; How to read it: A finding that holds in all of them does not hang on a single choice in the analysis. One that fails shows where the finding is weak.
Pre-registration, locked plan: The hypothesis, the model and the limits were written down and time-stamped with a checksum before the material was analysed.; How to read it: Nobody can then change the question after the answer is known. What was computed afterwards is called exploratory and reported as such.
Exploratory: An analysis made after the result was known, without a locked plan.; How to read it: It can give ideas to test, but its p-values do not count that the question was chosen afterwards and are therefore too optimistic.
Replication: The same method, unchanged, tested on stones that were not included when the finding was made.; How to read it: A finding that replicates on new material is much stronger than one that holds only where it was found.
Shuffling, permutation test: The labels are shuffled at random thousands of times, for example which stone has which name, to see how often chance gives an equally strong pattern.; How to read it: If only one in ten thousand shuffles is as strong, p is about 0.0001.
Null model, random points: What the stones are compared with: how it would look had they stood at random. For places, randomly chosen points on land in the same district as the stones are used.; How to read it: If the stones stand closer to water than the points, people chose the place, rather than raising the stone wherever there was land. The points are drawn near the stones, so that the comparison concerns the place within the settled district, not the district within the country.
AUC: The probability that a randomly chosen case from one group has a lower value than a randomly chosen case from the other, for example that a stone stands closer to water than a random point.; How to read it: 0.5 means no difference and 1 that every stone stands closer than every point. An AUC of 0.63 means the stone stands closer in 63 comparisons out of 100. The measure does not depend on the number of stones, so it shows how large the difference is, not only that it exists.
Rank-sum test (Wilcoxon): A test of whether the values in one group are generally lower than in another. All values are ordered by size, and one sees whether one group's positions in the order are lower than chance gives.; How to read it: The test is not swayed by a few very large distances, only by the order. It gives the p-value; the AUC shows how large the difference is.
Bootstrap: A way to get the uncertainty of a measure: new samples are drawn from the material with replacement, thousands of times, and the measure is computed in each.; How to read it: The middle 95 per cent of the values give a 95 % interval. Drawing whole places rather than single stones takes into account that stones in a group are not independent.
Scan statistic: A search for the area where a trait is most common, with a test of whether it is more common there than chance gives.; How to read it: The p-value counts all the areas that were tried, so that one is not fooled by the most extreme area always looking special.
Rank correlation (Spearman's rho): How well two orderings follow each other, from −1 to 1.; How to read it: 0 means no association, 0.3 a weak one, 0.5 a moderate one and 0.8 a strong one. A partial correlation is the same measure with a third factor held constant.
Cross-validation: A model is trained on part of the stones and tested on the rest, in turn, so that every stone is judged by a model that has not seen it.; How to read it: The share right in the cross-validation shows how the model handles stones it does not know, not how well it remembers those it was trained on.
Blind test: The answer is hidden from the method, for example who carved the stone, and the method's suggestion is then compared with the answer.; How to read it: A blind test locked in advance is the strongest way to show that a method works, because the limits cannot be moved afterwards.
Calibration: Whether a model's probabilities agree with how often it is right.; How to read it: A well-calibrated model that says 70 % is right seven times in ten. A cautious model is right more often than it says.
McNemar's test: Compares two methods on the same stones by counting the stones where only one of them was right.; How to read it: If one method alone is right on many more stones than the other alone, the test gives a small p.
Random forest: A model that builds hundreds of decision trees on random parts of the stones and traits and lets them vote.; How to read it: It handles many traits and the relations between them without stating them in advance. Its probabilities are often cautious and are best read as a ranking.