BND - The Lab

🧪 The Lab

How we test the machine that reads the stories

76experiments so far
20in use on the site
29answered
2running

Our motif index is built by language models running on our own computer, with a person checking their work. Every change to how the models file and find stories is tried here first, on the person's own decisions, with its hypothesis written down before it runs, and kept only if it helps. Each experiment is named after a figure from folklore. The misses are here too: an idea that didn't work is a finding. The numbers are the experiments' own shorthand; the methods log tells the story of each change in plain words.

🏭 in use on the site 20 ✅ answered 29 ⏳ running 2 🔁 being run again 1 🙋 waiting on a decision 2 🤷 unproven either way 3 🗑️ tried, not kept 12 📝 not run yet 1 📝 planned 6

🏭 in use on the site E0.2

Sisyphus

An off-the-shelf reranker

🔬 AUC 0.887 with the likenesses and Gemma

🏭 in use on the site E0.5

the Bremen Town Musicians

Many finders, weighted on the person's filings

🔬 the learned shortlist

🏭 in use on the site E0.6

Athena

The confidence model on every signal

🔬 AUC 0.885; the fit on filings

🏭 in use on the site E1

Lethe

Subtract the day's news from claims and notes

🔬 kept

🏭 in use on the site E11

Babel

Every embedding model

🔬 Qwen3-Embedding 8B with Snowflake Arctic 2: top-12 82.3% to 85.2%

🏭 in use on the site E6

Ariadne

Which order to check in

🔬 least certain first

🏭 in use on the site E7

Matryoshka

The layered finder

🔬 kept

🏭 in use on the site E9

Pygmalion

The reranker taught the person's taste

🔬 in use

✅ answered E12

Jack

Seed motifs in the shortlist (from A1)

🔬 not buried: a quirk of the weights lifts motifs with no claims

✅ answered E13

Anansi

A fuller telling for finding a claim's motifs (from A1)

🔬 claim and telling together: right motif in the top 12 68% to 73% (top 8 63% to 69%)

🤷 unproven either way E5

Arachne

Spread scores over rests-on and related links

🔬 promising, unproven

🗑️ tried, not kept E0.1

the Oracle

Gemma's own yes or no

🔬 AUC 0.818 but overconfident; replaced by Nimble (J1)

🗑️ tried, not kept E2

Morpheus

Imagined instances of each motif

🔬 nothing

🗑️ tried, not kept E3

the Pleiades

Motifs travel in bundles

🔬 82.9% against 82.3%

🗑️ tried, not kept E4

the Three Wise Men

A jury of readers

🔬 nothing

🗑️ tried, not kept E8

Narcissus

The pick prompt showing a motif's nearest claims

🔬 no difference

🏭 in use on the site J1

the Norns

A jury of models for the judge

🔬 Nimble 9B judges as well as Gemma 26B, faster; second sitting (Oct 17): pplx-decider +0.010, 7x slower, Nimble kept

🏭 in use on the site J2

Solomon

The judge as the casebook reasons

🔬 the element won: AUC 0.54 to 0.79

🏭 in use on the site F10

Pandora

Pandora: the tellings inside the person's biggest motifs (a dry run)

🔬 all 36 motifs split into 227 tellings; 5 judged (all yes); blind set unjudged

🏭 in use on the site F11

the Danaids

How much the confidence model's leak flatters it

🔬 the leak was real and small (about 0.01 AUC); fixed Oct 14

🏭 in use on the site F16

the Magic Mirror

The taught reranker against the untaught, at each teaching and forward

🔬 no better: untaught 0.776 vs taught 0.758 held out, 0.738 vs 0.739 forward; teaching keeps the winner

🏭 in use on the site F17

Castor and Pollux

The checker's suggestions from the learned shortlist, as filing

🔬 the person's hand-added motif among the 12 candidates 62.9% to 68.4% (+3.0 to +9.0); 4.7 of 12 change

🏭 in use on the site F18

Cordelia

The Brothers Grimm's winning wording in every claim prompt, measured forward

🔬 Oct 21: posts 20.7 to 15.9 words; fact-checks not shorter; rewording fall confounded

🏭 in use on the site F3

Ma'at

A fixed yardstick: 200 held-out decisions, weighed weekly

🔬 weekly; Oct 18: first set AUC 0.915, the torn set 0.809

✅ answered F1

the Gordian Knot

Splitting motifs where they hold two tellings: Atropos's word cut on motifs

🔬 no: words can't see it (odd one out 11% against chance 14%; cut AUC 0.67)

✅ answered F14

Tyche

Bootstrap intervals in the harness; the Oct 8-10 gains read again

🔬 Babel clears the noise (+2.9, +1.7 to +4.2); Matryoshka did on Oct 8 but adds 0.2 now; the Pleiades inside the fold-split noise

✅ answered F15

Thumbelina

Recall by motif size, and the alike floor

🔬 a leak: one-claim motifs found 100% held out; without it top 12 86.7% to 72.7% (84.8% on motifs of 2+); two-claim motifs 68%; the floor costs 0.7%

✅ answered F19

Theseus

A telling as its course of action, no topic (SemEval 2026 Task 4's recipe), beside Aesop

🔬 no: alone 38% at top 12; summed with plain, -5 points

✅ answered F2

Cerberus

The motifs the machine missed: was your "also" in the shortlist?

🔬 half: the motif added in the top 12 50%, top 40 72% (the one chosen 56%, 86%; plain weights)

✅ answered F21

Charybdis

Straight news as a sink: offered beside the wrong motifs, is it picked where nothing fits?

🔬 no: offered, it changes wrong picks on nothing-fit claims by +0.9 points (needed -10); takes half of plain reports

✅ answered F23

Ptah

The note first, then the name: new motifs named after their scope note is written

🔬 no clear gain: names +0.008 likeness, near matches +3.4 points, but 7 points more abstract labels; blind pairs pending

✅ answered F24

Baron Munchausen

A telling, not a report: a report-or-telling signal and bulk passes counted for less, in the confidence model

🔬 not kept: report signals moved nothing (AUC within 0.002); the gate, blind on Oct 19-20, caught 24% (40% asked)

✅ answered F25

Indra's Net

Each motif mirrored in 20 versions (other casts, settings, voices) for the shortlist to catch claims by

🔬 not kept: top-12 +0.3 (87.5% to 87.8%); folktales +1.9; versions point right (66%) but say what the note says

✅ answered F26

Tiresias

The point a teller makes (what they mean, not why), written by Gemma from the claim and from its sources, as shortlist signals

🔬 not kept: top-12 +0.3% with sources (interval from -0.0%), +0.0% bare; moves +0.5%; 9 of 159 hard misses back

✅ answered F27

Ali Baba

The tellers' own words: each motif's phrases by weighted log-odds over its tellings, names out, as shortlist signals

🔬 not kept: top-12 +0.1% (interval from -0.3%), moves -0.3%; the phrases are mostly the topic in single words

✅ answered F4

Aesop

A telling abstracted as the proverb it is an instance of

🔬 no: alone 20% at top 12 against plain 71%; summed with plain, -17 points

✅ answered F5

Loki

A telling abstracted as the same story told about three other subjects

🔬 no: alone 41% at top 12; summed with plain, -6 points

✅ answered F6

Hermagoras

A telling abstracted as its rhetoric: stasis, driver, time shape, speech act

🔬 yes: same-motif pairs share a speech act +18 points, a stasis +13; carved claims stand out; next: as confidence features

✅ answered F7

Procrustes

Can the machine tell when nothing fits, and propose instead of forcing?

🔬 best shortlist fit spots nothing-fit at AUC 0.84; a line near 0.7 catches half, 9% of fits

✅ answered F8

the Chimera

Are claims from posts several tellings stitched into one?

🔬 yes, nearly all: ~4 claims a group, 3+ motif and one-motif groups alike; motifs on different sub-claims 46 of 59

✅ answered F9

Isis

Single posts filed on their own, then grouped by motif

🔬 single posts reach 79.5% of the person's motifs, 74.5% land in them; layered beat stitched 75 to 3

⏳ running F12

Prometheus

The shortlist forward: what each new claim is shown, against the final word

🔬 weekly; 1st week: 56% shown, 19% existed unshown, 25% made later

⏳ running F13

Persephone

Seeds let into the shortlist

🔬 weekly; 1st week: a seed shown for 55% of real claims, picked for 2%; 1 of 5 kept so far

🔁 being run again F28

the Judgement of Paris

Which model picks the motifs from the shortlist: Gemma 26B against Clef, Clef Flash and Nimble v3

🔬 the three are decision models (no completion), so they cannot pick; Gemma alone: found 53.6%, wrong 33.7%

🗑️ tried, not kept F20

the Blind Men and the Elephant

The claim writer shown the group's most varied posts, not ten at random

🔬 varied ten no better than random (65% about each); split covers 79% but 3.6 claims

🗑️ tried, not kept F22

Psyche

The claim organizer alongside the filing: a claim split where its motifs fall on different posts

🔬 splits about a quarter whatever the filing (28% vs 25%); its proposals never reached the person

✅ answered A1

Scheherazade

Our pipeline on 1,518 ATU-labelled folktales

🔬 right type in the shortlist ~9 in 10, picked 4 in 5, judge AUC 0.93-0.95

✅ answered A2

Rip Van Winkle

The 1896 newspapers: does filing hold up off-distribution; controls

🔬 96% filed, 40% sure; 24 new motifs (5% of filings); conspiracy controls read as class conflict; spot-check pending

✅ answered A3

Echo

Viral Texts: do reprints of one piece file alike?

🔬 copies of one piece file alike: Jaccard 0.32 within against 0.03 between; a shared motif in 59%

📝 planned A4

Cassandra

Greek and Roman speeches (Catilinarians the control)

📝 planned A5

Fama

Early modern news: Delpher 1618-99, Lancaster newsbooks

📝 planned A6

Argus

LOCO: conspiracy against matched mainstream documents, a control

📝 planned A7

Nestor

The Congressional Record 1873-2011: motifs over 140 years

📝 planned A8

the Sphinx

The Q drops: do their shapes meet 1896's?

📝 planned A9

Rashomon

ClaimsKG: does one claim file alike across fact-checkers?

✅ answered S1

Janus

A new event or the same one updated?

🔬 Gemma agrees 78%, Nimble 60%

✅ answered S3

Rumpelstiltskin

What makes headlines one story: TF-IDF, embeddings or hybrids

🔬 precision at equal coverage: TF-IDF 0.25-0.64, mxbai 0.88, hybrids 0.92-0.95; recall sample too small (3)

✅ answered S4

Atropos

Seams in stories: embeddings, then TF-IDF cuts

🔬 cuts scoring under 0.6 right 18 of 18, 0.6-0.7 69%, over 0.8 rarely

✅ answered S5

Sinbad

A long story split at each new development, replayed over every story of 36 hours or more

🔬 not as asked: 54% of long stories split, 10.6% into 5+ parts; caught 4 of 5 known developments but missed the Fort Hood cancellation; blind verdicts pending

📝 not run yet S2

Penelope

Re-sorting a saga's headlines into parts

🔬 three dry runs: k-means plus merging lumps or splits wrongly

✅ answered U1

the Golem

The machine alone: our filing from an empty index

🔬 it lumps: 63 motifs for 600 claims where you have 465; B-cubed precision 0.19, recall 0.53

✅ answered U1b

the Sorcerer's Apprentice

Golem II: the Golem's broad motifs split by Gemma into the tellings inside

🔬 63 motifs became 225; B-cubed precision 0.19 to 0.54, recall 0.53 to 0.44

🏭 in use on the site P1

Puck

Nimble scoring the proposals

🔬 ranking AUC 0.772 against 0.755

🏭 in use on the site P2

Adam

The Oct 10 praxis: the judge's question and the naming rules

🔬 naming rules live Oct 10

🙋 waiting on a decision P2b

Nobody

The naming rules without a leading "The"

🔬 house style kept

✅ answered D1

Cinderella

Does the judge read the must-have element?

🔬 yes: AUC 0.821 with its own element, 0.64-0.65 with another motif's

🙋 waiting on a decision D4

Proteus

Tell-Tale Hat: which motifs overlap or drift

🔬 22 candidate pairs, 6 drifting motifs

🤷 unproven either way D2

Bluebeard

Show the judge the person's past removals?

🔬 AUC 0.796 to 0.809, within the noise

🗑️ tried, not kept D3

the Hydra

Simulated annotators: does agreement say when to trust the judge?

🔬 no: the confidence model stays at AUC 0.885

🗑️ tried, not kept D5

Punch and Judy

Tangherlini's actants as signals

🔬 strong alone (AUC 0.899) but the embeddings already carry it

🗑️ tried, not kept R1

Goliath

A bigger reranker (4B)

🔬 not worth it

🗑️ tried, not kept C1

Goldilocks

Small models first, Gemma only when unsure

🔬 Gemma alone picks with the fewest wrong

🏭 in use on the site M1

the Changeling

Gemma 26B or Qwen3 30B for the match question

🔬 Gemma; the 30B retired

🤷 unproven either way W1

the Brothers Grimm

Writing claims with rules from the person's corrections

🔬 Gemma closer than Qwen; rules unproven

🏭 in use on the site G1

Mother Goose

Genre descriptions from the praxis

🔬 genre calls right 53% to 64-67%

🗑️ tried, not kept G2

Noah

Group descriptions for the drafter

🔬 no help

E · the shortlist: finding a claim's motifs