Data dictionary

What each file and field in a snapshot holds, and what the labels mean. Every snapshot carries a copy of this page as DATA_DICTIONARY.md and codebook.md.

Files in a snapshot

cases.jsonl
One case per line, as JSON. Every field is listed below.
images.zip
The released screenshots, named by image_key. Personal details are covered by solid black boxes.
codebook.md
What each label means and how labels group into broad classes.
DATA_DICTIONARY.md
This dictionary, as it stood when the snapshot was cut.
manifest.json
Counts, checksums, the licence and the takedown contact. Its keys are listed below.

Identity and provenance

id
string
Stable case id across snapshots, in the form system:source id.
source_system
legacy | current
Which CheckMate platform the case came from. legacy is the original WhatsApp bot, March 2023 to January 2026. current is the platform in use since October 2025. Both ran side by side from October 2025 to January 2026.
first_date
date, YYYY-MM-DD
The day the message was first sent to CheckMate, in Singapore time. Day resolution only.
snapshot_added
string, YYYY-MM-DD
The snapshot in which this case was first published. A case's row is frozen then and copied into later snapshots unchanged, unless a later snapshot hides a personal detail we had missed. Earlier snapshots are never changed, so the newest one is the safest to use.
modality
text | image | image+caption
What the submitter sent.

Content

Personal details of reporters and bystanders are replaced by numbered placeholders such as <NAME_1> or <PHONE_2>, consistent across text, caption and note within a case. Other people's names are partly masked PayNow-style, R****a T*n. Scammer numbers, handles, links and payment details are kept.

text
string, may be null
Message text after redaction.
caption
string, may be null
The submitter's caption on an image, after redaction. Absent on most cases.
image_key
string, may be null
Filename of the released screenshot, bench-<sha256>.jpeg, the content hash of the released file. Redactions are burned in as solid black boxes.
has_image
boolean
Whether the case has a released image.

Crowd labels

These are the consensus of volunteer checkers. They are noisy and are not ground truth.

label
string
Granular crowd label, one of scam, illicit, untrue, misleading, spam, accurate, legitimate, satire, irrelevant, unsure. unsure means checkers couldn't reach a verdict.
broad_label
bad | caution | good | satire | nothing, may be null
Five-class rollup of label. The codebook lists which labels fall in each class. Null for unsure, which has no verdict to roll up.
is_scam
boolean, may be null
Binary task label. scam and illicit are true, and other labels are false, misinformation included. irrelevant and unsure are null and sit outside the scam task.
truth_score
number, may be null
Mean checker truth score for information-type items, on the scale named in truth_score_scale.
truth_score_scale
5-point | 6-point, may be null
Which scale the votes were cast on. 5-point runs from 1 to 5 and 6-point from 0 to 5. Legacy cases first sent between 3 March and 6 September 2024 are 5-point, and everything else is 6-point. Compare scores only within a scale.
category_votes
object {category: count}
Full distribution of substantive checker votes. Abstentions are excluded. Legacy votes use a single info option for untrue, misleading and accurate, with the truth score cast separately.
votes_total
integer
Sum of category_votes.
votes_top
integer
Count for the plurality category.
votes_top_category
string, may be null
The plurality category.
votes_tied_top
boolean
Two categories share the top count.
vote_agreement
number, 0 to 1, may be null
votes_top divided by votes_total. Null when there were no votes, which happens for legacy items the bot resolved by matching an earlier message.
is_controversial
boolean, may be null
Platform flag set when checkers were split.

The AI note and what people made of it

has_ai_note
boolean
Whether the pipeline produced a community note.
ai_note
string, may be null
The English community note sent to the user, redacted with the same placeholders as the text.
ai_note_sources
array of string, may be null
URLs the note cited.
note_ratings
object {great, acceptable, unacceptable}
How checkers rated the note, as counts. Divide by votes_total if you want shares, with care: who bothers to rate depends on the label.
note_downvoted
boolean, may be null
Whether the person who received the note gave it a thumbs down.

Release metadata

scrub_status
human_approved | auto_scrubbed
human_approved when every part of the case that needed a privacy judgement was judged by a person. auto_scrubbed when the text decision was the automated pipeline's alone. Every released image was reviewed by a person.

What the labels mean

Volunteer checkers vote on each message, and the label is their consensus. It is noisy and isn't ground truth.

scam
bad, is_scam true
Fraud. Someone is deceiving the reader to get their money or their personal details.
illicit
bad, is_scam true
An offer of something illegal, such as unlicensed moneylending or vice. The current platform merged this option into scam in February 2026.
untrue
bad, is_scam false
A claim checkers judged false, going by its mean truth score.
misleading
caution, is_scam false
A claim whose mean truth score landed in the middle of the scale.
spam
caution, is_scam false
Marketing and promotions, often forwarded en masse.
accurate
good, is_scam false
A claim checkers judged true, going by its mean truth score.
legitimate
good, is_scam false
Content checkers verified as genuine.
satire
satire, is_scam false
Clear satire or parody.
irrelevant
nothing, is_scam null
Nothing in it can be checked, or checkers couldn't tell what it was.
unsure
null, is_scam null
Checkers couldn't reach a verdict. Their broad_label and is_scam are null. The current platform sometimes recorded a split like this as harmful or harmless, and we count those as unsure too.

Broad classes

bad
Scams, illegal offers and false claims.
caution
Misleading claims and spam. We may move spam to nothing after review.
good
Claims checkers judged accurate, and messages they verified as genuine.
satire
Satire and parody.
nothing
Messages with nothing to check.

The scam flag

is_scam is true for scam and illicit, and false for other labels, misinformation included. A false claim can mislead people without being after their money. irrelevant and unsure are null, so those cases sit outside the scam task.

Truth scores

When checkers vote that a message is information, each one also gives it a truth score. The mean score decides between untrue, misleading and accurate. The current platform scores from 0 to 5: below 1.5 is untrue, 1.5 to 3.75 is misleading, and anything higher is accurate.

Truth scores on the legacy platform

The legacy platform changed its scale twice. At first it scored from 0 to 5, with untrue below 1.5 and misleading up to 3.5. Its code moved to a 1 to 5 scale on 3 March 2024. Untrue started below 2 and moved to below 2.5 in June, and misleading went up to 4. The code went back to 0 to 5 on 7 September 2024, and a week later it settled on the current platform's cut-offs. These are the defaults written in the code, with the dates the code changed. The live settings were kept outside the code and weren't recorded, so they may have differed. truth_score_scale says which scale a case used, and scores only compare within one scale.

The consensus bar

A case meets the consensus bar when its label isn't unsure, it has at least one checker vote, and the top two categories aren't tied. The manifest counts how many cases in a snapshot meet it.

Votes on the legacy platform

Legacy checkers voted info for untrue, misleading and accurate alike, then gave a truth score. For those cases, vote_agreement measures agreement on info, which is coarser than the final label.

manifest.json

version
The snapshot id.
schema_version
Version of the file layout this dictionary describes. It changes when a field is added, renamed or removed.
frozen_at
When the snapshot was built, in UTC.
cutoff
The last day included. Cases first sent to CheckMate after it wait for a later snapshot.
previous_snapshot
The snapshot before this one, or null for the first.
licence
The licence the data is released under.
contact
Where to send takedown requests and questions.
takedown_days
How many days we take to remove a case after a valid request.
counts
The total, how many cases are new in this snapshot, how many earlier cases we corrected to hide a missed personal detail, and breakdowns by source system, modality, label and broad label. It also counts cases with an image or an AI note, and cases that meet the consensus bar.
scrub
Cases by scrub_status, whether auto-scrubbed text is included, and how many distinct images were released and reviewed by a person.
leak_rate
How often personal details got past redaction, measured on a sample after release, for images and for text. Each figure is null until its sample is checked, and the note says where things stand.
agreement
How often people and the automated pipeline agreed on redaction. Null for now. The methodology page has the figures measured so far.
codebook_version
sha256 of codebook.md. It changes whenever the codebook's text changes, so two snapshots with the same value used the same labels.
source_commit
The version of our release code that built the snapshot. The dictionary and codebook files are generated from it.
excluded
How many cases were left out, and why, as counts only. The reasons are explained below.
withheld_hashes
sha256 of the id of every withheld case. They show the withheld cases existed at the cutoff without revealing them.
image_base_url
The folder inside images.zip. Put it in front of image_key to find an image.
preview_ids
The ids of the cases shown on the website for this snapshot.
files
Size in bytes and sha256 of every other file in the snapshot.

Why cases were excluded

withheld
Kept out of every snapshot so models can be benchmarked on messages they cannot have trained on.
withdrawn
Taken out at someone's request.
after_cutoff
First sent after the cutoff.
unusable
There was no content left to retrieve.
unpublishable_image
Its image showed one person's document, or had nothing left worth seeing once redacted. The case goes with the image.
label_not_final
Voting was still open.
not_yet_redacted
Redaction hadn't run on it yet.
not_yet_scrubbed
Waiting for the privacy pipeline.
text_not_human_approved
Its text was cleaned by the pipeline alone, and this snapshot only includes text a person approved.
dropped_from_previous_snapshot
Released in an earlier snapshot, but it no longer qualifies.
no_label
The case has no label.
unknown_label
Its label isn't one of the labels in the codebook.
excluded
Marked for exclusion without a recorded reason.
other
Didn't fit any of the reasons above.

Deliberately not released

  • Submitter phone numbers, WhatsApp ids and any user id. Never extracted.
  • Text and captions before redaction, and the per-row record of what was masked.
  • Exact timestamps. Day resolution only.
  • OCR text of images. It duplicates the image and leaks what the boxes hide.
  • The withheld slice: cases whose id hash falls in the bottom 15%. Their hashes are listed in each manifest as proof they existed.
  • The language of each case.