Data dictionary
What each file and field in a snapshot holds, and what the labels mean. Every snapshot carries a copy of this page as DATA_DICTIONARY.md and codebook.md.
Files in a snapshot
cases.jsonl- One case per line, as JSON. Every field is listed below.
images.zip- The released screenshots, named by image_key. Personal details are covered by solid black boxes.
codebook.md- What each label means and how labels group into broad classes.
DATA_DICTIONARY.md- This dictionary, as it stood when the snapshot was cut.
manifest.json- Counts, checksums, the licence and the takedown contact. Its keys are listed below.
Identity and provenance
idstring- Stable case id across snapshots, in the form system:source id.
source_systemlegacy | current- Which CheckMate platform the case came from. legacy is the original WhatsApp bot, March 2023 to January 2026. current is the platform in use since October 2025. Both ran side by side from October 2025 to January 2026.
first_datedate, YYYY-MM-DD- The day the message was first sent to CheckMate, in Singapore time. Day resolution only.
snapshot_addedstring, YYYY-MM-DD- The snapshot in which this case was first published. A case's row is frozen then and copied into later snapshots unchanged, unless a later snapshot hides a personal detail we had missed. Earlier snapshots are never changed, so the newest one is the safest to use.
modalitytext | image | image+caption- What the submitter sent.
Content
Personal details of reporters and bystanders are replaced by numbered placeholders such as <NAME_1> or <PHONE_2>, consistent across text, caption and note within a case. Other people's names are partly masked PayNow-style, R****a T*n. Scammer numbers, handles, links and payment details are kept.
textstring, may be null- Message text after redaction.
captionstring, may be null- The submitter's caption on an image, after redaction. Absent on most cases.
image_keystring, may be null- Filename of the released screenshot, bench-<sha256>.jpeg, the content hash of the released file. Redactions are burned in as solid black boxes.
has_imageboolean- Whether the case has a released image.
Crowd labels
These are the consensus of volunteer checkers. They are noisy and are not ground truth.
labelstring- Granular crowd label, one of scam, illicit, untrue, misleading, spam, accurate, legitimate, satire, irrelevant, unsure. unsure means checkers couldn't reach a verdict.
broad_labelbad | caution | good | satire | nothing, may be null- Five-class rollup of label. The codebook lists which labels fall in each class. Null for unsure, which has no verdict to roll up.
is_scamboolean, may be null- Binary task label. scam and illicit are true, and other labels are false, misinformation included. irrelevant and unsure are null and sit outside the scam task.
truth_scorenumber, may be null- Mean checker truth score for information-type items, on the scale named in truth_score_scale.
truth_score_scale5-point | 6-point, may be null- Which scale the votes were cast on. 5-point runs from 1 to 5 and 6-point from 0 to 5. Legacy cases first sent between 3 March and 6 September 2024 are 5-point, and everything else is 6-point. Compare scores only within a scale.
category_votesobject {category: count}- Full distribution of substantive checker votes. Abstentions are excluded. Legacy votes use a single info option for untrue, misleading and accurate, with the truth score cast separately.
votes_totalinteger- Sum of category_votes.
votes_topinteger- Count for the plurality category.
votes_top_categorystring, may be null- The plurality category.
votes_tied_topboolean- Two categories share the top count.
vote_agreementnumber, 0 to 1, may be null- votes_top divided by votes_total. Null when there were no votes, which happens for legacy items the bot resolved by matching an earlier message.
is_controversialboolean, may be null- Platform flag set when checkers were split.
The AI note and what people made of it
has_ai_noteboolean- Whether the pipeline produced a community note.
ai_notestring, may be null- The English community note sent to the user, redacted with the same placeholders as the text.
ai_note_sourcesarray of string, may be null- URLs the note cited.
note_ratingsobject {great, acceptable, unacceptable}- How checkers rated the note, as counts. Divide by votes_total if you want shares, with care: who bothers to rate depends on the label.
note_downvotedboolean, may be null- Whether the person who received the note gave it a thumbs down.
Release metadata
scrub_statushuman_approved | auto_scrubbed- human_approved when every part of the case that needed a privacy judgement was judged by a person. auto_scrubbed when the text decision was the automated pipeline's alone. Every released image was reviewed by a person.
What the labels mean
Volunteer checkers vote on each message, and the label is their consensus. It is noisy and isn't ground truth.
scambad, is_scam true- Fraud. Someone is deceiving the reader to get their money or their personal details.
illicitbad, is_scam true- An offer of something illegal, such as unlicensed moneylending or vice. The current platform merged this option into scam in February 2026.
untruebad, is_scam false- A claim checkers judged false, going by its mean truth score.
misleadingcaution, is_scam false- A claim whose mean truth score landed in the middle of the scale.
spamcaution, is_scam false- Marketing and promotions, often forwarded en masse.
accurategood, is_scam false- A claim checkers judged true, going by its mean truth score.
legitimategood, is_scam false- Content checkers verified as genuine.
satiresatire, is_scam false- Clear satire or parody.
irrelevantnothing, is_scam null- Nothing in it can be checked, or checkers couldn't tell what it was.
unsurenull, is_scam null- Checkers couldn't reach a verdict. Their broad_label and is_scam are null. The current platform sometimes recorded a split like this as harmful or harmless, and we count those as unsure too.
Broad classes
bad- Scams, illegal offers and false claims.
caution- Misleading claims and spam. We may move spam to nothing after review.
good- Claims checkers judged accurate, and messages they verified as genuine.
satire- Satire and parody.
nothing- Messages with nothing to check.
The scam flag
is_scam is true for scam and illicit, and false for other labels, misinformation included. A false claim can mislead people without being after their money. irrelevant and unsure are null, so those cases sit outside the scam task.
Truth scores
When checkers vote that a message is information, each one also gives it a truth score. The mean score decides between untrue, misleading and accurate. The current platform scores from 0 to 5: below 1.5 is untrue, 1.5 to 3.75 is misleading, and anything higher is accurate.
Truth scores on the legacy platform
The legacy platform changed its scale twice. At first it scored from 0 to 5, with untrue below 1.5 and misleading up to 3.5. Its code moved to a 1 to 5 scale on 3 March 2024. Untrue started below 2 and moved to below 2.5 in June, and misleading went up to 4. The code went back to 0 to 5 on 7 September 2024, and a week later it settled on the current platform's cut-offs. These are the defaults written in the code, with the dates the code changed. The live settings were kept outside the code and weren't recorded, so they may have differed. truth_score_scale says which scale a case used, and scores only compare within one scale.
The consensus bar
A case meets the consensus bar when its label isn't unsure, it has at least one checker vote, and the top two categories aren't tied. The manifest counts how many cases in a snapshot meet it.
Votes on the legacy platform
Legacy checkers voted info for untrue, misleading and accurate alike, then gave a truth score. For those cases, vote_agreement measures agreement on info, which is coarser than the final label.
manifest.json
version- The snapshot id.
schema_version- Version of the file layout this dictionary describes. It changes when a field is added, renamed or removed.
frozen_at- When the snapshot was built, in UTC.
cutoff- The last day included. Cases first sent to CheckMate after it wait for a later snapshot.
previous_snapshot- The snapshot before this one, or null for the first.
licence- The licence the data is released under.
contact- Where to send takedown requests and questions.
takedown_days- How many days we take to remove a case after a valid request.
counts- The total, how many cases are new in this snapshot, how many earlier cases we corrected to hide a missed personal detail, and breakdowns by source system, modality, label and broad label. It also counts cases with an image or an AI note, and cases that meet the consensus bar.
scrub- Cases by scrub_status, whether auto-scrubbed text is included, and how many distinct images were released and reviewed by a person.
leak_rate- How often personal details got past redaction, measured on a sample after release, for images and for text. Each figure is null until its sample is checked, and the note says where things stand.
agreement- How often people and the automated pipeline agreed on redaction. Null for now. The methodology page has the figures measured so far.
codebook_version- sha256 of codebook.md. It changes whenever the codebook's text changes, so two snapshots with the same value used the same labels.
source_commit- The version of our release code that built the snapshot. The dictionary and codebook files are generated from it.
excluded- How many cases were left out, and why, as counts only. The reasons are explained below.
withheld_hashes- sha256 of the id of every withheld case. They show the withheld cases existed at the cutoff without revealing them.
image_base_url- The folder inside images.zip. Put it in front of image_key to find an image.
preview_ids- The ids of the cases shown on the website for this snapshot.
files- Size in bytes and sha256 of every other file in the snapshot.
Why cases were excluded
withheld- Kept out of every snapshot so models can be benchmarked on messages they cannot have trained on.
withdrawn- Taken out at someone's request.
after_cutoff- First sent after the cutoff.
unusable- There was no content left to retrieve.
unpublishable_image- Its image showed one person's document, or had nothing left worth seeing once redacted. The case goes with the image.
label_not_final- Voting was still open.
not_yet_redacted- Redaction hadn't run on it yet.
not_yet_scrubbed- Waiting for the privacy pipeline.
text_not_human_approved- Its text was cleaned by the pipeline alone, and this snapshot only includes text a person approved.
dropped_from_previous_snapshot- Released in an earlier snapshot, but it no longer qualifies.
no_label- The case has no label.
unknown_label- Its label isn't one of the labels in the codebook.
excluded- Marked for exclusion without a recorded reason.
other- Didn't fit any of the reasons above.
Deliberately not released
- Submitter phone numbers, WhatsApp ids and any user id. Never extracted.
- Text and captions before redaction, and the per-row record of what was masked.
- Exact timestamps. Day resolution only.
- OCR text of images. It duplicates the image and leaks what the boxes hide.
- The withheld slice: cases whose id hash falls in the bottom 15%. Their hashes are listed in each manifest as proof they existed.
- The language of each case.