How the dataset was made
Every case started as a message someone forwarded to CheckMate because it looked suspicious. Before any of it could be shared, the personal details of the people who forwarded messages, and of bystanders caught in them, had to go. This page explains the rule we used, how it was applied to text and to screenshots, how well the automated parts performed when a person checked them, and what was dropped altogether.
The rule
Leave a detail alone unless it is clear that it belongs to the person who forwarded the message, or to an innocent third party who is not part of the scam. Everything else stays.
That means a scammer's phone number, WhatsApp handle, phishing link, PayNow account and made-up company name all remain in the data. They are the evidence, and masking them would destroy what the dataset exists to preserve. Public figures, brands, agencies and news outlets stay too. A name or number with no context that tells us whose it is stays as well, because ambiguous means keep.
What goes: the recipient's own name in a greeting, their number or email quoted back at them, their avatar or their contacts' names in the frame of a screenshot they took, friends in a reaction bar, members of a private group, a real person's face in a photo the reporter took, and anyone's NRIC.
An earlier version of the pipeline masked anything it was unsure about. On real messages that default stripped out scammer details wholesale, so it was replaced by the rule above, and the automated output became a set of hints for a person rather than a decision.
Text
Text goes through three passes. First, pattern matching finds candidate spans: phone numbers, NRIC and FIN numbers, email addresses, bank account numbers, links, handles and long runs of digits. A language model then reads each span in the context of the whole message and sorts it into one of five buckets, from scammer-operated to private.
Second, a stronger model applies the rule to every candidate, with the message, its caption, the crowd's label and the screenshot in front of it. For each span it proposes one of three outcomes. Keep. Redact, naming whether the span belongs to the reporter or to a third party. Or partial, which is used for two things: any personal name that is neither the reporter's nor a public figure's is masked PayNow-style, so Regina Tan becomes R****a T*n, and a link whose tail is a one-off code is cut after the last slash so the domain stays readable but the code cannot be followed.
Third, the masks are applied. A redacted span becomes a numbered placeholder such as <NAME_1> or <PHONE_2>, and the same placeholder is used wherever that span appears in the message, the caption and the AI-written note, so a reader can still tell that two mentions refer to the same thing. If a span the pipeline meant to mask survives anywhere in the output, the build stops.
How accurate the automated decisions were
A person checked 155 text cases by hand and the pipeline's decisions were compared with theirs. On the first 25 calibration cases, 59 of 60 span decisions matched. On a further 128 cases, chosen from those the pipeline had wanted to mask, 476 span decisions were compared: 98.3% agreed, with Cohen's kappa of 0.97. Where both chose to redact, they always agreed on who the detail belonged to.
The other text cases carry the pipeline's decisions only. Each released row says which: scrub_status is human_approved when a person made every call the case needed, and auto_scrubbed when the text decision was the pipeline's alone. Researchers who want only the hand-checked subset can filter on it.
Screenshots
Screenshots leak far more than text does: chat headers, contact names, profile photos and reaction bars all sit around the message itself. So every one of the 2,677 screenshots was reviewed by a person. There is no automated-only tier for images.
The automated pass still runs first, as a starting point. Optical character recognition finds every line of text and where it sits on the image, and a vision model returns a verdict, clean, redact or needs a person, with proposed boxes over anything it thinks should go. The reviewer then opens each image in a labelling tool with the proposals already drawn, and keeps, moves, deletes or adds boxes under the same rule as for text. A box is tagged as covering the reporter's detail or a third party's.
On 404 screenshots where the model's proposals and the person's final boxes were compared, the two agreed on the verdict 83.7% of the time, with kappa 0.68. Of the 479 boxes the person drew, the model had missed 110. That miss rate is why no screenshot is published on the model's word alone.
Redactions are painted onto the image as solid black rectangles. Blurring and pixelation are not used because they can be reversed. The painted file is a new file with a new content hash, so a released image key never points at an unredacted original. Of the 2,677 screenshots, 1,621 needed nothing, 913 had boxes painted in, and 143 were dropped.
Screenshots that were dropped, and why
Some images cannot be made safe with boxes. A screenshot that is one private person's document, an identity card, a full bank statement, a medical record, is that person, and blacking out parts of it leaves a document that is still theirs. The reviewer marks such an image unpublishable and it is dropped, together with every case that used it, because a case without its screenshot would misrepresent what was sent.
The same fate goes to an image that adds nothing, such as a blank or corrupted upload. The count of cases dropped this way appears in each snapshot's manifest.
What else is left out
Cases with no retrievable content. Exact timestamps, which are reduced to the day. The text that optical character recognition read off screenshots, because it duplicates the image and would leak what the boxes hide. The text and captions before redaction, and the per-case record of what was masked.
And a fixed 15% of cases, chosen by hashing each case id, that never appears in any snapshot. It is kept back so that models can be benchmarked on messages they cannot have trained on. Each manifest lists the hashes of the withheld ids as proof they existed at release time.
What remains uncertain
No redaction reaches zero leakage. A scrubbed message can still be distinctive enough to be searched for. A reviewer can miss a name in a crowded screenshot. The figures above are agreement rates between a person and a pipeline, measured on samples, and the residual rate of personal details in the auto-scrubbed text has not yet been measured on the full corpus. That is why images come with conditions on redistribution and re-identification, and why anyone appearing in a case can ask for it to be removed by writing to contact@checkmate.sg. We act within 14 days and tell everyone on the download log.
The tools that did this work are open: the detectors, the model prompts, the labelling tool and the scripts that apply masks and paint boxes live in the checkmate-sg repositories. The label taxonomy is in the data dictionary and the codebook that ships with every snapshot.