Finding secrets in screenshots with Vision on macOS, and what OCR misses
I post a lot of screenshots of my own work: terminals, config files, dashboards. Sooner or later one of them has an API key in it. So ScreenPost, my Mac app for posting screenshots to X, finds secrets in a screenshot and covers them before anything leaves the Mac. This is how that works with Apple's Vision framework, the problems I ran into, and the cases it still misses.
Reading the text: Vision on the device
The first step is plain OCR with VNRecognizeTextRequest. Two settings matter:
recognitionLevel = .accurate. The fast level is good enough for prose and bad at the long random strings that secrets are made of.usesLanguageCorrection = false. Language correction tries to turn what it reads into real words. For a sentence that helps. For a key likeghp_followed by 36 random characters it does the opposite: it "fixes" characters and the result no longer looks like a key.
Everything runs on the Mac. No image or text is uploaded to find the secrets, which matters for a feature whose whole point is privacy.
Deciding what counts as a secret
Vision returns lines of text. Each line goes through one set of rules (the same rules the app uses everywhere it looks for secrets, so the editor and the AI features can never disagree):
- Personal values: email addresses, IPv4 addresses, and card-like numbers of 13 to 19 digits, with or without spaces or dashes.
- Known key formats: tokens that start with a known prefix, such as
sk-,sk_live_,ghp_,github_pat_,glpat-,xoxb-,AKIA,AIza, oreyJ(the start of a JWT), with at least a few characters after the prefix. - Assignments: the value in
KEY=valueor"apiToken": "value"when the name contains a word like KEY, TOKEN, SECRET, PASSWORD, AUTH, COOKIE, SESSION or DSN, and the value is at least 8 characters. Only the value is covered, not the name, so the screenshot still shows what the line was. - URLs: the password in
scheme://user:password@host, and query values likecode=,state=,access_token=orclient_secret=. - Random-looking strings: an assigned value of 16 or more characters, or a standalone token of 20 or more, that mixes lower case, upper case and digits.
Two OCR quirks I had to handle
Look-alike letters. Vision sometimes reads a Latin letter as its Cyrillic or Greek twin: an а that is not an a, an о that is not an o. To a regular expression, ѕk- with a Cyrillic ѕ is not sk-. So before matching, the text is folded to Latin look-alikes one character for one character, which keeps every position the same, and the matches are mapped back to the original text.
Split tokens. OCR sometimes breaks a long token with a space, so ghp_ and the rest of the key come back as two words. A short body right after a known prefix followed by one space and more token is treated as one value.
Covering the value, not the whole line
This was the hardest part. Vision can give you a bounding box for part of a line, but it works at word level. API_KEY=abc123... has no spaces, so it is one word, and the box Vision returns for "just the value" is the box of the whole token.
Covering the whole line is safe but ugly, and it hides the context that makes the screenshot worth posting. So when a line is set in a monospaced font, the app can estimate where the value starts from the character count, since every character has the same width. It needs evidence before trusting that:
- the line's own words show an equal width per character, or
- if the line is a single word with no evidence of its own, a clear majority of the lines at the same text height, at least four of them, share its width per character within 1%.
When the evidence comes from the other lines, the bar gets extra padding. And when Vision cannot place the value at all, the whole line is covered. Every uncertain case fails toward covering more.
An opaque bar, not a blur
Each match becomes an opaque black bar that replaces the pixels underneath. A blur keeps a version of the original pixels in the image, and with short or predictable text it can sometimes be reversed. I wrote more about that in how to redact an API key in a Mac screenshot. In the editor the bars stay editable, so you can remove a false positive, and the exported image only ever contains the covered version.
What it still misses
Automatic redaction is a safety net, not a guarantee. These are the gaps I know about:
- IPv6 addresses and phone numbers. Only IPv4 has a pattern, and a phone number is too short to look like a card.
- Names, street addresses and faces. These are not patterns in text, so they need a person's eye.
- Short secrets with no clue. A 12-character password that is not next to a word like PASSWORD and has no known prefix looks like any other word.
- Text OCR cannot read. Tiny, low-contrast or blurred text, and anything in a QR code, never reaches the rules.
- Values broken across two lines. Each line is checked on its own.
It also errs the other way: a long order number can look like a card number and get covered. That is the trade I chose. A false positive costs one click to remove, a miss costs a rotated key.
So the habit stays the same: look at the screenshot before you post it. The difference is that the obvious things are already covered when you do.
ScreenPost is a Mac app that takes a screenshot from capture to a posted X post, with redaction, annotations and framing built in. $29 once, 7-day free trial: screenpost.app. It needs macOS 14 or later on Apple silicon.