ScreenPost blog

Finding secrets in screenshots with Vision on macOS, and what OCR misses

I post a lot of screenshots of my own work: terminals, config files, dashboards. Sooner or later one of them has an API key in it. So ScreenPost, my Mac app for posting screenshots to X, finds secrets in a screenshot and covers them before anything leaves the Mac. This is how that works with Apple's Vision framework, the problems I ran into, and the cases it still misses.

Reading the text: Vision on the device

The first step is plain OCR with VNRecognizeTextRequest. Two settings matter:

Everything runs on the Mac. No image or text is uploaded to find the secrets, which matters for a feature whose whole point is privacy.

Deciding what counts as a secret

Vision returns lines of text. Each line goes through one set of rules (the same rules the app uses everywhere it looks for secrets, so the editor and the AI features can never disagree):

Two OCR quirks I had to handle

Look-alike letters. Vision sometimes reads a Latin letter as its Cyrillic or Greek twin: an а that is not an a, an о that is not an o. To a regular expression, ѕk- with a Cyrillic ѕ is not sk-. So before matching, the text is folded to Latin look-alikes one character for one character, which keeps every position the same, and the matches are mapped back to the original text.

Split tokens. OCR sometimes breaks a long token with a space, so ghp_ and the rest of the key come back as two words. A short body right after a known prefix followed by one space and more token is treated as one value.

Covering the value, not the whole line

This was the hardest part. Vision can give you a bounding box for part of a line, but it works at word level. API_KEY=abc123... has no spaces, so it is one word, and the box Vision returns for "just the value" is the box of the whole token.

Covering the whole line is safe but ugly, and it hides the context that makes the screenshot worth posting. So when a line is set in a monospaced font, the app can estimate where the value starts from the character count, since every character has the same width. It needs evidence before trusting that:

When the evidence comes from the other lines, the bar gets extra padding. And when Vision cannot place the value at all, the whole line is covered. Every uncertain case fails toward covering more.

An opaque bar, not a blur

Each match becomes an opaque black bar that replaces the pixels underneath. A blur keeps a version of the original pixels in the image, and with short or predictable text it can sometimes be reversed. I wrote more about that in how to redact an API key in a Mac screenshot. In the editor the bars stay editable, so you can remove a false positive, and the exported image only ever contains the covered version.

What it still misses

Automatic redaction is a safety net, not a guarantee. These are the gaps I know about:

It also errs the other way: a long order number can look like a card number and get covered. That is the trade I chose. A false positive costs one click to remove, a miss costs a rotated key.

So the habit stays the same: look at the screenshot before you post it. The difference is that the obvious things are already covered when you do.

ScreenPost is a Mac app that takes a screenshot from capture to a posted X post, with redaction, annotations and framing built in. $29 once, 7-day free trial: screenpost.app. It needs macOS 14 or later on Apple silicon.

ScreenPost is a Mac app that takes a screenshot or recording from capture to a posted X post: redact secrets, beautify, post or schedule. $29 once, 7-day free trial. See how it works.