---
title: 'From OCR to Pixels Without Losing the Fallback'
excerpt: 'GeekBye stopped reading your screenshots with OCR and started sending the actual image to vision models — which is the easy part. The instructive part is the optimization that skipped OCR entirely for vision users, quietly broke the fallback to non-vision models, and had to be walked back to a version where the safety net runs in parallel and costs nothing.'
date: '2026-07-19'
author: 'Steven'
authorAvatar: '/images/blog/authors/steven.jpg'
coverImage: '/images/blog/covers/from-ocr-to-pixels-without-losing-the-fallback.svg'
tags: ['Engineering', 'AI', 'Desktop', 'GeekBye Releases']
keywords:
  - 'send screenshot to vision model vs ocr'
  - 'image optimization before sending to llm sharp'
  - 'ocr fallback when vision model unavailable'
  - 'base64 image request payload size vision model tokens'
  - 'resize jpeg quality vision model token cost'
  - 'graceful degradation fallback capability flag'
lastModified: '2026-07-19'
tldr: "GeekBye used to turn a screenshot into an AI answer by running OCR on-device (Apple Vision on macOS, a PowerShell OCR on Windows), sending only the extracted text and discarding the image — which throws away layout, diagrams, and code indentation. v1.8.6 switched to vision-first: a `sharp` step resizes the screenshot to 1920px wide and recompresses it to JPEG quality 80 (turning a 3–8 MB PNG into a ~200–500 KB image), sends it as base64, and lets the model read the pixels, with OCR kept only as a fallback. The real lesson is the wobble that followed: the first optimization skipped OCR *entirely* for vision users, which broke the case where the backend falls back from a vision model to a non-vision one — now there was no text to fall back to. The fix wasn't to un-skip OCR; it was to gate the skip on a *second* capability flag (does the fallback support vision?) and run the insurance OCR in parallel with image optimization via `Promise.all`, so the safety net overlaps the fast path and costs ~0 wall-clock until something actually needs it."
keyTakeaways:
  - 'OCR and sending the pixels are a real tradeoff: OCR gives a tiny text payload but discards layout, diagrams, and code indentation; sending the image preserves everything the model can see but needs a vision-capable model and a bigger (optimized) payload — GeekBye went vision-first with OCR as fallback'
  - 'The dangerous optimization was skipping OCR entirely for vision users: it silently removed the text the backend needed when it fell back from a vision model to a non-vision one — you can optimize away a fast path safely, but not the safety net'
  - "The fix is to gate on the fallback's capability, not the primary's: a second `fallbackVision` flag decides whether to run insurance OCR, and running it in parallel with image optimization (`Promise.all`) makes the safety net free in wall-clock time when it isn't used"
  - 'Optimize the image before you send it to a vision model: `sharp` resizing to 1920px and JPEG quality 80 turns a 3–8 MB PNG into ~200–500 KB, cutting both upload size and image-token cost, because vision models bill by resolution'
  - 'Skipping deferred work creates downstream traps: the OCR-skip introduced a queue deadlock (completeness was defined as "has OCR text," which vision screenshots never would) and a stale capability flag that had to be reset on logout and config-fetch failure'
---

There's a genuine fork in the road for any app that lets you screenshot something and ask an AI about it. You can run **OCR** — extract the text on-device, send a tiny string, and throw the picture away. Or you can **send the pixels** — ship the actual image and let a vision-capable model read it. OCR is cheap and small but blind to everything that isn't text: the layout, the diagram, the indentation of the code, the red squiggle under the error. Pixels preserve all of that but cost more bytes and require a model that can see. GeekBye started on the OCR side of that fork and, across v1.8.6 and v1.8.7, walked over to the pixels side. That switch is the easy part of this story. The interesting part is the optimization that came with it, broke the fallback, and had to be undone.

## The before: read the text, drop the picture

Until this arc, a screenshot became an answer like this: you press the capture shortcut, the client runs OCR — Apple's Vision framework through a Swift `VisionOCR` binary on macOS, a PowerShell script on Windows, both behind one `IOcrEngine` abstraction — and the extracted text gets wrapped into a string and sent to the backend as `screenshotText`. The image itself never left the machine. That's efficient, and for a wall of plain text it's fine. But a screenshot is rarely just text. It's a job posting with a two-column layout, a coding problem with a diagram, an error with a stack trace whose indentation is the clue. OCR flattens all of that into a line-by-line transcription and the model answers a thinner question than the one you asked.

## The switch: send the pixels (v1.8.6)

v1.8.6 added the other path. A new `imageOptimizer.ts`, built on `sharp` and therefore cross-platform, does exactly three things to a raw screenshot:

```js
sharp(pngFilePath).resize({ width: 1920, withoutEnlargement: true }).jpeg({ quality: 80 })
```

Resize to at most 1920px wide (never upscale a smaller capture), convert PNG to JPEG, quality 80. The output is raw base64 with the media type carried in a separate field, and it rides to the backend in two new request fields alongside the old one: `screenshotImage` and `screenshotImageMediaType`, independent of `screenshotText`. The backend can now receive an image, text, or both.

That optimization step is not cosmetic. A raw screenshot PNG is 3–8 MB; the resized JPEG is roughly 200–500 KB. That's an order of magnitude off the upload, and it also cuts the model's bill — vision models tile an image by its resolution and charge by the tile, so capping the width at 1920 and recompressing directly reduces both the bytes on the wire and the image tokens you pay for, with no meaningful loss of legibility for on-screen text.

How does the app know whether the model can even see? The backend advertises it. The `/api/config` response gained an `ai.vision` block keyed by plan, and the client caches a single boolean — `aiSupportsVision` — on every config fetch. When vision is supported, optimize the image; when it isn't, fall back to OCR. So the _capability_ is a backend fact, cached client-side, and the _per-screenshot decision_ to run or skip OCR is made on the client from that flag. The backend still owns which model actually answers.

## The optimization that was too clever

Here's where it gets instructive. The first cut of the vision path ran OCR and image optimization together, in parallel. Then a "perf" commit tightened it into an either/or: if the plan supports vision, optimize the image and **skip OCR entirely**; if it doesn't, run OCR and skip the image work. On paper this is the obvious win — why extract text the vision model doesn't need? The changelog even advertises it: "OCR is skipped when the AI can read the image directly."

The problem is the word _fallback_. The backend doesn't have one model; it has a primary and a fallback, and the fallback might be a **non-vision** model. Picture the sequence: your plan's primary model supports vision, so the client proudly skips OCR and sends only the image. The primary provider hiccups. The backend falls back to a text-only model — which is now handed an image it cannot read and _no text_, because the client optimized the text away. The fast path had quietly deleted the safety net. You can drop a redundant fast path safely; you cannot drop the thing the fast path was a shortcut _around_.

## The fix: gate on the fallback, not the primary

The fix is the part worth stealing. It wasn't to go back to always running OCR — that would throw away the win for the common case. It was to make the skip conditional on a **second** capability flag: does the _fallback_ support vision? The backend's config gained a `fallbackVision` block; the client caches `aiFallbackSupportsVision` next to the first flag. Now the decision has four branches:

- Primary and fallback both see → optimize the image, skip OCR. The fast, common path.
- Primary sees but fallback is blind → optimize the image **and** run OCR, as insurance for the fallback.
- Image optimization throws → OCR as a hard fallback.
- Plan has no vision at all → OCR only.

And the second branch — the insurance case — is where the last move matters. Running OCR _after_ image optimization would add its latency back on top. So the two run together under one `Promise.all`: image optimization and OCR overlap, and the OCR that exists only in case the backend needs it finishes inside the time the image was being optimized anyway. The safety net is free in wall-clock terms whenever it isn't used. That's the whole trick in one sentence: **keep the cheap fallback, gate it on whether your fallback provider actually needs it, and parallelize it so it costs nothing until it saves you.**

## Two smaller traps the skip sprang

Aggressively skipping deferred work has a way of surfacing every place that quietly assumed the work would happen. Two came out in PR review the same week.

**A queue deadlock.** The screenshot queue considered an item "ready" when it had OCR text. Vision screenshots now never got OCR text — by design — so the completeness check `hasIncompleteOCR` waited forever and the queue hung. The fix redefines "ready" as _has an optimized image **or** has OCR text_, which is what the pipeline now actually guarantees.

**A stale capability flag.** `aiSupportsVision` is cached, which means it can go stale in exactly the wrong direction: a user logs out, or a config fetch fails, and the client is left believing it may still send images to a model that no longer supports them. The flag now resets on logout and on config-fetch failure — a cached capability has to have a story for when the thing it describes goes away.

## A word on the numbers

Because honesty is the point of this series: the commit messages carry figures — image optimization "~30ms" versus OCR "~100–150ms," payloads "~200–500 KB vs ~3–8 MB PNG" — and they're plausible, but they are the author's own inline estimates, not the output of a benchmark. There is no measured end-to-end before/after latency anywhere in the history. Treat the speed argument as sound design intent, not a proven result. The payload-size reduction is the one number you can trust, because it falls straight out of resizing and recompressing.

## Three things this release taught us

1. **Sending the pixels beats reading the text — when the model can see.** OCR discards everything a screenshot communicates besides characters. A vision model reads the layout, the diagram, and the indentation, and an optimized JPEG makes that affordable. But it's a capability bet, which is why it needs a fallback.
2. **You can optimize away a fast path, never the safety net.** Skipping OCR for vision users was correct for the primary and wrong for the fallback. The bug wasn't the skip; it was skipping based on the primary's capability when the fallback's was the one that mattered.
3. **A free safety net is a parallelized one.** Gate the insurance work on whether it's actually needed, then run it alongside the fast path so it overlaps. Done right, graceful degradation costs nothing until the day it's the only reason your feature still works.

For the previous chapter in the v1 story, [the two failure modes of a click-through overlay](/blog/the-two-failure-modes-of-a-click-through-overlay) (v1.8.5); and for the whole arc, [the anatomy of shipping software to perfection](/blog/anatomy-of-perfection).