Should LLMs just treat text content as an image?
seangoedecke.com63 points by ingve 6 days ago
63 points by ingve 6 days ago
Although thus isn’t directly related to the idea in the article, I’m reminded that one of the most effective hacks I’ve found for working with ChatGPT has been to attach screen shots of files rather than the files themselves. I’ve noticed the model will almost always pay attention to an image and pull relevant data out of it, but it requires a lot of detailed prompting to get it to reliably pay attention to text and pdf attachments instead of just hallucinating their contents.
Look long enough at literature on any machine learning task, and someone invariably gets the idea to turn the data into an image and do machine learning on that. Sometimes it works out (turning binaries into images and doing malware detection with a CNN surprisingly works), usually it doesn't. Just like in this example the images usually end up as a kludge to fix some deficiency in the prevalent input encoding.
I can certainly believe that images bring certain advantages over text for LLMs: the image representation does contain useful information that we as humans use (like better information hierarchies encoded in text size, boldness, color, saturation and position, not just n levels of markdown headings), letter shapes are already optimized for this kind of encoding, and continuous tokens seem to bring some advantages over discrete ones. But none of these advantages need the roundtrip via images, they merely point to how crude the state of the art of text tokenization is
Doesn't this more or less boil down to OCR scans of books having more privileged information than a plaintext file? In which case a roundtrip won't add anything?
[0] https://web.archive.org/web/20140402025221/http://m.nautil.u...
Exactly. The example the article give of reducing resolution as a form of compression highlights the limitations of the visual-only proposal. Blurring text is a poor form of compression, preserving at most information about paragraph sizes. Summarizing early paragraphs (as context compression does in coding agents) would be much more efficient.
This doesn’t cite the very significant example of DeepVariant (and as of 10/16/25 DeepSomatic) which convert genomic data to images in order to find DNA mutations. This has been done since the late 2010s
https://google.github.io/deepvariant/posts/2020-02-20-lookin...
Andrej Karpathy made an interesting comment on the same paper: https://x.com/karpathy/status/1980397031542989305
> It makes two characters that look identical to the eye look as two completely different tokens internally in the network. A smiling emoji looks like a weird token, not an... actual smiling face, pixels and all
This goes against my limited understanding of how LLMs work — and computers generally for that matter. Isn’t that rendering of a smiling emoji still just a series of bits that need to be interpreted as a smiley face? The similar looking characters point makes more sense to me though assuming it’s something along the lines of recognizing that “S” and “$” are roughly the same thing except for the line down the middle. Still that seems like something that doesn’t come up much and is probably covered by observations made in the training corpus.
All that said, Karpathy knows way more than I will ever know on the subject, and I’m only posting my uninformed take here in hopes somebody will correct me in a way I understand.
You’re reading it backwards. He is not praising that behavior, he is complaining about it. He is saying that bots _should_ parse smiling face emoji’s as smiling face emoji’s, but they don’t do that currently because as text they get passed as gross unicode that has a lot of ambiguity and just happens to ultimately get rendered as a face to end users.
This means that current tokenisers are bad, and something better is needed if text rendering + image input is a better tokeniser.
My understanding is that text tokens are too rigid. The way we read is not to process each character (tokens for LLMs) precociously but to see a word or sometimes a collection of familiar words and make sense of writing. That concept that we understand from written text is really what we read and not letter or words exactly. This is why we can easily read written text with typos. They are just similar enough. By letting LLMs not to be too hung up on exact tokens and "skim" through text we can make them more efficient just like how humans efficiently read.
You miss that we already have ‘context’ when we begin reading something, and that probably enables our fast reading. Maybe there’s a way to give that background setting information to an llm but then we could also just have it read the entire input stream
Google AI mode:
User: cna uto nfeir waht tihs si msieplnig?
Google: Here are top web results for exploring this topic [few websites containing or discussing gibberish in general]
User: what was my previous question?
Google: Your previous question was "cna uto nfeir waht tihs si msieplnig?". I was able to understand this through a combination of contextual understanding and sub-word tokenization, which allowed me to infer the intended meaning despite the numerous typos. The corrected question is: "Can you infer what this is misspelling?".
GPT-5 thinking doesn't struggle with this. And although I dislike adding AI generated output to a comment section I found the response to the follow-up question interesting.
q: c u i t m o t s? a: “Can you infer the meaning of this sentence?”
Language was spoken long before it was written (or so it seems.) This article almost suggests that sound might be a superior input medium over either digital text or images.
I've never worked in that area, but recall reading about how images of spectrograms are often superior inputs to neural nets in comparison to the raw audio data.
Speech to text and text to speech typically operate on the audio spectrogram, specifically the Mel-scale spectrum. This is a filtered spectrogram that decreases the noise in the data. Thus, they are not working on the images of these spectra but the computed values -- each spectral slice will be a matrix row or column of values.
The theory is that vowels and voiced consonants have a fundamental frequency and 5-6 frequencies above that. For vowels the first two frequencies are enough to identify the vowel. For rhotic vowels (r-sounding vowels like American stARt) the 3rd frequency is important.
By converting the audio to the Mel-scale spectrum, it is easier to detect these features. Text to speech using the Mel-spectrum works by modelling and generating these values, which is often easier as the number of parameters is lower and the data is easier to work with [1].
[1] There are other approaches to text to speech such as overlapping short audio segments.
There’s some poor logic in this writeup. Yes, images can contain more information than words, but the extra information an image of a word conveys is usually not relevant to the intent of the communication, at least not for the purposes assumed in this writeup. Ie, pre-converting the text you would have typed into ChatGPT and uploading that as an image instead will not better convey the meaning and intent behind your words.
If it gives better results (something that there’s no evidence presented of), that’d be interesting, but it wouldn’t be because of the larger data size of the uploaded image vs the text.
The amount of video/imagery to make a million tokens vs the amount of text to do the same is a surprisingly low ratio. Did they have the same intuition?
A picture is worth a thousand words
and reading (aka “ocr”) is the fastest way for the brain to process language.
A series of tokens is one-dimensional (a sequence). An image is 2-dimensional. What about 3D/4D/... representation (until we end up with an LLM-dimensional solution ofc).
This isn't exactly true, as tokens live in the embedding space, which is n-dimensional, like 256 or 512 or whatever (so you might see one word, but it's actually an array of a bunch of numbers). With that said, I think it's pretty intuitive that continuous tokens are more efficient than discrete ones, simply due to the fact that the LLM itself is basically a continuous function (with coefficients/parameters ∈ ℝ).
We call an embedding-space n-dimensional, but in this context I would consider it 1-dimensional, as in it's a 1d vector of n values. The terminology just sucks. If we described images the same way we describe embeddings a 2 megapixel image would have to be called 2-million-dimensional (or 8-million-dimensional if we consider rgba to be four separate values)
I would also argue tokens are outside the embedding space, and a large part of the magic of LLMs (and many other neural network types) is the ability to map sequences of rather crude inputs (tokens) into a more meaningful embedding space, and then map from a meaningful embedding space back to tokens we humans understand
There's a new paper from ICCV which basically tries to render every modality as images: https://openaccess.thecvf.com/content/ICCV2025/papers/Hudson...
reminds me of the difference between fasttext and word2vec.
fasttext can learn words it haven't seen before by combining words from ngrams, word2vec can learn better meaning of the whole words, but then missing out on the "unknown words".
image tokens are "text2vec" here, while text tokens are a proxy towards building a text embedding of even unseen before texts.
But does this not miss the "context" that the embeddings of the text tokens carry? An LLM embedding of a text token has a compressed version of the entire preceding set of tokens that came before it in the context. While the image embeddings are just representations of pixel values.
Sort of at the level of word2vec, where the representation of "flies" in "fruit flies like a banana" vs "time flies like an arrow" would be the same.
Wouldn't it be ironic if Comic Sans turned out to be the most efficient font for LLM OCR understanding.
or maybe 3d objects, since that's closer to what real life is and how the brain shaped itself around?
Seems wildly counterintuitive to me frankly.
Even if true though not sure what we’d do with it. The bulk of knowledge available on the internet is text. Aside from maybe YouTube so I guess it could work for world model type things? Understanding physical interactions of objects etc
Trivial to convert text to images to process. But counter-intuitive to me too.
I ϲаn guаrаntее thаt thе ОСR ϲаn't rеаd thіs sеntеnсе ϲоrrесtlу.
What’s correct though? Even as a human, I read that “correctly”. Using weird representations of C doesn’t change the word?
I would even say that OCR can rеаd the sеntеnсе ϲоrrесtlу, while a tokenizer can't.
Yeah OCR would be much more likely to read that sentence the way a human would.
A lot of Cyrillic characters: https://apps.timwhitlock.info/unicode/inspect?s=I+%CF%B2%D0%...
Future headline: "The unreasonable effectiveness of text encoding"
the enshittifiers simply haven't yet weighted image processing fees with potential token charges; once they have, your cost advantage goes bye bye