A photo is just a grid of numbers to a computer — no edges, no objects, no meaning. Here's how machine learning turns that grid of numbers into "that's a cat," and where the approach still gets fooled.
The machine learning article in this series explained how a model learns rules from labelled examples rather than being explicitly programmed with them. Computer vision applies that same learning process to images — training a model on huge numbers of labelled pictures until it learns to recognise patterns that correspond to real objects, faces, or scenes. This guide looks at how an image actually becomes numbers a model can learn from, how it recognises what's in a picture, and where this kind of pattern-matching still breaks down.
A digital photo is a grid of tiny squares called pixels, and each pixel is stored as a number, or a set of numbers, representing its colour — typically an amount of red, green, and blue light combined. A computer never "sees" a picture the way a person does; it only ever receives this grid of numbers, and every step of computer vision is really a mathematical operation performed on that grid.
A small black-and-white image might be stored as a 28-by-28 grid of numbers between 0 (black) and 255 (white). "Recognising a face" in that grid means finding a mathematical pattern in those numbers that reliably shows up whenever a face is present — not literally looking at a picture.
Most computer vision systems are built on a type of neural network — introduced in the machine learning article as layers of connected artificial neurons — called a convolutional neural network, or CNN. A CNN scans small patches of an image with filters that each learn to detect one simple pattern, like a horizontal edge, a curve, or a patch of a certain colour.
Early layers in the network detect simple patterns like edges and colours. Each following layer combines the patterns found by the layer before it into something more complex — edges become shapes, shapes become parts like an eye or a wheel, and parts become whole objects like a face or a car — building up recognition in stages rather than all at once.
| Network Depth | What It Typically Detects |
|---|---|
| Early layers | Simple edges, colours, and textures |
| Middle layers | Shapes and combinations of edges, like curves or corners |
| Later layers | Recognisable parts and whole objects, like a face, a wheel, or a word |
"Computer vision" actually covers several distinct tasks, each answering a different question about an image:
A self-driving car's vision system typically needs all three at once: classifying the general scene, detecting and boxing every pedestrian and vehicle, and segmenting the road surface precisely enough to know exactly where it's safe to drive.
Just as the machine learning article described training a spam filter on labelled emails, a computer vision model is trained on a large dataset of images that have already been labelled by humans — a picture of a dog tagged "dog," a photo with a stop sign tagged with a box around it. The model makes a guess, compares it to the correct label, and adjusts its internal parameters slightly to reduce that error, repeating this process across millions of examples until its guesses become reliably accurate.
Medical imaging is one of the highest-stakes applications of this same approach: models trained on large sets of labelled X-rays or scans can flag a likely tumour for a radiologist to review, acting as a second set of eyes rather than a replacement for the doctor's own judgement.
Facial recognition is a specific application of classification: a model learns to represent a face as a set of numbers, then compares that representation against a database to find a match. This raises the same kind of contested questions the AI ethics article in this series covered around bias and privacy — accuracy has historically varied across different demographic groups, and its use for surveillance without consent raises separate concerns from its use to unlock a personal phone.
Because a model is matching statistical patterns rather than genuinely understanding a scene, it can be fooled in ways a person wouldn't be. Adversarial examples are images deliberately altered with tiny, often invisible-to-humans changes that cause a model to misclassify them completely — a stop sign with a few carefully placed stickers has been shown to be misread as a speed limit sign by some systems, despite looking completely normal to a human driver.
Vision models also struggle with unfamiliar contexts: a model trained mostly on photos taken in daylight may perform far worse at night, and one trained mostly on one country's road signs may not generalise well to another's, since it never learned patterns for cases outside its training data.
Beyond self-driving cars and medical imaging, computer vision powers automatic photo tagging and search, quality control on manufacturing lines that spot defective products faster than a human inspector, accessibility tools that describe a scene aloud for someone who is blind or low-vision, and agricultural drones that identify crop disease across a field from the air.
Computer vision doesn't give a machine anything like human sight — it gives it a very good statistical method for matching patterns in grids of numbers to labels it has seen before, at a scale and speed no human could match. That's genuinely powerful for the tasks it's trained on, but it's also exactly why it can fail in ways a sighted person never would: it isn't seeing a scene, it's recognising a pattern, and those two things only look the same until they don't.
10 questions. Select an answer for each, then submit to see your score instantly.