Autocorrect, spam filters, and translation apps all rely on the same foundation as large language models — just applied to smaller, more specific problems. Here's how a computer turns human language into something it can actually work with.
The large language model article in this series covered chatbots built to generate open-ended conversation. Natural language processing, or NLP, is the broader field that chatbots belong to — the study of getting computers to work with human language at all, whether that means generating a full response or something much narrower, like detecting whether an email is spam or translating a sentence into another language. This guide covers the foundational ideas that make any of that possible, several years before today's large chatbots existed.
A computer, at its core, only processes numbers — a point covered in the "How Computers Work" article in this series. Human language is made of words, grammar, and meaning that don't naturally exist as numbers at all. Every NLP technique starts by solving some version of the same problem: converting text into a numerical form a model can actually process, while preserving as much of its meaning as possible.
Even the simplest NLP task, like checking whether a word is spelled correctly, requires text to first be broken into a form a computer can compare against something, rather than being read the way a person reads a page.
The first step in almost every NLP system is tokenization — splitting text into smaller units called tokens, which might be whole words, parts of words, or even individual characters, depending on the system. This is the same basic concept the LLM article in this series described for how a language model reads and generates text, just applied more broadly across many other NLP tasks beyond chatbots.
Once text is tokenized, each token needs to become a number, or more precisely a list of numbers, in a way that captures something about its meaning — this is called a word embedding. Words used in similar contexts end up with similar embeddings, which is how a model can tell that "happy" and "joyful" are related concepts, without ever being explicitly told so, purely from patterns in how those words tend to appear near similar other words across huge amounts of text.
| Step | What Happens |
|---|---|
| Tokenization | Text is split into smaller units, like words or word-pieces |
| Embedding | Each token is converted into a list of numbers capturing something about its meaning |
| Processing | A model analyses those numbers to perform a specific task |
Every one of these relies on the same tokenize-then-embed foundation, just trained toward a much narrower, specific goal than generating open-ended conversation.
Email providers run spam detection as an NLP classification task continuously in the background, analysing incoming text for patterns statistically associated with spam — certain phrases, formatting, and sender characteristics — and routing flagged messages away from the main inbox before a person ever sees them.
Human language is full of ambiguity that people resolve instantly using context, but which is a real technical challenge for an NLP system. The word "bank" means something completely different in "river bank" versus "savings bank," and a sentence like "I saw the man with the telescope" can genuinely mean either that the telescope belonged to the observer or to the man observed. Modern systems handle this far better than older ones by considering surrounding context rather than looking at a single word in isolation, but ambiguity remains one of the core reasons language processing is harder than it might first appear.
Large language models are, in a real sense, NLP taken to a much larger scale — the same tokenization and embedding foundation, combined with a vastly larger model and training dataset, aimed at open-ended generation rather than one narrow task. Understanding tokenization and embeddings from this article is exactly what makes the LLM article's explanation of "predicting the next token" concrete rather than abstract — it's the same underlying machinery, just scaled up dramatically.
Long before today's chatbots, NLP was already quietly running behind autocorrect, spam filters, and translation apps — narrower tools solving the same core problem of turning human language into something a computer can process. Recognising that shared foundation makes it clear that a modern chatbot isn't a completely different kind of technology from these everyday tools; it's the same idea, built bigger and aimed at a much more open-ended goal.
10 questions. Select an answer for each, then submit to see your score instantly.