How AI works
How does ChatGPT work?
ChatGPT predicts the next word, one piece at a time, using a large language model trained on huge amounts of text and then tuned with human feedback. Here is how that works, in plain English.
Short answer
ChatGPT runs on a large language model (): a neural network trained on a huge amount of text to predict the next , a word or piece of a word. It writes every reply one token at a time, each time picking a likely next piece. Extra training with human feedback taught it to follow instructions and answer helpfully, which is why it feels like a conversation.
It predicts the next word, very well
Type "The capital of France is" and ask what comes next. Almost anyone says "Paris". ChatGPT does something similar. It reads your text and works out how likely every possible next piece of text is, then picks one. Then it does it again, with that new piece added, until the answer is done.
Those pieces are called tokens. A token is often a whole short word, sometimes part of a longer one: "unbelievable" might be split into "un", "believ" and "able". You can see how this works in the lesson on tokenisation.
The model doesn't always take the single most likely token. A setting called temperature adds a little randomness, which is why you can ask the same question twice and get two different answers.
Step 1: pretraining on a huge library of text
The model starts as a giant neural network with billions of adjustable numbers called . At first it predicts nonsense.
During pretraining it reads a vast amount of text from books, websites, code and other sources. For each snippet it guesses the next token, checks the real answer, and nudges its parameters so the right token becomes a little more likely next time. That nudging is gradient descent, repeated trillions of times on thousands of chips.
After pretraining the model is a powerful autocomplete. Ask it a question and it might continue with more questions, because that's what often follows a question on the web. It knows a lot, but it isn't yet an assistant.
Step 2: teaching it to be a helpful assistant
To turn autocomplete into ChatGPT, the model gets more training:
- Supervised fine-tuning. People write example conversations showing good answers, and the model is trained on them. It learns the shape of a helpful reply.
- Reinforcement learning from human feedback (RLHF). The model writes several answers, people rank them, and a separate model learns to predict which answers people prefer. The is then trained to score well on that. OpenAI described this method in its 2022 InstructGPT paper.
This stage is also where models learn to refuse harmful requests, admit uncertainty and follow a consistent style. Newer "reasoning" models get extra training that rewards working through a problem step by step before answering. Read more in the lesson on training LLMs.
The transformer: how it keeps track of meaning
The "T" in GPT stands for transformer, a design introduced by Google researchers in the 2017 paper "Attention Is All You Need". Its key idea is attention: when the model looks at a word, it can weigh every other word in the text to work out what that word means here.
In "The cat sat on the mat because it was tired", attention helps the model link "it" to "the cat", not "the mat". Stack dozens of these attention layers and the model can follow long, complicated text. The transformers lesson walks through it with pictures.
What happens when you send a message
- Your message, and the earlier messages in the chat, are turned into tokens.
- Everything fits in the model's context window, the amount of text it can look at once. Very long chats may push early messages out.
- The model predicts the reply one token at a time and streams it to your screen, which is why answers appear word by word.
- Some versions can also use tools: searching the web, running code or reading a file you upload, then using the results in the answer.
The model doesn't learn from your chat as you talk. Its parameters stay fixed while it answers. Some services may use conversations to train future models unless you turn that off, so check the privacy settings of the app you use.
Why ChatGPT sometimes gets things wrong
Because it predicts likely text rather than looking facts up, ChatGPT can produce fluent, confident answers that are false. This is called hallucination, and we explain it fully in why AI hallucinates.
Its knowledge also stops at a training cut-off date, unless it searches the web. And it can be swayed by how a question is phrased. Treat it like a fast, well-read helper whose work you still check.
Do Claude, Gemini and other chatbots work the same way?
Broadly, yes. Claude, Gemini, Llama, Mistral and most other chatbots are large language models built on the , pretrained on large amounts of text and then tuned to be helpful. They differ in their , size, and safety work, and in the tools they can use, which is why they have different strengths. The lesson meet the LLMs compares them gently.
Key takeaways
- At its core, ChatGPT predicts the next token, over and over, very well.
- It learns language and facts in pretraining, then learns to be a helpful assistant from example chats and human feedback.
- The transformer and its attention mechanism let it use the whole conversation when choosing each word.
- It can sound sure and still be wrong, so check anything that matters.
Learn it properly, free
- Meet the chatbots: LLMsBeginner · 6 min
- How text becomes tokensIntermediate · 7 min
- Attention is all you needAdvanced · 8 min
- How LLMs are trainedAdvanced · 7 min
Terms in this post
Questions
What does GPT stand for?
GPT stands for Generative Pre-trained Transformer: generative because it creates text, pre-trained because it first learns from a large amount of text, and transformer after the neural network design it uses.
Does ChatGPT understand what it says?
Researchers disagree on the right word. ChatGPT has no experiences or intentions, but to predict text well it builds internal patterns that capture a lot of meaning and reasoning. It can explain ideas well and still make mistakes no person with real understanding would make.
Does ChatGPT search the internet?
Not by default in every version. The model itself answers from what it learned in training. Some versions can run a web search and use the results, and they usually show links to the pages they used.
Does ChatGPT learn from my conversations?
Not during the conversation: the model's parameters don't change while it answers you. Some providers may use past conversations to train future models unless you opt out in the settings, so check before sharing anything private.
Sources and further reading
- [1hr Talk] Intro to Large Language Models · Andrej Karpathy · Free video
- The Illustrated Transformer · Jay Alammar
- Attention Is All You Need · Vaswani et al., 2017
- Training language models to follow instructions with human feedback · Ouyang et al. (OpenAI), 2022