Small Language Model
- In Turkish
- Küçük Dil Modeli
In short
A small language model (SLM) is a language model with far fewer parameters than the largest LLMs, cheap and fast enough to run on one GPU, a laptop or a phone.
What is a small language model?
A small language model, or SLM, works just like a large language model: it is a transformer trained to predict the next token. The difference is size. There is no official cutoff, but today the name usually covers models from a few hundred million up to roughly 10 or 15 billion parameters, while the largest LLMs have hundreds of billions or more. Examples include Microsoft's Phi, Google's Gemma, Meta's smaller Llama models and the smaller Qwen models.
Small models have improved quickly thanks to more and better-filtered training data, often including synthetic text written by larger models, and to knowledge distillation, where a small student model learns to imitate a large teacher. Microsoft's Phi-3-mini, with 3.8 billion parameters, scored close to GPT-3.5 on common benchmarks in 2024 while being small enough to run on a phone. Quantization shrinks such models further, so one with a few billion parameters fits in a few gigabytes of memory.
Teams choose SLMs for cost, speed and control. A small model responds faster, costs a fraction as much per request and can run on the user's own device, so data never leaves it and the feature keeps working offline. Fine-tuning one for a narrow job, such as sorting support tickets, pulling fields out of invoices or routing requests inside an AI agent, is cheap, and on that single job it often matches a much larger general model. It is like a pocket dictionary next to an encyclopedia: it covers less, but it goes everywhere and opens faster.
SLM vs LLM is a question of scale and trade-offs, not a different technology. A large model knows more, follows complicated instructions better and handles long, multi-step reasoning more reliably; a small one is narrower and makes more mistakes outside the tasks it was tuned for. Many products use both: a small model handles the frequent, simple requests and hands the hard ones to a large model. "Small" also keeps moving: GPT-2, a headline model in 2019, had 1.5 billion parameters, fewer than many of today's SLMs.
Key takeaways
- A small language model works like an LLM but has far fewer parameters, roughly up to 10 or 15 billion.
- There is no official size cutoff, and what counts as small keeps shifting.
- Teams pick SLMs for lower cost, lower latency, privacy and offline use on the device.
- Fine-tuned for one narrow task, a small model can match a much larger general one.
- Large models still win on broad knowledge and complex, multi-step reasoning.
Example
from transformers import pipeline
# About 0.5 billion parameters: small enough for an ordinary laptop CPU
generator = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
messages = [{
"role": "user",
"content": "Label this ticket as billing, bug or other: 'I was charged twice.'",
}]
result = generator(messages, max_new_tokens=10)
print(result[0]["generated_text"][-1]["content"]) # e.g. billingReaders ask
How small is a small language model?
There is no official threshold. Today the term usually means anything from a few hundred million to about 10 or 15 billion parameters, small enough to run on a single GPU, a laptop or, at the lower end, a phone. Some researchers define it by where it can run rather than by a parameter count.
Can a small language model run on a phone?
Yes. Models of 1 to 4 billion parameters, usually quantized to 4 bits, run on recent phones, and both Android and Apple's platforms ship built-in on-device models that apps can call. They handle summarizing, rewriting and classifying text well but are less reliable for open-ended questions.
Should I use an SLM or an LLM?
Start from the task. If it is narrow and repeated, such as classification or extraction, try a small model, ideally fine-tuned, and compare it with a large one on your own evals. If it needs broad knowledge or long reasoning, a large model is usually worth the extra cost.
See also
- LLMAI & Machine Learning, p. 27An LLM is a machine learning model trained on huge amounts of text that generates language by repeatedly predicting the next most likely piece of text.
- Model ParametersAI & Machine Learning, p. 32Model parameters are the internal numbers, such as weights and biases, that a machine learning model learns in training and uses to turn inputs into outputs.
- QuantizationAI & Machine Learning, p. 39Quantization is a technique that shrinks an AI model by storing its parameters in fewer bits, such as 8 or 4 instead of 16, so inference is faster and cheaper.
- Fine-tuningAI & Machine Learning, p. 20Fine-tuning is the process of taking a pretrained machine learning model and training it further on a smaller, specific dataset to adapt it to one task.
- InferenceAI & Machine Learning, p. 26Inference is the stage where a trained machine learning model is used to make predictions or generate output from new data, without changing what it learned.
- Edge ComputingDevOps & Cloud, p. 24Edge computing runs code and processes data close to where users or devices are, instead of in a distant central data center, to reduce latency.
Sources
- Abdin et al.: Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024)arxiv.org(opens in a new tab)
- Belcak et al.: Small Language Models are the Future of Agentic AI (2025)arxiv.org(opens in a new tab)
- Hinton et al.: Distilling the Knowledge in a Neural Network (2015)arxiv.org(opens in a new tab)
Spotted a mistake or something missing on this page?Suggest an edit