Why will small language models become more popular than large language models?
Large language models (LLMs) came to prominence with the release of OpenAI’s ChatGPT . Since then, several companies have also released their own large language models, but more are now leaning toward small language models (SLMs).
SLMs are now becoming more popular with leading AI companies like Open AI, Google, Microsoft, Anthropic, and Meta launching small language models.
These models are more suitable for simple tasks and do not require large resources, so they are expected to dominate the artificial intelligence sector more than large language models. However, large language models will not disappear, but will be used in advanced applications.
Here are more details about SLMs, how they differ from large language models, and why they will shape the future of AI.
First: What are small language models?
A small language model (SLM) is a type of AI model that has few parameters. Like large language models, SLM models can generate text and perform many other tasks. However, these models use fewer datasets for training, have fewer parameters, and require less computational power to train and run than large models.
On the other hand, SLMs focus on core functions and can be integrated into different devices, such as mobile devices. For example, Google's Gemini Nano is a small language model that runs on mobile devices. Because of its small size, Nano can run on the phone itself and be used without an internet connection.
In addition to Google's Nano model, there are many other small language models from leading and emerging AI companies. Most notable are:
- Microsoft Phi-3 model.
- OpenAI's GPT-4o mini model.
- Claude 3 Haiku template by Anthropic.
- Llama 3 model from Meta .
Second: The most prominent differences between small language models and large language models:
The main difference between small language models and large language models is the size of the model, which is measured in terms of the number of parameters. SLMs typically have millions or even a few billion parameters, while large language models have even more, possibly trillions.
For example: The GPT-3 model released in 2020 has 175 billion parameters, the GPT-4 model has about 1.76 trillion parameters, and SLMs such as Microsoft's Phi-3-mini, Phi-3-small, and Phi-3-medium have 3.8, 7, and 14 billion parameters, respectively.
Another difference between SLMs and LLMs is the amount of data used for training; SLMs are trained on smaller amounts of data than LLMs, and this affects the model’s ability to solve complex tasks. This means that LLMs are suitable for solving different types of complex tasks that require advanced reasoning capabilities, while SLMs are suitable for simple tasks.
Third: Why will small language models become more popular than large language models?
In most common use cases for language models, SLMs are better positioned than LLMs to become the mainstream models used by businesses and consumers to perform a wide variety of tasks. While large language models have many advantages that make them capable of solving complex tasks, SLMs will become more popular in the future for the following reasons:
1- Low cost of training and maintenance:
SLMs require less training data than LLMs, making them a more viable option for individuals and small-to-medium businesses with limited training data, limited funding, or both. LLMs require large amounts of training data, which means they require significant computational resources to train and run.
To clarify, OpenAI CEO Sam Altman confirmed that training the GPT-4 model cost the company more than $100 million, while speaking at an event at MIT, according to Wired .
Another example is Meta’s OPT-175B model, which the company says was trained using 992 NVIDIA A100 GPUs, each costing around $10,000, according to CNBC . That means the training cost is around $9 million, not including other expenses like power, salaries, and more.
Given these numbers, it is not feasible for small and medium-sized companies to train large language models (LLMs). SLMs require fewer resources and lower operating costs, making them easier for many companies to develop and use.
2- Response speed:
Response speed is another area where SLMs have an advantage over LLMs due to their smaller size; SLMs have lower latency and are better suited for systems that require fast responses such as voice response systems for digital assistants.
The ability to run these forms on the device also means that your request does not need to go to online servers and back to respond to your query, resulting in faster responses compared to large forms.
3- High accuracy:
Current LLMs are trained using large datasets of internet data, and may not always be accurate, so you shouldn’t trust everything a chatbot says. On the other hand, SLMs are trained using limited, high-quality data, which makes them more accurate than LLMs.
On the other hand, SLMs can be fine-tuned more than LLMs through focused training on specific tasks or domains, leading to better accuracy in those domains than large models.
4- Ability to run on devices:
SLMs require less computing power than LLMs, which means they can be integrated into devices like smartphones and self-driving vehicles. This offers several advantages for both businesses and users. SLMs help keep user data private because they process user requests and data locally rather than sending them to the cloud. This is also important for businesses because they don’t need to run large servers to handle AI tasks.