The DeepSeek model activates only about 37 billion parameters out of its total 600+ billion parameters during inference, compared to models like Llama that activate all parameter. This results in dramatically reduced compute costs for both training and inference.

Others have been using mixture of experts (MoE) but DeepSeek R1 aggressively scaled to the number of experts within the model.

Othe Key efficiency improvements in DeepSeek's architecture include:
Enhanced attention mechanisms with sliding window patterns, optimized key-value caching and multi-head attention.

Advanced position encoding innovations, including rotary position embeddings and dynamic calibration.

A novel routing mechanism that replaces the traditional auxiliary loss with a dynamic bias approach, improving expert utilization and stability.

These innovations have led to a 15-20% improvement in computational efficiency compared to traditional transformer implementations.
Amazon, Microsoft, Google and Meta are still proceeding with large data center buildouts for several reasons:

The surge in AI compute for reasoning and AI agents requires more compute and the increased efficiency enables more value to be delivered. Jevons paradox (economics) occurs when advancement make a resource more efficient to use but the effect is overall demand increases causing total consumption to rise. This was seen with cheaper personal computers meant the demand for computers increased 100 times from tens of millions to billions of units. The top 4 companies plan to spend $310 billion on AI infrastructure and research.

Deepseek came out at prices per million token that was far cheaper than OpenAI but OpenAi and Google Gemini have competitive and even better pricing.

The AI inference price improvements have been consistent but the surprise from Deepseek is that this latest push was not by OpenAI or Meta.