DeepseekAI Q&A's: Expert Insights and Answers
DeepSeek AI is capturing global attention across the technology landscape. Celebrated as a "Sputnik moment" in artificial intelligence, DeepSeek introduces a bold new perspective that challenges the high-cost, closed-source models prevalent in Silicon Valley by offering an accessible, efficient, and open alternative. In this blog post, we delve into what DeepSeek AI is, how it works, and why it is igniting excitement and thoughtful discussion in the tech industry.
A New Contender in the AI Arena
DeepSeek AI is a pioneering Chinese artificial intelligence research company established in July 2023 by Liang Wenfeng, a hedge fund entrepreneur driven by a passion for technology and innovation. Headquartered in Hangzhou, Zhejiang, and backed by the Chinese hedge fund High-Flyer, DeepSeek sets itself apart from many Western AI companies by embracing an open-source philosophy. Unlike proprietary approaches that keep training data and model architectures secret, DeepSeek releases its models with open weights under the MIT license, empowering anyone to download, use, and modify the code. This approach opens the door to expansive collaboration and rapid innovation.
What questions does DeepSeek not answer?
While DeepSeek AI is revolutionizing accessibility through its open-source framework, there remain important areas it does not address. The profound philosophical implications of AI—such as developing ethical decision-making frameworks—venture beyond DeepSeek’s primary technological focus.
Although DeepSeek offers the transparency that many developers value, it does not define which ethical guidelines should govern AI decision-making in sensitive sectors such as healthcare and judicial systems.
DeepSeek’s core mission is technological advancement; thus, topics like the nuanced societal impacts of AI (for example, the employment challenges brought by automation) require broader collaboration beyond the AI community.
Another aspect that remains unexplored is its applicability in culturally nuanced contexts beyond China’s socio-economic and cultural environment, leaving room for further global integration of AI technologies.
Additionally, while DeepSeek makes strategic advances, it does not exhaustively examine the environmental factors associated with large-scale AI deployment, such as energy consumption and sustainability initiatives.
How Rich is DeepSeek's Founder?
Hurun Global Rich List 2025
Liang Wenfeng is valued at $4 billion, placing him among China’s wealthiest individuals.
Forbes 2025
His net worth is estimated at $1 billion, reflecting his stake in both DeepSeek and the hedge fund High-Flyer.
Forbes Daily
A separate report cites his wealth at $4 billion, spotlighting DeepSeek’s market impact and valuation.
Drivers of Wealth
DeepSeek’s Valuation
DeepSeek’s accessible open-source AI models, including R1, have disrupted global tech markets, leading to a $108 billion loss for competitors like Nvidia. Analysts estimate DeepSeek’s valuation at over $1 billion, with Liang retaining approximately 84% of the company.
High-Flyer Hedge Fund
Co-founded by Liang in 2015, High-Flyer manages a $13.79 billion portfolio using AI-driven quantitative strategies. This hedge fund played a crucial role in providing the necessary capital for DeepSeek’s development.
Market Impact
DeepSeek’s cost-effective AI models—priced at a mere $.01 per inference compared to OpenAI’s $.12—have rekindled investor enthusiasm in Chinese technology, contributing to a 19.8% boost on the Hang Seng Index in 2024.
Contextual Factors
AI Boom in China
The Chinese government’s supportive stance on AI innovation, coupled with reduced regulatory oversight for private tech firms, has been key to amplifying Liang’s success.
Global Competition
DeepSeek’s emergence has placed pressure on U.S. tech giants, even as Nvidia’s founder Jensen Huang faced a staggering $128 billion drop in wealth.
Risks and Challenges
National Security Scrutiny
DeepSeek’s AI models have naturally raised national security questions.
Valuation Volatility: DeepSeek’s pre-revenue status and its reliance on open-source licensing may influence long-term profitability.
Final Insight: Liang Wenfeng’s wealth underscores a unique blend of AI innovation, quantitative finance expertise, and impeccable timing. Even with valuation discrepancies, his significant net worth reflects the transformative power of China’s AI ecosystem. Investors are encouraged to keep a close watch on DeepSeek’s journey toward commercialization alongside geopolitical developments.
When was DeepSeekAI launched?
DeepSeekAI was officially launched in July 2023, marking the beginning of a new era in AI innovation. Its commitment to an open-source philosophy uniquely positions it in a landscape crowded by proprietary models and highlights the power of collective progress over restrictive practices.
Pioneering New Frontiers in AI, DeepSeekAI is not just another AI entity—it is a turning point that showcases China’s proactive steps toward democratizing AI technologies. With its open-minded ideology, DeepSeekAI invites researchers, developers, and innovators worldwide to engage in collaborative progress. This approach accelerates development cycles and broadens the application of AI across diverse fields.
The Implications of its Launch
DeepSeekAI’s debut has challenged the status quo, fostering a rich dialogue about the future of artificial intelligence. It represents the potential for a globally connected AI ecosystem where inclusivity and accessibility are pivotal. This launch stands as a beacon of inspiration, reminding us of the extraordinary achievements possible when ambitious goals are pursued with transparency and ethical commitment.
A Call to Innovators and Leaders
Reflecting on DeepSeekAI’s remarkable inception, it is clear that we have entered a transformative journey. The company invites leaders, visionaries, and creators to join in building a future where AI serves as a catalyst for positive societal change. DeepSeekAI is a story of promise and possibility, inspiring a generation of innovators to dream boldly and push boundaries further than ever before.
In summary, DeepSeekAI's launch is not merely a milestone in AI history—it is a beacon that signals a new paradigm. It inspires us to envision and create a smarter, more interconnected world. Let us confidently embrace the opportunities ahead as we collectively shape the technologies of tomorrow.
Why is DeepSeekAI unique?
DeepSeekAI embodies a significant transformative shift in AI innovation.
It distinguishes itself with an unwavering commitment to open-source, boldly challenging conventional proprietary practices. By making its AI models publicly accessible and modifiable, DeepSeekAI fosters a culture of collaborative innovation. This collective approach accelerates progress by engaging a broad community of developers and researchers in meaningful, transformative work.
How does DeepSeekAI work?
DeepSeekAI’s uniqueness stems from its steadfast open-source philosophy, disrupting the conventional proprietary model ecosystem. By providing unrestricted access to its models, DeepSeekAI democratizes AI development. This inclusivity not only accelerates innovation but also encourages a broad spectrum of contributions from developers and researchers globally.
Some sources estimate its value at 150 Billion USD, while others suggest 13 Billion USD.
This strategic model stands as a catalyst for global technological inclusion.
Moreover, DeepSeekAI is perfectly positioned to harness both its advanced tools and the enriched resources of its community. This synergy fuels a new wave of AI applications that are versatile across various domains. It exemplifies the potential of AI to benefit everyone, encouraging a collaborative future.
DeepSeekAI does more than forge a new path—it inspires a vision grounded in collective progress. By embracing an open-source strategy, it sparks a renaissance built on transparency and accessibility, promising to redefine future interactions with technology worldwide.
What is DeepSeaKAI?
DeepSeekAI represents a remarkable convergence of advanced technology and open-source accessibility, powered by an unyielding drive for innovation and collaboration. At its core lies a sophisticated machine learning framework designed for leading-edge AI research, employing deep learning algorithms to process and analyze vast datasets.
Within DeepSeekAI, neural networks form the foundation, training on diverse data inputs to enhance precision and adapt dynamically to various tasks. The innovative architecture ensures these networks remain flexible and scalable, continually incorporating new insights and performance improvements. It is an inspiring vision of possibility—one that heralds a future where collective innovation propels technological progress at unparalleled speeds.
Cost-Efficiency Through Innovation
One of DeepSeek’s most admirable achievements is delivering state-of-the-art AI performance at a fraction of the cost. For instance, DeepSeek claims that training its flagship model, DeepSeek-V3, cost roughly US$6 million compared to the US$100 million or more typically invested by competitors such as OpenAI on models like GPT-4. This dramatic reduction in expense is made possible by utilizing innovative architectures and optimization techniques, including:
- Mixture-of-Experts (MoE): DeepSeek’s models selectively activate a subset of parameters (for example, 37 billion out of 671 billion total in V3) for each token, significantly reducing computational demands.
- Multi-Head Latent Attention (MLA): This novel strategy compresses the model’s key-value cache into a latent vector, ensuring efficient inference without compromising performance.
- Reinforcement Learning (RL) for Reasoning: With the R1 series, DeepSeek pioneers reinforcement learning techniques that boost reasoning capabilities without the high costs associated with extensive supervised fine-tuning.
Open-Source Advantages
By releasing its AI models as open source, DeepSeek democratizes access to high-performance language models. This openness invites developers around the globe to experiment, modify, and enhance these models, accelerating the pace of AI innovation remarkably. This open-weight strategy offers a refreshing contrast to the “black-box” approaches common among many U.S.-based models, where the underlying architectures and training methods remain concealed.
citeturnsearch16
DeepSeek's Competitive Edge
Disrupting Silicon Valley's Business Model
DeepSeek’s breakthrough has sent ripples throughout the tech industry. Its cost-effectiveness has excited developers and unsettled established market players alike. For example, major tech stocks—especially those of Nvidia—experienced a notable downturn after DeepSeek’s emergence, prompting investors to rethink the high pricing of proprietary models when an open-source alternative offers such compelling value.
citeturnnews50
A "Sputnik Moment" for AI
Venture capitalist Marc Andreessen aptly described DeepSeek’s influence as an "AI Sputnik moment." Just as the launch of Sputnik in 1957 ushered in a new era in space exploration and spurred U.S. innovation, DeepSeek’s rapid progress heralds a future where resource limitations no longer hinder breakthrough AI advancements. This transformation is sparking meaningful debates among industry leaders and policymakers regarding global AI leadership and the role of export controls on advanced semiconductors.
Market Impact & Global Reaction
Wall Street and Silicon Valley Shake-Up
DeepSeek’s disruptive market entry has led to dramatic shifts in investor sentiment. On the very day its chatbot application soared to the top of Apple’s App Store in the United States, shares of Nvidia and other tech giants experienced sharp declines, erasing billions in market value. Analysts are now rethinking long-held business models based on costly, closed-source AI infrastructures.
citeturnnews51
Geopolitical Implications
The rise of DeepSeek AI reflects broader geopolitical dynamics. U.S. export controls on advanced semiconductors—designed to slow China’s AI progress—have prompted Chinese researchers to innovate using less powerful hardware (like Nvidia’s H800 chips). This resilience not only underscores the strength of China’s tech ecosystem but also challenges the conventional belief that only high-end chips can deliver the most advanced AI performance. Consequently, policymakers worldwide are re-examining strategies and forming new alliances in the AI domain.
citeturnnews52
Open-Source, Censorship, and Ethical Considerations
The Open-Source Promise
For many, the greatest appeal of DeepSeek lies in its open-source model. By lowering barriers to entry, it nurtures a collaborative and innovative AI landscape. This approach promises accelerated advances in AI that are no longer the exclusive domain of a few billion-dollar corporations, making them accessible to startups, researchers, and even individual enthusiasts. The potential for collective innovation is immense, resonating with the early ideals that shaped organizations like the original OpenAI.
citeturnnews54
Censorship and Data Security Concerns
Of course, openness comes with its own set of challenges. DeepSeek’s models are subject to self-censorship in order to comply with Chinese regulations. This means topics considered sensitive—such as the Tiananmen Square massacre, human rights issues, or controversies related to Taiwan—might be automatically filtered or moderated. For businesses and developers outside China, this raises important questions about data security and content neutrality. Some critics worry that such censorship could restrict the model’s utility in global applications and potentially impact privacy and free access to information.
Why DeepSeek AI Could Be a Game Changer for Your Business
For businesses eager to harness the potential of AI without incurring astronomical costs, DeepSeek offers a compelling value proposition. Consider these remarkable advantages:
- 1. Cost Savings: With training expenses estimated at just a fraction of those for comparable U.S. models, DeepSeek can significantly reduce your AI "utility bills."
- 2. Customization: The open-source nature of DeepSeek allows you to fine-tune and optimize the model to meet your specific needs, whether that be customer service, coding assistance, or advanced data analysis.
- 3. Rapid Innovation: As more developers and companies adopt and enhance DeepSeek’s models, you stand to benefit from an ever-evolving ecosystem that is continually at the forefront of AI breakthroughs.
- 4. Competitive Edge: Integrating a highly skilled, cost-effective AI solution could provide your business with a strategic advantage, especially in industries where quick data processing and automation are vital.
- 5. Global Perspective: While concerns about censorship remain, many companies are successfully adapting and localizing the outputs to meet diverse market needs.
Table of Contents
- Introduction
- Model Summary
- Model Downloads
- Evaluation Results
- Chat Website & API Platform
- How to Run Locally
- License
- Citation
- Contact
1. Introduction
We introduce DeepSeek-V3, a highly skilled Mixture-of-Experts (MoE) language model featuring 671B total parameters with an activation of 37B for each token. With a focus on efficient inference and cost-effective training, DeepSeek-V3 implements Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, building on the success of DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and introduces a multi-token prediction training objective that drives greater performance. Pre-trained on a diverse set of 14.8 trillion high-quality tokens, the model undergoes further refinement through Supervised Fine-Tuning and Reinforcement Learning stages, unlocking its full potential. Comprehensive evaluations indicate that DeepSeek-V3 outperforms other open-source models and reaches performance levels comparable to leading closed-source models. Notably, it achieved all this using only 2.788M H800 GPU hours during training, with a remarkably stable process devoid of irrecoverable loss spikes or rollbacks.
2. Model Summary
Architecture: Innovative Load Balancing Strategy and Training Objective
- Building on the efficient design of DeepSeek-V2, we introduce an auxiliary-loss-free strategy for load balancing that minimizes traditional performance trade-offs.
- We explore a Multi-Token Prediction (MTP) objective, demonstrating its value to model performance. Additionally, it serves as a tool for speculative decoding, enhancing inference speed.
Pre-Training: Towards Ultimate Training Efficiency
- We have developed an FP8 mixed precision training framework, validating for the first time the feasibility and effectiveness of FP8 training on an extremely large-scale model.
- By co-designing algorithms, frameworks, and hardware, we overcame the communication bottleneck inherent in cross-node MoE training, nearly achieving full computation-communication overlap. This breakthrough significantly enhances our training efficiency and brings down training costs, allowing model scale-up without incurring extra overhead.
- At an economical cost of only 2.664M H800 GPU hours, we pre-trained DeepSeek-V3 on 14.8T tokens to produce the strongest open-source base model available today. The subsequent training phases required a mere .1M GPU hours.
Post-Training: Knowledge Distillation from DeepSeek-R1
- We have developed an innovative methodology for distilling reasoning capabilities from the long Chain-of-Thought (CoT) model, specifically drawing from one of the DeepSeek R1 series models, into the standard LLM framework of DeepSeek-V3. This pipeline elegantly integrates the verification and reflection patterns of R1, significantly enhancing DeepSeek-V3’s reasoning performance while maintaining control over output style and length.
3. Model Downloads
| Model | #Total Params | #Activated Params | Context Length | Download | | :--- | :--- | :--- | :--- | :--- | | DeepSeek-V3-Base | 671B | 37B | 128K | 🤗 Hugging Face | | DeepSeek-V3 | 671B | 37B | 128K | 🤗 Hugging Face |
Note
The total size of DeepSeek-V3 models on Hugging Face is 685B, which includes 671B for the main model weights and an additional 14B for the Multi-Token Prediction (MTP) Module weights.
To ensure optimal performance and flexibility, we have collaborated with hardware vendors and open-source communities to offer multiple deployment options locally. Instructions for local setup can be found in Section 6: How to Run Locally.
For those interested in further technical details, please explore README_WEIGHTS.md for comprehensive insights on the main model weights and the MTP Modules. Note that MTP support is actively evolving, and we welcome community input.
4. Evaluation Results
Base Model
Standard Benchmarks
| | Benchmark (Metric) | # Shots | DeepSeek-V2 | Qwen2.5 72B | LLaMA3.1 405B | DeepSeek-V3 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | | Architecture | - | MoE | Dense | Dense | MoE | | | # Activated Params | - | 21B | 72B | 405B | 37B | | | # Total Params | - | 236B | 72B | 405B | 671B | | English | Pile-test (BPB) | - | .606 | .638 | .542 | .548 | | | BBH (EM) | 3-shot | 78.8 | 79.8 | 82.9 | 87.5 | | | MMLU (Acc.) | 5-shot | 78.4 | 85. | 84.4 | 87.1 | | | MMLU-Redux (Acc.) | 5-shot | 75.6 | 83.2 | 81.3 | 86.2 | | | MMLU-Pro (Acc.) | 5-shot | 51.4 | 58.3 | 52.8 | 64.4 | | | DROP (F1) | 3-shot | 80.4 | 80.6 | 86. | 89. | | | ARC-Easy (Acc.) | 25-shot | 97.6 | 98.4 | 98.4 | 98.9 | | | ARC-Challenge (Acc.) | 25-shot | 92.2 | 94.5 | 95.3 | 95.3 | | | HellaSwag (Acc.) | 10-shot | 87.1 | 84.8 | 89.2 | 88.9 | | | PIQA (Acc.) | -shot | 83.9 | 82.6 | 85.9 | 84.7 | | | WinoGrande (Acc.) | 5-shot | 86.3 | 82.3 | 85.2 | 84.9 | | | RACE-Middle (Acc.) | 5-shot | 73.1 | 68.1 | 74.2 | 67.1 | | | RACE-High (Acc.) | 5-shot | 52.6 | 50.3 | 56.8 | 51.3 | | | TriviaQA (EM) | 5-shot | 80. | 71.9 | 82.7 | 82.9 | | | NaturalQuestions (EM) | 5-shot | 38.6 | 33.2 | 41.5 | 40. | | | AGIEval (Acc.) | -shot | 57.5 | 75.8 | 60.6 | 79.6 | | Code | HumanEval (Pass@1) | -shot | 43.3 | 53. | 54.9 | 65.2 | | | MBPP (Pass@1) | 3-shot | 65. | 72.6 | 68.4 | 75.4 | | | LiveCodeBench-Base (Pass@1) | 3-shot | 11.6 | 12.9 | 15.5 | 19.4 | | | CRUXEval-I (Acc.) | 2-shot | 52.5 | 59.1 | 58.5 | 67.3 | | | CRUXEval-O (Acc.) | 2-shot | 49.8 | 59.9 | 59.9 | 69.8 | | Math | GSM8K (EM) | 8-shot | 81.6 | 88.3 | 83.5 | 89.3 | | | MATH (EM) | 4-shot | 43.4 | 54.4 | 49. | 61.6 | | | MGSM (EM) | 8-shot | 63.6 | 76.2 | 69.9 | 79.8 | | | CMath (EM) | 3-shot | 78.7 | 84.5 | 77.3 | 90.7 | | Chinese | CLUEWSC (EM) | 5-shot | 82. | 82.5 | 83. | 82.7 | | | C-Eval (Acc.) | 5-shot | 81.4 | 89.2 | 72.5 | 90.1 | | | CMMLU (Acc.) | 5-shot | 84. | 89.5 | 73.7 | 88.8 | | | CMRC (EM) | 1-shot | 77.4 | 75.8 | 76. | 76.3 | | | C3 (Acc.) | -shot | 77.4 | 76.7 | 79.7 | 78.6 | | | CCPM (Acc.) | -shot | 93. | 88.5 | 78.6 | 92. | | Multilingual | MMMLU-non-English (Acc.) | 5-shot | 64. | 74.8 | 73.8 | 79.4 |
Note
Best results are highlighted in bold. A gap of .3 or less signals comparable performance. DeepSeek-V3 achieves superior performance on most benchmarks, notably in math and code tasks. For additional evaluation insights, please refer to our paper.
Context Window
DeepSeek-V3 consistently performs across varying context window lengths up to 128K, as demonstrated by the Needle In A Haystack (NIAH) tests.
Chat Model
Standard Benchmarks (Models larger than 67B)
| | Benchmark (Metric) | DeepSeek V2-0506 | DeepSeek V2.5-0905 | Qwen2.5 72B-Inst. | Llama3.1 405B-Inst. | Claude-3.5-Sonnet-1022 | GPT-4o 0513 | DeepSeek V3 | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | | Architecture | MoE | MoE | Dense | Dense | - | - | MoE | | | Activated Params | 21B | 21B | 72B | 405B | - | - | 37B | | | Total Params | 236B | 236B | 72B | 405B | - | - | 671B | | | English | MMLU (EM) | 78.2 | 80.6 | 85.3 | 88.6 | 88.3 | 87.2 | 88.5 | | | MMLU-Redux (EM) | 77.9 | 80.3 | 85.6 | 86.2 | 88.9 | 88. | 89.1 | | | MMLU-Pro (EM) | 58.5 | 66.2 | 71.6 | 73.3 | 78. | 72.6 | 75.9 | | | DROP (3-shot F1) | 83. | 87.8 | 76.7 | 88.7 | 88.3 | 83.7 | 91.6 | | | IF-Eval (Prompt Strict) | 57.7 | 80.6 | 84.1 | 86. | 86.5 | 84.3 | 86.1 | | | GPQA-Diamond (Pass@1) | 35.3 | 41.3 | 49. | 51.1 | 65. | 49.9 | 59.1 | | | SimpleQA (Correct) | 9. | 10.2 | 9.1 | 17.1 | 28.4 | 38.2 | 24.9 | | | FRAMES (Acc.) | 66.9 | 65.4 | 69.8 | 70. | 72.5 | 80.5 | 73.3 | | | LongBench v2 (Acc.) | 31.6 | 35.4 | 39.4 | 36.1 | 41. | 48.1 | 48.7 | | | Code | HumanEval-Mul (Pass@1) | 69.3 | 77.4 | 77.3 | 77.2 | 81.7 | 80.5 | 82.6 | | | LiveCodeBench (Pass@1-COT) | 18.8 | 29.2 | 31.1 | 28.4 | 36.3 | 33.4 | 40.5 | | | LiveCodeBench (Pass@1) | 20.3 | 28.4 | 28.7 | 30.1 | 32.8 | 34.2 | 37.6 | | | Codeforces (Percentile) | 17.5 | 35.6 | 24.8 | 25.3 | 20.3 | 23.6 | 51.6 | | | SWE Verified (Resolved) | - | 22.6 | 23.8 | 24.5 | 50.8 | 38.8 | 42. | | | Aider-Edit (Acc.) | 60.3 | 71.6 | 65.4 | 63.9 | 84.2 | 72.9 | 79.7 | | | Aider-Polyglot (Acc.) | - | 18.2 | 7.6 | 5.8 | 45.3 | 16. | 49.6 | | | Math | AIME 2024 (Pass@1) | 4.6 | 16.7 | 23.3 | 23.3 | 16. | 9.3 | 39.2 | | | MATH-500 (EM) | 56.3 | 74.7 | 80. | 73.8 | 78.3 | 74.6 | 90.2 | | | CNMO 2024 (Pass@1) | 2.8 | 10.8 | 15.9 | 6.8 | 13.1 | 10.8 | 43.2 | | | Chinese | CLUEWSC (EM) | 89.9 | 90.4 | 91.4 | 84.7 | 85.4 | 87.9 | 90.9 | | | C-Eval (EM) | 78.6 | 79.5 | 86.1 | 61.5 | 76.7 | 76. | 86.5 | | | C-SimpleQA (Correct) | 48.5 | 54.1 | 48.4 | 50.4 | 51.3 | 59.3 | 64.8 |
Note
All models were evaluated with the output length limited to 8K. Benchmarks with fewer than 100 samples were tested multiple times with varying temperature settings to yield highly skilled final results. DeepSeek-V3 stands as the best-performing open-source model while also demonstrating competitive performance against leading closed-source models.
Open Ended Generation Evaluation
| Model | Arena-Hard | AlpacaEval 2. | |---|---|---| | DeepSeek-V2.5-0905 | 76.2 | 50.5 | | Qwen2.5-72B-Instruct | 81.2 | 49.1 | | LLaMA-3.1 405B | 69.3 | 40.5 | | GPT-4o-0513 | 80.4 | 51.1 | | Claude-Sonnet-3.5-1022 | 85.2 | 52. | | DeepSeek-V3 | 85.5 | 70. |
Note
These results are from English open-ended conversation evaluations. For AlpacaEval 2., we used the length-controlled win rate as the metric.
5. Chat Website & API Platform
You can interact with DeepSeek-V3 via DeepSeekAI’s official website: chat.deepseek.com
Additionally, we provide an OpenAI-Compatible API at DeepSeek Platform: platform.deepseek.com
6. How to Run Locally
DeepSeek-V3 can be deployed locally using a variety of hardware and open-source community software:
- DeepSeek-Infer Demo: A simple and lightweight demo supports FP8 and BF16 inference.
- SGLang: Full support for the DeepSeek-V3 model in both BF16 and FP8 inference modes is available, with Multi-Token Prediction support coming soon.
- LMDeploy: This tool enables efficient FP8 and BF16 inference for both local and cloud deployments.
- TensorRT-LLM: This framework currently supports BF16 inference and INT4/8 quantization, with FP8 support forthcoming.
- vLLM: DeepSeek-V3 is supported with FP8 and BF16 modes for tensor and pipeline parallelism.
- AMD GPU: Run DeepSeek-V3 on AMD GPUs via SGLang using either BF16 or FP8 modes.
- Huawei Ascend NPU: DeepSeek-V3 is also compatible with Huawei Ascend devices.
Since our framework natively adopts FP8 training, we provide FP8 weights exclusively. If you require BF16 weights for experimentation, you can use the provided conversion script to transform the weights.
6.2 Inference with SGLang (recommended)
SGLang currently supports MLA optimizations, DP Attention, FP8 (W8A8), FP8 KV Cache, and Torch Compile. These features deliver outstanding latency and throughput among open-source frameworks.
Notably, SGLang v.4.1 fully supports running DeepSeek-V3 on both NVIDIA and AMD GPUs, making it a highly versatile and highly skilled solution.
SGLang also supports multi-node tensor parallelism, which facilitates running the model on multiple network-connected machines.
Multi-Token Prediction (MTP) is currently in development, and you can follow progress through the optimization plan.
Launch instructions from the SGLang team are available here: https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3
6.3 Inference with LMDeploy (recommended)
LMDeploy is a flexible, high-performance inference and serving framework designed for large language models, including DeepSeek-V3. It supports both offline pipeline processing and online deployment, seamlessly integrating with PyTorch-based workflows.
For comprehensive, step-by-step instructions on running DeepSeek-V3 with LMDeploy, please refer to InternLM/lmdeploy#296
6.4 Inference with TRT-LLM (recommended)
TensorRT-LLM now supports DeepSeek-V3, offering precision options such as BF16 and INT4/INT8 weight-only. Support for FP8 is in progress and will be released soon. You can try out the custom branch of TRTLLM designed specifically for DeepSeek-V3 here: https://github.com/NVIDIA/TensorRT-LLM/tree/deepseek/examples/deepseek_v3.
6.5 Inference with vLLM (recommended)
vLLM version .6.6 supports inference for DeepSeek-V3 in both FP8 and BF16 modes on NVIDIA and AMD GPUs. Beyond standard inference techniques, vLLM offers pipeline parallelism, enabling the model to run on multiple machines connected by a network. For detailed guidance, please refer to the vLLM documentation. You might also be interested in following the enhancement plan.
6.6 Recommended Inference Functionality with AMD GPUs
In partnership with the AMD team, we have achieved Day-One support for AMD GPUs using SGLang, compatible with both FP8 and BF16 precision. For detailed instructions, please refer to the SGLang guide.
6.7 Recommended Inference Functionality with Huawei Ascend NPUs
The MindIE framework developed by the Huawei Ascend community has successfully adapted DeepSeek-V3 in BF16 mode. For step-by-step instructions on running DeepSeek-V3 on Ascend NPUs, please follow the provided guidance.
DeepSeek AI represents much more than a polished chatbot—it embodies a paradigm shift in how advanced AI models are built, deployed, and accessed. By challenging the high costs and secretive nature of proprietary systems, DeepSeek is democratizing AI and stimulating a vibrant global dialogue on cost, accessibility, and innovation. Whether you are a developer, a business leader, or an AI enthusiast, keeping a close eye on DeepSeek is essential as it continues to drive discovery and shape both the technological and geopolitical landscapes.
As the tech industry evolves and the boundaries of AI innovation expand, DeepSeek stands as a testament to the power of resourcefulness and collaborative effort. It challenges established norms and may well redefine the future of artificial intelligence.
Stay tuned for more insights on the most advanced AI developments and how they could transform your business strategy. If you found this post valuable, please share it widely and subscribe for future updates!
References available upon request.
Want your blog posts to actually get noticed?
With DeepRankAI, creating content that ranks well and brings in more traffic doesn’t have to be a grind. Our AI is tuned to help you write posts that connect with people and search engines—so your site gets the attention it deserves.
👉 Check out DeepRankAI and see how it can make content creation a whole lot easier.
