DeepSeek Sparse Attention From Scratch
I added a DeepSeek Sparse Attention from-scratch implementation to the LLMs-from-scratch repository, thanks to an excellent reader contribution.
The folder includes a README, a standalone GPT-style reference implementation, and tests:
The main idea behind DeepSeek Sparse Attention is to replace a fixed sparse pattern with a learned sparse pattern. Instead of using only a local window, the mechanism uses a lightweight indexer and selector to decide which prior tokens are worth attending to.
For more background, I also have a local DeepSeek Sparse Attention concept page and a gallery explainer that compare it with regular causal attention and sliding-window attention.
Source: lightly edited website version of my Substack note.
Read Next
Inkling: A New Open-Weight 975B MoE with a Few Surprises
Short note on Thinking Machines Lab's 975B Inkling open-weight model, its benchmark profile, sparse MoE design, short convolutions, embedding RMSNorm, and
200,000 Subscribers
Short note celebrating Ahead of AI reaching 200,000 subscribers.
GPT 5.6 Has 72 Possible Configurations. What's A Good Default?
Short note on how GPT 5.6 model and effort choices map onto training-time and inference-time scaling, producing 72 configurations.