Kimi-Linear is a 3B active, <6T tokens experiment. Its architecture is nothing sci-fi (except it works) – NoPE MLA + fancy GatedDeltaNet.
this very strongly suggests to me that a) Gemini long-context attention doesn't have any secret sauce b) it's all about TPUs. No "Titans".
Context Arena Update: Added kimi-linear-48b-a3b-instruct [11-08] and kimi-k2 (Thinking) [11-06] to the MRCR leaderboards.
The Linear 48b results are fascinating! It actually outperforms the new Gemini 3.0 Pro Thinking on 4-needle and 8-needle tasks at higher context lengths (512k+). I've added it to 2needle, 4needle, and 8needle.
kimi-k2 (Thinking) lands lower on the leaderboards (Rank #22 for 2-needle AUC @ 128k), with a hard context ceiling around 262k. I did not run it for 2needle and 4needle.
All results at:
The performance curve for the Linear model is distinct: while it underperforms Gemini 3 significantly at shorter contexts (<=256k) on the difficult 8-needle test, its degradation slope is much flatter. Gemini starts higher and drops fast; Kimi starts lower but holds steady, overtaking Gemini at the higher end.
However, note that kimi-linear-48b has noticeable performance drops past 128k on the easier 2 & 4 needle tests. Additionally, due to lower token efficiency compared to Gemini/GPT, only ~60% of the 1M token tests successfully ran (hitting limits/OOM). So some caution with the results at the 1M level.
kimi-linear-48b results:
2-Needle Performance (@ 128k / @ 1M):
- AUC: 96.5% (vs Gem 3: 99.5%) / 81.7% (vs Gem 3: 85.5%)
- Pointwise: 96.0% (vs Gem 3: 99.0%) / 77.0% (vs Gem 3: 72.2%)
4-Needle Performance (@ 128k / @ 1M):
- AUC: 85.5% (vs 85.8%) / 62.7% (#1, beating Gem 3: 57.3%)
- Pointwise: 83.7% (vs 80.8%) / 51.5% (#1, beating Gem 3: 34.3%)
8-Needle Performance (@ 128k / @ 1M):
- AUC: 54.9% (vs 73.0%) / 43.8% (#1, beating Gem 3: 39.0%)
- Pointwise: 49.0% (vs 54.2%) / 35.3% (#1, beating Gem 3: 24.5%)
A very different architectural approach yielding impressive stability at scale. Because of its current price point, it is very competitive for long context (MRCR).
Enjoy.
@Kimi_Moonshot
@GoogleDeepMind @googleaidevs
@OpenAI @OpenAIDevs
6,8 rb
16
Konten pada halaman ini disediakan oleh pihak ketiga. Kecuali dinyatakan lain, OKX bukanlah penulis artikel yang dikutip dan tidak mengklaim hak cipta atas materi tersebut. Konten ini disediakan hanya untuk tujuan informasi dan tidak mewakili pandangan OKX. Konten ini tidak dimaksudkan sebagai dukungan dalam bentuk apa pun dan tidak dapat dianggap sebagai nasihat investasi atau ajakan untuk membeli atau menjual aset digital. Sejauh AI generatif digunakan untuk menyediakan ringkasan atau informasi lainnya, konten yang dihasilkan AI mungkin tidak akurat atau tidak konsisten. Silakan baca artikel yang terkait untuk informasi lebih lanjut. OKX tidak bertanggung jawab atas konten yang dihosting di situs pihak ketiga. Kepemilikan aset digital, termasuk stablecoin dan NFT, melibatkan risiko tinggi dan dapat berfluktuasi secara signifikan. Anda perlu mempertimbangkan dengan hati-hati apakah trading atau menyimpan aset digital sesuai untuk Anda dengan mempertimbangkan kondisi keuangan Anda.