Llama 4: Did Meta just push the panic button?

<p><a href="https://www.interconnects.ai/p/llama-4" target="_blank">https://www.interconnects.ai/p/llama-4</a></p><p>Where Llama 2’s and Llama 3’s releases were arguably some of the top few events in AI for their respective release years, Llama 4 feels entirely lost. Meta has attempted to reinvent their formula of models with substantial changes in size, architecture, and personality, but a coherent narrative is lacking. Meta has fallen into the trap of taking too long to ship, so the bar is impossible to cross successfully.</p><p>Looking back at the history of Meta’s major open models, the sequence is as follows:</p><p>* <strong>OPT</strong> – Released <strong>May 3, 2022</strong> (<a href="https://ai.meta.com/blog/democratizing-access-to-large-scale-language-models-with-opt-175b/" target="_blank">ai.meta.com</a> | 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B, 175B): A foundational open model that is underrated in the arc of language modeling research.</p><p>* <strong>LLaMA</strong> – Released <strong>February 24, 2023</strong> (<a href="https://ai.meta.com/blog/large-language-model-llama-meta-ai/" target="_blank">ai.meta.com</a> | 7B, 13B, 33B, 65B): The open weight model that powered the Alpaca age of early open chat models.</p><p>* <strong>Llama 2</strong> – Released <strong>July 18, 2023</strong> (<a href="https://www.interconnects.ai/p/llama-2-from-meta?utm_source=publication-search" target="_blank">our coverage</a> | <a href="https://about.fb.com/news/2023/07/llama-2/" target="_blank">about.fb.com</a> | 7B, 13B, 70B): The open standard for academic research for its time period. <a href="https://www.interconnects.ai/p/llama-2-part-2?utm_source=publication-search" target="_blank">Chat version had some bumps</a>, but overall a major win.</p><p>* <strong>Llama 3</strong> – Released <strong>April 18, 2024</strong> (<a href="https://www.interconnects.ai/p/llama-3-scaling-open-llms-to-agi-aac?utm_source=publication-search" target="_blank">our coverage</a> | <a href="https://ai.meta.com/blog/introducing-meta-llama-3-the-most-capable-openly-available-llm-to-date/" target="_blank">ai.meta.com</a> | 8B, 70B): The open standard for its time. Again, fantastic base models.</p><p>* <strong>Llama 3.1</strong> – Released <strong>July 23, 2024</strong> (<a href="https://www.interconnects.ai/p/llama-405b-open-frontier-model?utm_source=publication-search" target="_blank">our coverage</a> | <a href="https://ai.meta.com/blog/introducing-llama-3-1-our-most-capable-models-to-date/" target="_blank">ai.meta.com</a> | 8B, 70B, 405B): Much improved post training and the 405B marked the first time an open weight model competed with GPT-4!</p><p>* <strong>Llama 3.2</strong> – Released <strong>September 25, 2024</strong> (<a href="https://www.interconnects.ai/p/molmo-and-llama-3-vision?utm_source=publication-search" target="_blank">our coverage</a> | <a href="https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/" target="_blank">ai.meta.com</a> | 1B, 3B, 11B, 90B): A weird, very underperforming vision release, outshined by Molmo on the same day.</p><p>* <strong>Llama 3.3</strong> – Released <strong>December 6, 2024</strong> (<a href="https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md#L237-L240" target="_blank">github.com</a> | 70B): Much improved post-training of the smaller 3.1 models, likely in <a href="https://www.interconnects.ai/p/tulu-3" target="_blank">response to other open releases</a>, but largely a minor update.</p><p>* <strong>Llama 4</strong> – Released <strong>April 5, 2025</strong> (<a href="https://ai.meta.com/blog/llama-4-multimodal-intelligence/" target="_blank">ai.meta.com</a> | 17A109B, 17A400B): What we got today.</p><p>The time between major versions is growing, and the number of releases seen as exceptional by the community is dropping. Llama 4 consists of 3 models, quoting from the blog post, notes in brackets mine:</p><p>* Llama 4 Scout, a <strong>17 billion active parameter model with 16 experts [and 109B total parameters, ~40T training tokens]</strong>, is the best multimodal model in the world in its class and is more powerful than all previous generation Llama models, while fitting in a single NVIDIA H100 GPU.</p><p>* Llama 4 Maverick, a <strong>17 billion active parameter model with 128 experts [and 400B total parameters, ~22T training tokens]</strong>.</p><p>* These models are our best yet thanks to distillation from <strong>Llama 4 Behemoth</strong>, <strong>a 288 billion active parameter [and 2T total parameters]</strong> model with 16 experts that is our most powerful yet and among the world’s smartest LLMs…. we’re excited to share more details about it even while it’s still in flight.</p><p>Here are the reported benchmark scores for the first two models, which are available on many APIs and to download on HuggingFace.</p><p>Where Llama models used to be scaled across different sizes with almost identical architectures, these new models are designed for very different classes of use-cases.</p><p>* Llama 4 Scout is similar to a Gemini Flash model or any ultra-efficient inference MoE.</p><p>* Llama 4 Maverick’s architecture is very similar to DeepSeek V3 with extreme sparsity and many active experts.</p><p>* Llama 4 Behemoth is likely similar to Claude Opus or Gemini Ultra, but we don’t have substantial information on these.</p><p>This release came on a Saturday, which is utterly bizarre for a major company launching one of its highest-profile products of the year. The consensus was that Llama 4 was going to come at Meta’s <a href="https://techcrunch.com/2025/02/18/meta-announces-llamacon-its-first-generative-ai-dev-conference/" target="_blank">LlamaCon</a> later this month. In fact, it looks like this release may have been pulled forward from today, the 7th, from a <a href="https://github.com/meta-llama/llama-models/commit/1ba7d45b4f6f3f89feed1e6fae8a0c4b8cc2932f" target="_blank">commit in the Meta Llama Github</a>:</p><p>One of the flagship features is the 10M (on Scout, Maverick is 1M) token context window on the smallest model, but even that didn’t have any released evaluations beyond Needle in a Haystack (NIAH), which is seen as a necessary condition, but not one that is sufficient to say it is a good long-context model. Some more modern long-context evaluations include <a href="https://arxiv.org/abs/2404.06654" target="_blank">RULER</a> or <a href="https://arxiv.org/abs/2502.05167" target="_blank">NoLiMa</a>.</p><p><a href="https://x.com/jaxgriot/status/1908908374978429167" target="_blank">Many</a>, <a href="https://x.com/TheXeophon/status/1908900306580074741" target="_blank">many</a> <a href="https://x.com/natolambert/status/1908895656535871936" target="_blank">people</a> <a href="https://github.com/lm-sys/FastChat/issues/3670#issuecomment-2614213658" target="_blank">have</a> commented on how Llama 4’s behavior is drastically different in LMArena — which was their flagship result of the release — than on other providers (even when following Meta’s <a href="https://www.llama.com/docs/model-cards-and-prompt-formats/llama4_omni/" target="_blank">recommended system prompt</a>). Turns out, from the blog post, that it is just a different model:</p><p>Llama 4 Maverick offers a best-in-class performance to cost ratio with an <strong>experimental chat version</strong> scoring ELO of 1417 on LMArena.</p><p>Sneaky. The results below are fake, and it is a major slight to Meta’s community to not release the model they used to create their major marketing push. We’ve seen many open models that come around to maximize on ChatBotArena while destroying the model’s performance on important skills like math or code. We’ll see where the released models land.</p><p>Regardless, here’s the plot Meta used. Look at the fine print at the bottom too.</p><p>This model is actually the one tanking the technical reputation of the release because its character is juvenile. The actual model on other hosting providers is quite smart and has a reasonable tone!</p><p>ArtificialAnalysis rated the models as “<a href="https://x.com/ArtificialAnlys/status/1908890796415414430" target="_blank">some of the best non-reasoning models</a>,” beating leading frontier models. This is complicated because we shouldn’t separate reasoning from non-reasoning models; we should just evaluate on reasoning and non-reasoning domains separately, as discussed in the <a href="https://www.interconnects.ai/p/gemini-25-pro-googles-second-ai-chance" target="_blank">Gemini 2.5 post</a>. So-called “reasoning models” often top non-reasoning benchmarks, but the opposite is rarely true.</p><p>Other independent evaluation results range from <a href="https://raw.githubusercontent.com/KCORES/kcores-llm-arena/refs/heads/main/scripts/llm_benchmark_results_normalized.png" target="_blank">medium</a> to <a href="https://eqbench.com/creative_writing_longform.html" target="_blank">bad</a> and <a href="https://x.com/paulgauthier/status/1908976568879476843" target="_blank">confusing</a> — I suspect very weird results are hosting issues with the very long context models. At the same time, the <a href="https://x.com/emollick/status/1908609043741933914" target="_blank">Behemoth model is outclassed by Gemini 2.5 Pro</a>. To list some of the major technical breakthroughs that Meta made (i.e. new to Llama, not new to the industry):</p><p>* Mixture of expert architectures, enabling <a href="https://x.com/scaling01/status/1908657167869100482" target="_blank">Llama 4 to be trained with less compute than Llama 3</a> even though they have more total parameters — a lot more.</p><p>* Very long context up to 10M tokens.</p><p>* Solid multimodal input performance on release day (and not a later model)</p><p></p><p>Interconnects is a reader-supported publication. Consider becoming a subscriber.</p><p></p><p>Sadly this post is barely about the technical details. Meta nuked their release vibes with weird timing and by having an off-putting chatty model that was easiest to find to talk to. The release process, timing, and big picture raise more questions for Meta. Did they panic and feel like this was their one shot at being state of the art?</p><p>The evaluation scores for the models are solid, they clear a fairly high bar. With these highly varied MoE architectures, it’s super hard to feel confident in an assessment of the model based on benchmarks, especially when compared to dense models or teacher-student distilled models. The very-long-context base models will be extremely useful for research.</p><p>The question here is: <strong>Why is Meta designing their models in the same way as other frontier labs</strong> when their audience is open-source AI communities and businesses, not an API serving business or ChatGPT competitor?</p><p>The model sizing for the likes of Gemini and ChatGPT is downstream of nuanced decisions based on a balance of training cluster size, inference needs, and performance trade-offs. These trade-offs are very different for open models, where you don’t pay inference, and many users are not hyperscale companies.</p><p>The model that becomes the “open standard” doesn’t need to be the best overall model, but rather a family of models in many shapes and sizes that is solid in many different deployment settings. Qwen 2.5, with <a href="https://huggingface.co/collections/Qwen/qwen25-66e81a666513e518adb90d9e" target="_blank">models at 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters</a>, is the closest to this right now. There’s actually far less competition in this space than in the space Meta chose to go into (and take on DeepSeek)!</p><p>One of these communities historically has been the LocalLlama subreddit, which named the entire community around running models at home around the Llama series of models — <a href="https://www.reddit.com/r/LocalLLaMA/comments/1jsl37d/im_incredibly_disappointed_with_llama4/" target="_blank">they’re not happy with Llama 4</a>. Another community is academics, where the series of models across different size ranges is wonderful for understanding language models and improving methods. These two groups are all GPU-poor, so memory-intensive models like these sparse mixture of experts price out even more participants in the open community (who tend to be memory-limited).</p><p>This is all on top of an onerous license that makes all artifacts that use Llama in the process be tagged with the “Llama-” name, the Llama license, the “Built with Llama” branding if used commercially, and use-case restrictions. This is at the same time when their competitors, i.e. DeepSeek, released their latest flagship model with an MIT license (which has no downstream restrictions).</p><p>A third group is potential businesses looking to <a href="https://www.interconnects.ai/i/158931739/the-state-of-the-open-closed-model-gap" target="_blank">use open models on-premises as open models close the gap to closed counterparts</a>. These feel like groups that would be sensitive to the extra legal risk that Llama’s license exposes them to.</p><p>On top of all of this weirdness, many of Meta’s “open-source” efforts are restricted in the European Union. Where the Llama 3.2 models blocked you if you tried to access them from Europe, Llama 4 is available for download but prohibits the use of vision capabilities in an <a href="https://www.llama.com/llama4/use-policy/" target="_blank">acceptable use policy</a>. This is not entirely Meta’s fault, as many companies are dealing with side effects of the EU AI Act, but regulatory exposure needs to be considered in Meta’s strategy.</p><p>Meta had a tight grasp on these communities, the Llama projects were rightfully loved, but now they feel lost. With Qwen 3 around the corner and countless other amazing open-weight models out now (and many more teased, <a href="https://natolambert.substack.com/p/some-thoughts-on-openai-returning" target="_blank">such as from OpenAI</a>), the competition is extreme.</p><p>The soul of the Llama series died by not releasing enough models frequently enough. Reclaiming that with <a href="https://x.com/natolambert/status/1882482021844549867" target="_blank">GenAI’s constant organizational headaches</a> looks like a Sisyphean task. What is Meta’s differentiation in the AI space? It still seems about enabling their own platforms to flourish, not about truly supporting open.</p><p>Meta’s GenAI organization has been showing major signs of cultural challenges thoughout its entire existence — including their <a href="https://www.bloomberg.com/news/articles/2025-04-01/meta-s-head-of-ai-research-to-leave-roiling-investment-push" target="_blank">head of AI research</a> leaving just a few days before this model was launched.</p><p>Sadly, the evaluations for this release aren’t even the central story. The vibes have been off since the beginning by choosing a weird release date. Over the coming weeks, more and more people will find reliable uses for Llama 4, but in a competitive landscape, that may not be good enough. Llama is no longer the open standard. Personally, this makes me sad. As an American, I want the default pieces of the <a href="https://www.interconnects.ai/p/making-the-us-the-home-for-open-source" target="_blank">open ecosystem to be run by American or American-friendly companies</a>.</p><p>With the macro pressure coming to Meta’s business and the increasing commoditization of open models, how is Zuckerberg going to keep up in face of shareholder pressure pushing back against the cost of the Llama project? This isn’t the first time he’s done so, but he needs to reevaluate the lowest level principles of their approach to open AI.</p> <br /><br />This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&#38;utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>