Podcast · 156 episodes on record

How scaling changes model behavior

Published

October 9, 2024

Length

11 min

Position

53 of 156

October 9, 2024

How scaling changes model behavior

0:00 / 11:47

0:00Choose a match or chapter to light up its moment11:47

Listen to the full episode

Excerpt playback isn’t available for this episode — use the full-episode link.

Search the Record

Moment search isn’t open for this episode yet

When it opens, you’ll be able to type any phrase and jump straight to the second it was said.

Nathan Lambert Episodes Around October 9, 2024

Episode 53 of 156 — walk the feed in the order it was published.

View all episodes

15:22

Llama 3.1 405b, Meta's AI strategy, and the new open frontier model ecosystem

Jul 23, 2024

7:55

GPT-4o-mini changed ChatBotArena

Jul 31, 2024

1:03:42

Interviewing Sebastian Raschka on the state of open LLMs, Llama 3.1, and AI education

Aug 1, 2024

10:23

OpenAI's Strawberry, LM self-talk, inference scaling laws, and spending more on inference

Sep 5, 2024

5:38

A post-training approach to AI regulation with Model Specs

Sep 10, 2024

11:31

Futures of the data foundry business model

Sep 11, 2024

18:51

[Article Voiceover] Reverse engineering OpenAI's o1

Sep 17, 2024

14:04

[Article Voiceover] Llama 3.2 Vision and Molmo: Foundations for the multimodal open-source ecosystem

Sep 27, 2024

1:08:39

Interviewing Riley Goodside on the science of prompting

Sep 30, 2024

10:29

[Article Voiceover] AI Safety's Crux: Culture vs. Capitalism

Oct 2, 2024

11:47This episode

How scaling changes model behavior

Oct 9, 2024

1:00:12

Interviewing Andrew Trask on how language models should store (and access) information

Oct 10, 2024

16:36

(Voiceover) Building on evaluation quicksand

Oct 16, 2024

54:21

Interviewing Arvind Narayanan on making sense of AI hype

Oct 17, 2024

11:23

(Voiceover) Claude's agentic future and the current state of the frontier models

Oct 23, 2024

10:19

(Voiceover) Why I build open language models

Oct 30, 2024

54:10

Interviewing Andrew Carr of Cartwheel on the State of Generative AI

Oct 31, 2024

1:15:45

Interviewing Tim Dettmers on open-source AI: Agents, scaling, quantization and what's next

Nov 7, 2024

11:22

(Voiceover) Saving the National AI Research Resource & my AI policy outlook

Nov 13, 2024

4:21

(Voiceover) Scaling realities

Nov 14, 2024

7:59

(Voiceover) Tülu 3: The next era in open post-training

Nov 21, 2024

10:26

(Voiceover) OLMo 2 and building effective teams for training language models

Nov 26, 2024

12:13

(Voiceover) OpenAI's o1 using "search" was a PSYOP

Dec 4, 2024

1:08:33

Interviewing Finbarr Timbers on the "We are So Back" Era of Reinforcement Learning

Dec 5, 2024

12:40

(Voiceover) OpenAI's Reinforcement Finetuning and RL for the masses

Dec 11, 2024

11:00

(Voiceover) The AI agent spectrum

Dec 18, 2024

View all episodes

Summary

How scaling changes model behaviorSome trends are reasonable to extrapolate, some are not. Even for the trends we are succeeding at extrapolating, it is not clear how that signal translates into different AI behaviors.Read it here: <a href="https://www.interconnects.ai/p/how-scaling-changes-model-behavior" target="_blank">https://www.interconnects.ai/p/how-scaling-changes-model-behavior</a>[00:00] How scaling changes model behavior[05:03] Metaphors for what scaling may solve[08:45] Short-term scaling is already de-riskedFig. 1: <a href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/openai-predictions.webp" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/openai-predictions.webp</a>Fig. 2: <a href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/openai-predictions.webp" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/scaling-laws.webp</a>Fig. 3: <a href="https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/openai-predictions.webp" target="_blank">https://huggingface.co/datasets/natolambert/interconnects-figures/resolve/main/manual/situational-awareness.webp</a> This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit <a href="https://www.interconnects.ai/subscribe?utm_medium=podcast&utm_campaign=CTA_2">www.interconnects.ai/subscribe</a>

Episode details and artwork are published by the show’s own feed. Sentinel plays short cited excerpts and links every episode back to its publisher.

← All episodes from Nathan Lambert

How scaling changes model behavior

Search the Record

Nathan Lambert Episodes Around October 9, 2024

Llama 3.1 405b, Meta's AI strategy, and the new open frontier model ecosystem

GPT-4o-mini changed ChatBotArena

Interviewing Sebastian Raschka on the state of open LLMs, Llama 3.1, and AI education

A recipe for frontier model post-training

Interviewing Ross Taylor on LLM reasoning, Llama fine-tuning, Galactica, agents

Nous Hermes 3 and exploiting underspecified evaluations

On the current definitions of open-source AI and the state of the data commons

OLMoE and the hidden simplicity in training better foundation models

OpenAI's Strawberry, LM self-talk, inference scaling laws, and spending more on inference

A post-training approach to AI regulation with Model Specs

Futures of the data foundry business model

[Article Voiceover] Reverse engineering OpenAI's o1

[Article Voiceover] Llama 3.2 Vision and Molmo: Foundations for the multimodal open-source ecosystem

Interviewing Riley Goodside on the science of prompting

[Article Voiceover] AI Safety's Crux: Culture vs. Capitalism

How scaling changes model behavior

Interviewing Andrew Trask on how language models should store (and access) information

(Voiceover) Building on evaluation quicksand

Interviewing Arvind Narayanan on making sense of AI hype

(Voiceover) Claude's agentic future and the current state of the frontier models

(Voiceover) Why I build open language models

Interviewing Andrew Carr of Cartwheel on the State of Generative AI

Interviewing Tim Dettmers on open-source AI: Agents, scaling, quantization and what's next

(Voiceover) Saving the National AI Research Resource & my AI policy outlook

(Voiceover) Scaling realities

(Voiceover) Tülu 3: The next era in open post-training

(Voiceover) OLMo 2 and building effective teams for training language models

(Voiceover) OpenAI's o1 using "search" was a PSYOP

Interviewing Finbarr Timbers on the "We are So Back" Era of Reinforcement Learning

(Voiceover) OpenAI's Reinforcement Finetuning and RL for the masses

(Voiceover) The AI agent spectrum

Summary