- The following is a conversation
with Jensen Huang,

CEO of NVIDIA, one of the most
important and influential

companies in the history
of human civilization.

NVIDIA is the engine powering
the AI revolution, and a lot

of its success can be directly
attributed to Jensen's sheer force of

will and his many brilliant bets and decisions
as a leader, engineer, and innovator.

This is Lex Fridman Podcast. And now
dear friends, here's Jensen Huang.

You've propelled NVIDIA into a new era in

AI, moving beyond its focus on chip
scale design to now rack scale

design. And I think it's fair to
say that winning for NVIDIA for a

long time used to be about building
the best GPU possible, and you still

do, but now you've expanded
that to extreme co-design of

GPU, CPU memory, networking,
storage, power cooling,

software, the rack itself, the pod
that you've announced, and even the

data center. So let's talk about
extreme co-design. What is the

hardest part of co-designing
system with that many complex

So first of all, the reason why extreme
co-design is necessary is because the

problem no longer fits inside one
computer to be accelerated by one GPU.

The problem that you're trying to
solve is you would like to go faster

than the number of computers
that you add. So you added

you know, 10,000 computers, but you
would like it to go a million times

faster. Then all of a sudden
you have to take the algorithm,

you have to break up the algorithm,
you have to refactor it,

you have to shard the pipeline, you have
to shard the data, you have to shard the

model. Now all of a sudden when you
distribute the problem this way,

not just scaling up the problem, but
you're distributing the problem,

then everything gets in the
way. This is the Amdahl's Law

problem where the amount of
speed up you have for something

depends on how much of the
total workload it is. And so

if computation represents 50% of the problem,
and I sped up computation infinitely

you know, I only sped up the total
workload by a factor of two.

Now all of a sudden, not only
do you have to distribute a

computation, you have to, you
know, shard the pipeline somehow.

You also have to solve
the networking problem

because you've got all of these
computers are all connected together.

And so distributed computing
at the scale that we do,

the CPU is a problem, the GPU is a
problem, the networking is a problem, the

switching is a problem. And
distributing the workload across all

It's just a massively complex computer
science problem. And so we just

gotta bring every technology to bear.
Otherwise, we scale up linearly or we

scale up based on the capabilities
of Moore's Law, which has

largely slowed because
Dennard scaling has slowed.

- I'm sure there's trade-offs there.
Plus you have a complete disparate

disciplines here. I'm sure you have specialists
in each one of these high bandwidth

memory, the network and the NVLink,
the NICs, the optics and the

copper that you're doing, the power delivery, the
cooling, all of that. I mean, there's like world

experts in each of those. How do you get
'em in a room together to figure out-

- What's the process—can you take me through
the process of the specialists and the

generalists? Like how do you put
together the rack when you know the

s- the set of things you have
to shove into a rack together?

Like what does that process look
like of designing it all together?

- Yeah. There's the first question,
which is: what is extreme co-design?

You're, you, we're optimizing
across the entire stack of software

from architectures to chips, to
systems, to system software, to the

algorithms, to the applications. That's
one layer. The second thing that you and

I just talked about is goes
beyond CPUs and GPUs and

networking chips and scale up
switches and scale out switches.

And then of course, you gotta
include power and cooling and all

of that because, you know, all
these computers are extremely,

extremely power hungry. They do
a lot of work and they're very

energy efficient, but they in
aggregate still consume a lot of

power. And so that's one. The
first question is, what is it?

The second question is, why is it, and
we just spoke about the reason, you

know you want to distribute the
workload so that you can exceed

the benefit of just increasing
the number of computers.

And then the third question is,
how is it, how do you do it?

And that's the, that's kind
of the miracle of this

company. You know, when you're designing a
computer, you have to have operating system of

computers. When you're
designing a company,

you should first think about what is it that you
want the company to produce. You know, I see

a lot of companies organization
charts, and they all look the same.

Hamburger organization charts,
soft organization charts, and car

company organization charts.
They all look the same.

And it doesn't make any sense to me. You
know, the goal of a company is to be the

company is to be the machinery, the
mechanism, the system that produces

the output. And that output is the
product that we like to create.

It is also designed, the architecture
of the company should reflect

the environment by which it
exists. It almost indirectly

says what you should do with the
organization. My direct staff is 60 people.

You know, I don't have one-on-ones with 'em because it's
impossible. You can't have, you can't have 60 people

on your staff if you're, you
know, gonna get work done and-

- So you still have 60 reports.
You still have across-

- More. And most stars at least
have a foot in engineering.

There's experts in memory, there's
experts in CPUs, there's experts in

- Yeah, GPUs and— Architecture,
algorithms, design, um—

- So, you constantly have an eye on the
entire stack, and you're having to, like,

intense discussions about the
designer of the entire stack?

- And no conversation is ever one
person. That's why I don't do

and all of us attack it. You know,
because we're doing extreme co-design.

And literally, the company is doing
extreme co-design all the time.

Jensen Huang: NVIDIA - The $4 Trillion Company & the AI Revolution | Lex Fridman Podcast #494

Show Full Transcript

Repurpose this content with AI