OpenAI’s Chief Economist on the Future of LLMs

UVA Darden.

The Disruption, a lively discussion between UVA Darden School of Business professors Yael Grushka Cocaine and Mike Lennox on cutting edge technologies and practices that are challenging the status quo.

It's hello yeah.

Hey, Mike.

Hey, Mike and everybody in the room.

Hello, hello.

Uh, usually we do our podcast, we're together in a studio, but it's a little bit different today.

So, first of all, where in the world is Jael?

I'm, uh, I'm in Charlottesville.

I'm, you know, keeping the fort here, uh, uh, standing.

I'm back here in our offices.

Uh, I miss being there with you in person, but I'm glad that we can do this podcast.

We, we miss you as well.

I know you're vice deaning there in Charlottesville, Park D ELA.

Uh, I am here in our Sands family grounds in Rosslyn, Virginia, uh, with our DC techn, and I have in front of me, you know, a few of our students.

Students and alumni, uh, who are, who are mous, so it's really excited to have them here for our podcast recording.

All right, so yo, I wanna begin with a trivia question.

Oh, here we go.

OK, you're gonna put me on the spot.

Yes, I think you already know this one.

What happened?

On November 30th, 2022. 0 yes, I remember, I remember the day that, uh, chat GPT OpenAI kind of turned the switch on for the rest of the world for to enjoy their new version of their large language model.

Is that the day, the day that my students looked at it?

Absolutely correct. 3.5 was released on that date.

It was a seminal moment.

The world.

The world changed.

Everyone figured out about these LLMs just to put some numbers on it, uh, within 5 days, they had a million subscribers.

Within 2 months, they had over 100 million subscribers.

It is arguably the fastest diffused technology perhaps in the history, uh, of the world there.

So just this subtle moment.

A fun fact for you, uh, one week before that, I had submitted my Manuscript for my book strategy in the Digital Age to the publisher.

So guess what's not mentioned in strategy in the digital age?

Chat GPT.

I do have, you know, I did talk about AI and LLMs, but, uh, you know, timing, timing was bad there.

I'm curious.

I remember that day, Mike.

I was in class and a student stopped me on my way out and he said, Have you seen this?

Did you see what it can do?

And he was sure that, you know, he never needed to come to class ever again.

It was very, very interesting.

We had this really funny back and forth, the two of us, and I was like, no, the Socratic method is really valuable still, so I wonder what he thinks now.

We might want to talk more about that, but I wanted to talk a little bit about the history here, because as you know, as well as I, this wasn't like Out of the blue, right?

Sometimes it's treated as if somehow on that day, OpenAI invented, you know, AI and uh large language models, but actually it goes back nearly 50 years.

So I'm gonna, as I often do, uh, date myself here.

So one of the first, if you want to call it language models, wasn't nearly what we have today, was something called Eliza, uh, that was available.

Uh, I actually had it installed on my Apple IIE that I had as a teenager.

And what Eliza did was allow you to have the facsimile of a conversation, uh, with the computer.

Uh, it was what we called expert systems.

So what you would have is basically prompt response.

A little bit, maybe even like some more modern technology like Alexa, where you ask at the weather.

And it automatically kind of gives you a response to that.

So yeah, it was in the 60s.

That was in the 60s, right?

By the time we were studying some of this in our um degrees that are kind of similar, you and I, you know, these systems continued to evolve like in the 90s, they accelerated as well.

I mean, we're not talking like, you know, not even in the 2000s, we're going back to, you know, the 1960s and uh 1970s.

I always talk about how when I was a system engineering student.

As an undergrad, I was learning neural networks.

I was learning some of the backbones of AI, uh, but we just weren't ready for prime time.

So I learned in my fourth year of my undergrad, I learned Lisp.

Do you remember Lisp?

Did you study?

I do remember Lisp.

Yeah, of course, of course, yeah.

Well, I was gonna say, you know, kind of two things happened.

You know, one, we started to apply neural networks to these language models in the 2000s.

Then there was this seminal advancement in 2017, where Google released a thought piece, a paper uh about these transform uh algorithms, if you will.

And it, it really took off from there to, to, again, the seminal moment in 2022.

Yeah, gram, what is that?

What is that transformer piece?

I mean, we were doing anagrams, we were doing like word to vec, all these things as data science was growing.

What exactly did the transformer technology kind of add to the, to the, to the mix?

Like, can we think about that?

Like what is the big innovation there by Google?

I mean, you, I don't know.

So, but you, you're the data scientist, so then you could answer this for me.

I'm a strategy professor at this point.

Well, some of it has to do with really, um, how, how they, how really, how, how far can it go, right?

When we were talking about a natural language processing before these uh transformers came out, it was really about the next word, the next few words.

It was really kind of piecemeal, and what really kind of jumps, you know, really.

Created like a jump in terms of the innovation here was the ability to do much longer paragraphs, phrases, you know, pieces and articles like now you're putting strings together now it's not just one word at a time, one prediction, one P at a time predicting what the next word is, but it's a bunch of these happening, uh, altogether and so the capabilities here of maybe what uh, what was very innovative, the neural network was just now taking off like, uh, exponentially, right?

So I'm curious, you know, how are you using LLM these days?

Uh, um, so, so in so many different ways, um, everything from, uh, you know, your basic workflows, which I think a lot of individuals are, are, are starting to do around like responses and letterheads and just starting, uh, lists of various kinds of various, uh, creativity like parties or events and things that you just need to kind of, uh, you need a partner, a thought partner to jumpstart you to give you kind of a starting point.

To, uh, uh, looking through data and looking for patterns, I teach with it.

I teach about it and using it, uh, to basically allow my students to have a quick entry point into like data science.

Um, we teach it in, in, in our coding class to see how we can learn how to code faster, um, so it's really helpful for anybody trying to.

Kind of master some, uh, entry level coding.

We insist that people understand the code, but this kind of helps them like find the right code way quicker, um, anything to, to imagining my next cases and how I'm gonna write them and teach them.

So it's really been taken over my entire workflow in many different ways.

What about you?

Do you use, uh, chat?

This is in the early, uh, days, if you will, like 2023 or so.

My niece was getting married and I was giving the blessing for the wedding, and I admit that I used chat GPT for the first draft, and then, you know, of course, customized it and the like, but it was, it was very, very helpful.

You, you mentioned a couple that I think are really fascinating to me, uh, you know, what we're now calling vibe Cody, the ability to use these LLMs as partners with you for Cody to just radically transform.

Forming the way we think about coding.

I was talking to Phil Borne, who you know well, our dean of the School of Data Science.

I was asking like, are we even gonna teach like Python or Microsoft BI or Tableau or these different systems?

And it's not clear that we even need to move forward because you can now engage with the LLM and it can do a lot of that, uh, data analysis.

Well, in a, in a way, and, um, I'll be curious to hear your thoughts, but My sense is is that some of that backbone is not necessarily going away and it's still gonna be very valuable and somewhat of a competitive advantage for somebody to actually understand what the LLM is running behind the scenes.

So if I've asked if for instance today I engaged with a a group that we're coming up with predictions um related to um uh cars, whether cars were good or bad, uh, in terms of the quality.

Um, they ran the models through Cha GPT, through Co-Pilott, through Gemini, but then the tools themselves generated, uh, Python code that it was valuable for those who understood Python.

They could use that as a, you know, jumpstart to a much more sophisticated model, and so it still helps to be an expert.

It's just that you're so much quicker to on board, so much quicker to get started, and you don't have to search as hard, um, really.

And it gets to the second use case that I'm hearing a lot about that I think is really interesting, which is the use of agent AIs for things like consulting.

Um, and so the idea being that, you know, if you are, you know, an MBA graduate and you go into one of the big 3 consulting firms, you're gonna be given an agent AI and they're gonna be helping you generate your PowerPoint decks, do your data analysis, and the like.

So this, again, kind of a co-creation that we're seeing with these LLMs and in some cases, it's pretty sophisticated analyses that are being done.

Yeah, and I know that we're gonna get to the downside and the fears and uh the world is uh filled with concerns and fear from these LLMs, right?

They can also be the, the big bad wolf kind of around the corner, but, um, on the positive side, wouldn't it be nice if those consultants that get paid a lot of money, um, to do their, their work, wouldn't it be nice if they don't spend their time putting PowerPoints together, but.

Actually spend their time talking to the people in the ground and understanding how to, you know, utilize the workforce better or working with the leaders in the companies they consult and, you know, walk them through an implementation plan like there's so much, I think probably better ways to use those really um exceptional individuals than sitting down creating a PowerPoint.

So in that sense I think that these are good, good transitions.

But you know, you're raising a point that I knew we were gonna try to tackle here, which is, uh, this fear of a jobless future with the use of these tools and the like.

And I think you highlight something that we have at least seen historically over arguably since technology began, which is, as technology replaces human skills, we tend to then move kind of higher order, higher order human knowledge and the like that we're able to apply to these are different decision problems.

Like, what, what's your take on the, on the jobless future, uh, worry?

Well, um, maybe it's not a huge surprise to those who've been listening to the podcast and to you, um, and to the audience that I am, um, probably I followed down on the optimistic side.

I do think that there are some really interesting questions around governance and around thinking about how to integrate new technology into our lives, um, but overall I am very, um, optimistic in the sense that I don't think that this is a, a.

One swoop it replaces all of us.

I know as you do that technology and this digital disruption requires change.

It requires reassignment and thinking through the value and the workflows in an organization and thinking about how we operate and the operational model that we, we kind of, uh, deploy.

And so I think it's just an opportunity to rethink a lot of the basic. and to work smarter and work better, uh, in so many ways.

So I'm optimistic on the opportunities here knowing that there's work to be done.

It's not gonna just walk in there and everything will be solved, but I think that ultimately we're gonna end up in a pretty good place.

So that's my hope.

Yes, there'll be transition.

There'll be some jobs that will, you know, shrink in terms of the, the, the need and the demand, but there'll be plenty of new opportunities that will emerge.

I should remind our listeners, the title of our podcast is Good Disruption.

So we tend to be optimistic with technology that many, but that we do debate whether good, bad, or no, uh, no disruption.

Well, look, I think it's time for us to bring in an expert to help us with our thinking about LLMs and the like, and it is an absolute pleasure that I have with us today, uh, my friend, my co-author, my former colleague at Duke, Ronnie Chatterjee.

Ronnie, before you say hello, just a quick introduction here.

Uh, Ronnie is a distinguished professor of, uh, business and policy at Duke University.

He is on leave right now though, as the chief economist for OpenAI.

Uh, in addition to this, you think, you, you do have to take a number of leaves, Ronnie.

I'm very jealous of your, your your business Duke to let you, uh, venture out in the world.

He's also served on uh Obama's Council of Economic Advisor.

He was also chief economist for the Department of Commerce under the Biden administration, actually leading implementation of the CHIPS Act.

Uh, Ronnie is everywhere and, and, and knows everything, I think.

So it's just an absolute pleasure to have him with us here today.

So Ronnie, thank you for joining us.

Oh, it's great to be here and honored to be with AL and you, Mike.

And honestly, if that was true, if I knew everything was everywhere, you know, Shat GBT should make me feel pretty worried as well, because it can do a lot of those things too.

But it's been an amazing experience, a learning experience over the last year at OpenAI and glad to share any insights that I have from that.

Hey, Sophia, where are you in the world, Ronnie, right now?

I'm actually sitting in Mike Lennox's old office at Duke University here back that's home in back home in Durham, North Carolina.

So it really is full circle.

And when I started at Duke, Mike was my senior colleague and really helped me adjust to the university, to being a professor, uh, to thinking about the role in the world I want to play.

And I think both, you know, Mike and I, we've, we wrote a book together, we thought a lot about the impact of business.

In society, and, you know, working to OpenAI where the mission is to benefit all humanity.

A lot of these themes, Mike, that you and I talked about over the years have come into full circle.

And Yale, I know a little less, but I do know about her pretty amazing management ability and her ability to really get things done in organizations.

And I will say that um as someone who manages people and has to work in an organization, it's that part of my skill set.

I'm also getting to flex at OpenAI as well.

So I really think we're going to have a cool conversation about some of this stuff.

Amazing.

Birmingham, maybe we get fit.

Just tell us about your day to day a little bit.

Like, what, what are you doing?

What is the, you're the very first chief economist at OpenAI.

So what is, what is the job?

What are you doing on on a day to day basis?

Well, I think what happened early on is there was a sense of people who are closest to the technology, closest to building these models that they were going to cause really a lot of transformation in the economy.

And to your point about the disruption, there's a lot of exciting things that can come from that and there's a lot of things that are unexpected.

And that the dynamic that we need to plan for.

And so I think what happened early on is people said, look, we need to figure out what the economic impact of AI is.

And mostly it was technologists who were thinking about this and slowly it started to permeate into academic and business circles and people started to kind of call an economists to come up with ideas around it.

And there's a long tradition, Mike, as you know, uh, the economists thinking about the impact of technology on jobs and on society.

And in the AI labs, in the frontier labs, they want to think about this in a more rigorous fashion.

They also had access to a lot of data and insight about the capabilities.

And so, OpenAI decided to create this position to allow an economist to come into the organization.

To analyze the data, and we've done that in a recent paper, we can talk about it in a bit and also get closer to the actual capabilities to really get close to the model builders and the people who are doing the post-training, the reinforcement learning to understand where this might go, to build accurate forecasts and indicators to help people prepare.

So that's why.

They have a chief economist.

The way I've designed the job is probably, you know, more emblematic of some of my own personal skills and interests, but really, that is why they designed the job to find someone who could analyze the data, figure out what's going on with AI, and then build accurate forecasts and indicators to help people prepare for the transformation to come.

And do you, um, how much do you spend, uh, thinking about yourself as an academic versus an economist thinking about open AI problem?

Like, can you easily separate what you know is going on and the franticness in which academia is trying to absorb some of this, or, uh, or, or is it there in the back of your mind all the time?

It's always in the back of my mind, yeah, I always thought it's so interesting to think about, we all wear all these different hats and I, I realize at this stage of life and, and, and career, you know, I'm bringing my academic insight into this organization and you know, interestingly enough, I think you'll both find this interesting, uh, it's a very research oriented organization.

So in some sense, while people might say, look, you're a PhD, you spent a lot of time in academia, isn't it a strange transition to an organization.

Um, outside of that, it's actually very natural.

If you think about the people who are building these models and, and doing research on improving them, a lot of them have academic backgrounds, of course, in different fields than mine, you know, Klaus Vel and other people who are listening, but, um, these are people who are actually super, um, Invested in the scientific method who understand research and its purpose.

And so someone like me who's a researcher in social science, um, I actually have a lot to talk with them about and I find some of my best conversations are jamming about economics and technology and people across the organization.

That part of the transition hasn't been as difficult as one might have imagined, and I don't have to check my academic or research brain at the door.

In fact, it's always part of me in every conversation.

Would it be true, I imagine that there's a lot of PhDs walking around the halls of open AI, people with, you know, either computer science, data science backgrounds and like.

It's very true.

Although not every researcher has a PhD or a master's degree, what's really interesting is, you know, a lot of the work we've done in the economics of innovation charts the role of universities in training researchers and also contributing to innovation that eventually makes its way to industry.

And with AI there was a lot of early work obviously coming out of fantastic universities, but you really see now in many ways, you mentioned the paper from Google, the frontier of a lot of the research is now being done inside companies, which I think is very interesting to think about why that is true about AI.

It's something I want to explore in my research.

And so I'd say a lot of PhDs running around open AI, um, a lot of people without PhDs though who are doing research in very technical areas, um.

We're, you know, noticeably fewer though people with PhDs in social science, because that's a kind of a newer thing.

And so as the first chief economist, I feel like I'm often carrying the flag for the social scientists to make sure that um that we're actually part of that conversation as well.

And I think that's where, you know, publishing papers, we released the NBER working paper called How People Use Chat GPT.

I always like to have titles and tell people what the paper's about.

So it's how people use chat GPT.

And That paper has been just so interesting in building bridges across the organization because people can see for the first time now, here's what economic research looks like and kind of work that this particular person does.

And so it's been really fun to do that.

Of course, it's engaged a lot of folks outside of the organization who have been relying on the data to make really interesting charts and visualizations and predictions about the future.

So it's been a really amazing thing both internally and externally to write a paper.

Um, how are people using a good question.

I shouldn't have buried the lead.

So when I came in, you know, the first thing I wanted to do, and I had studied how economists had worked in other organizations, I thought about what would be the first thing I would do or what I want to do as an outside researcher and going to Yallows Point.

And I said, look, Lena write a descriptive paper about how people are using AI because what's really lacking for a lot of the debates out there is the actual data on usage.

A couple of other labs had released kind of interesting papers on this or sometimes reports, and I thought, OK, this is really interesting as a start.

What can we do to be part of that conversation?

What can we do to enhance that conversation?

So we took over 1 million chat conversations and used classifiers without actually reading the message to group them into different categories.

And one of the key innovations we made is we were able in a privacy preserving way to match those conversations to some demographic information, things like inferred gender based on the name, things like occupation, role, tenure at company.

Being able to put that together was really important.

And I hear from academics all the time who say, you know, the who and the what of AI is really important in your paper and the first that's ever done that.

And so that's the contribution I think to.

Field, the takeaways from that who and what analysis are the following.

Basically, we find that chat GBT messages on the consumer side, about 70% of them are for non-work uses and 30% are for work.

Those wedding speeches, those wedding speeches like Michael, we're talking about.

Exactly, wedding speeches, recipes, fixing my daughter's bicycle.

A lot of people came to us and said, gosh, that means chachi BT.

It's only 30% of the messages, you know, of course we have 800 million weekly active users, so it's still, still a big number, but they said 30%, that's not a lot for work.

It must be more useful for wedding speeches than it is for Excel spreadsheets.

But remember, this paper is not about the enterprise chat data.

It's not about the API data, which is obviously the maybe the primary use in a lot of businesses.

It's not Codex, which is the coding tool.

It's not anything else.

It's only the consumer.

And so if I told you that a consumer product like TikTok or Netflix, 30% of your activity there was, was actually for work, you'd be pretty impressed by how productive TikTok and uh and uh Netflix make you.

So, it's interesting to see the hybrid use of the tool, 70/30 non-work and work.

The last thing I'll say is 3 broad categories what people are using it for asking, doing, and expressing.

Asking and doing being far above expressing.

People are asking for information, for guidance, right, about what to do using chat GBT.

They're also using it, and this number should increase over time as a genetic capabilities become more widely disseminated.

They're using it to do things, end to end tasks.

When you think about it like that, and our co-author David Deming has a great way of thinking about this, a professor at Harvard and, you know, another person who helped us really produce an academic style paper, he said, look, Chachi PT is like a co-pilot.

It's a co-worker.

It's helping you accomplish things and inform your decisions, help you make better decisions.

And that's quite different than the discussion right now when people think about AI.

In the popular press, where AI is really now so far a compliment, not a substitute to what workers are doing.

Now, to your point about good disruption, it's not clear if that will maintain or if that's always be the pattern, but I really thought it was important to get a snapshot of what's happening now with a large number of messages on the world's most popular platforms.

That, that's the takeaway.

I love how you wove in that co-pilot there with the talk about Cha GPT, but, uh, we'll come back to that, um, as, as you may know, um, I lead a Darden, I co-lead, uh, uh, institute, the Lacrosse Institute for the business of AI, the ethical business of AI, and one thing that we think about and maybe a different kind of disruption that we're talking about is also the business model disruption.

So I don't know, um, if maybe I know that this is a little bit, um, of.

An aside, but can we talk a little bit about, um, OpenAI as an organization?

Like how much do you spend thinking about the, the organization of the future, the technology organization of the future?

Like, are we there right now it's pretty cheap to use chat GPT.

I know that some people complain about the cost, but it's not as costly given all the value that it potentially can bring.

What are the thoughts around, uh, disruption there for the business model of how this technology gets, uh, uh, shared across the world?

So I'll say two things.

One is, um, you know, when you come into an organization and, and Mike and Yell, you've taught classes like this where you start with the mission statement of the organization, and we teach it with, um, with a straight face, and we say this stuff really matters.

And then sometimes you walk into an organization and, you know, the mission is kind of an afterthought or it's not something that really is alive anymore.

And then there's some organizations, and I think you two feel this way about Darden and I feel this way about Euclid, to be honest.

The mission actually does matter.

People do cite it.

They believe in it.

And yeah, I think we're all three of us attracted to organizations like that.

OpenAI is that kind of organization, and that doesn't mean, you know, we always do the right thing or we're perfect or anything like that.

What it means though is this idea of sort of AI to the benefit of all humanity drives a lot of decisions.

So when You know, they decided to basically build chat DBT.

They had the API.

They decided now to build this chatbot that ended up having 100 million users in 2 months.

It was a really specific decision to make it widely available and free because they wanted everyone to have access to that kind of intelligence.

So in terms of business model, you know, like that really galvanized the very beginnings of this.

And if you talk to people like Nick Turley, who was one of the early people who was the p.m. for Chat GBT, that idea of benefiting all humanity, it's still really important today and guides a lot of the decision.

I think to your point though, as things evolve, the question is, OK, what comes next?

I think enterprise AI is going to be a really important sort of piece of the puzzle.

I think we're in business school, so people like us really think about this, your institute is about this, but I think a lot of people are not thinking hard enough about how important organizations are going to be in the diffusion of AI, the frictions in that last mile in deploying organizations and how in many ways, right, the development of AI really runs through private sector.

Firms adopting this and getting the ROI that they need.

So I think the role of business schools, you know, much to the contrary of the student who said he'd never come to class again in 2022, we're gonna, it's, it's as important as ever.

We'll teach different stuff.

We'll have a lot needs to contribute because no way that takes off without the private sector really embracing it and finding it useful in my view.

That's the underpinning for the infrastructure bet.

It's the underpinning for the productivity benefit that people are, uh, are thinking about from AI, how organizations adopt it.

How they build on it, that'll really also determine some of the things about how open AI plays in the market.

You can imagine a frontier model layer where we are building frontier models and other people are building on top of our, of our APIs, building enterprise applications, specific verticals like healthcare or finance or law.

And you can imagine a world where we build a general purpose technology that can be used in all those different areas.

I think it's, it's still sort of up for debate exactly which model is going to take hold.

We'll also see different business models going the other direction of the value chain.

You see us making deals obviously with regards to sort of semiconductors, we have energy.

That's another angle I think about the value chains.

I can think about like Mike Lennox teaching Class 3 of the strategy course where he thinks about value chain strategy.

Open data could be a case in that one as well and in a really interesting way.

It might already or it might already have been written that I wanna do like a double click.

One of the question is, where do you see the technology today and in the near future evolving?

I mean, clearly this has moved very quickly, you know, exponential improvement in the underlying technology.

This has led to a lot of fear, um, but yet we're at a point right now where I'm almost feeling the, you know, we're at that part of the hype cycle where, oh, we, we thought we would be, you know, AGI and the, the world would be controlled by these, these LLMs, and we're not quite there.

So, so what's, what's the current thinking around OpenAI in terms of that path to improvement that we're seeing?

I think it reflects on the biggest sort of lesson that I've taken, the biggest thing I've learned since joining.

People ask me all the time, you know, you, what's the thing that you, you know, change your mind on and what's the thing you've learned the most about?

I think understanding how quickly the capabilities of the models were accelerating is probably the biggest lesson for me.

You know, being a professor, being in public service, I understood.

That this was a powerful tool.

I understood some of the ways we could use it, but it wasn't until I really got into OpenAI and understood how the capabilities were moving against these evaluations of these benchmarks that are set that I really started to understand the pace of change.

And so I think the the attitude among folks who are seeing that data is that it is rapidly improving.

We released something called GDP V.

So we have a fantastic team um that does frontier eals and what eals are, and I, you know, they're really common in some parts of business schools and other parts not.

I didn't know as much about EDAs when I joined.

Think of them as basically, you know, evaluations of model capabilities.

When it comes to like a math quiz, you can think about evaluating the model on in an objective way, figuring out how many problems they got right.

When it comes to more real-world tasks, things you might do, uh, some of our MBAs might do, right, working in finance or technology or consulting, you can give it real-world tasks and ask AI to try to do those.

What you're finding is in those real-world tasks, AI is getting pretty good at actually doing things at a human level or in some cases be to humans with producing.

A spreadsheet, writing a legal brief, doing a market research report.

And so if you look at where it is today on GDP bow, roughly half of these what you might call really important digital tasks in the typical white-collar job can be done by AI and preferred by humans choosing the AI over a human output in those cases. 50% might not be good enough in some jobs yet, but one thing to look about is, look at that pace of change.

Over the last year, it's been a straight up into the left trend.

And so if you think about building the models and continue to do in the post-training to understand how to hill climb on those evals, you're going to see rapid improvement on these, on these items.

And so, that's the thing that people are paying attention to.

They're very optimistic about it.

And honestly, um, it's a, it's a research agenda that people are excited about as much as like any point or outcome, right?

Because at the end of the day, if we keep improving those capabilities.

Um, a lot of the changes that we're talking about in the world, the positive changes, thinking about innovation, curing diseases, right, solving a lot of economic and social challenges that becomes more in reach.

So I think a lot of optimism, a lot of excitement, but, um, but also a lot of humility that like, you know, the people are taking, people are doing this, you know, they've seen their field transform in the last several years and the impossible comes to the possible.

Everyone knows that at some point, you know, they can always be a time where you're not making as much progress as you were before.

And so there's a lot of humility around that too, and a lot of hard work going into it.

Any thoughts on that the AGI question, you know, artificial general intelligence.

There's, there's a lot talked about it.

I don't think I've ever seen a real concrete definition of it.

You casually, you see different definitions, like, you know, the one that I've been working with is like the ability for, um, for an AI system to perform economically valuable tasks, like, you know, that are not, you know, a, a majority of economic tasks, but better than a human.

You can argue what are economically valuable tasks and what it means to be better.

I think the key thing for me is, um, just because AI can do things doesn't mean it will do things or that we will let it do things.

It could be that AI is really good at teaching my kid pre-algebra, but I still might want a human teacher for many, many reasons.

And so I think um if you look at autonomous vehicles, the capabilities of Waymo.

Incredibly high in terms of their ability to sort of do self-driving cars, but it's still not disseminated all over the world, right?

So clearly there's some frictions in that last mile.

And so I think the AGI debate is an AGI is an important, incredibly important, but what's also important to me is seeing that really powerful AI is being deployed in organizations and improve people's lives, and that is, that's gonna be a really key focus, I think, going forward too.

I wanna pick up on that and talk a little bit, uh, I mean, in our podcast we often talk about the good and the bad and, you know, good disruption, bad disruption, and there are plenty of folks who have raised concerns and hesitations around these LLMs, especially when it comes, and maybe this relates a little bit to the autonomous vehicle example that you just gave related to responsibility, liability, and kind of understanding the implications of the, uh, potential decision making that ultimate.

Ultimately these tools get to have um uh once we empower them to make more decisions and not just make generate uh insights or predictions.

So what are you or um or OpenAI um together how are you thinking about those kind of concerns or those that critique and like putting some guardrails or like what, what, what kind of things are you, are you thinking about to protect us from some of the downside?

I think there's a few different ways, some of them I'm working directly on and other ones my colleagues are.

So in terms of me working directly, you know, when I came into OpenAI, I thought a lot about um doing studies, which I've been doing on the impact of AI on downstream outcomes, you know, let's say employment or, you know, sort of firm performance, um, and thinking about that as my primary role.

When I got to OpenAI, I realized that, you know, you have a, a model that is talking to 800 million weekly active users and people are asking it questions like, for example, uh around, let's say financial advice or which which things to invest in.

I thought, gosh, you know, my impact as an economist could also be in helping sort of shape model behavior so that we make sure that it's actually getting the right kind of advice based on the first principles from economics and finance.

And so for me, it's like thinking about evals and training the models becomes as important as the downstream studies I've been doing.

That's one way I've been thinking a lot about, for me, in particular, the advice getting factor of AI is really important.

I've written a lot of papers on How advice impacts entrepreneurship.

And I think advice is one of those things that is really scarce.

And if we could, good advice is scarce, the advice is not scarce, good advice is, if we could scale the good advice, think about all the businesses, particularly in emerging markets, but here in the US too, they're stuck between small and medium size that aren't growing if they can't afford a big consultant, right?

They can't afford to hire an additional person.

What if you could put that advice and scale the intelligence that we have.

Into AI and deliver that at a much lower cost or even free.

Those are the kind of things I think are really, really important.

Those are the kinds of things I think are gonna make this a good disruption if we can scale because I think when we get overly focused on sort of the disruption piece, we sometimes forget that like there's lots of people who don't have access to these services to begin with, who couldn't afford a consultant, couldn't hire a financial advisor.

That's where AI becomes powerful.

There are other folks, deeper experts and I obviously on these issues who are working on AI safety.

It's a big.

Priority and really animates the whole mission of OpenAI.

Uh, safety can come in many different ways, all the way from national security to sort of protecting our users and like you've seen a lot of our public, uh, sort of writings on things like syncopy and emotional dependence.

That's a key part of the work at OpenAI.

And the last piece is just like making sure people have the skills they need in the new economy, you know, it grows of how quickly the transformation, um, goes through, we want to make sure that people learn skills and they can match the jobs.

Requiring those skills and so we announced a partnership with Walmart to work on certifications around AI and hope that we can use those to make a more efficient matching market for employment so people can get the skills they need and they get jobs that they would want to use those skills at.

Those are some of the things that are going on around the company.

Definitely a lot of interest in this, um, animated by the mission and, uh, and a lot of partnerships, which I think are good because, you know, obviously we don't have all the knowledge on this stuff in-house.

Yeah, Jack and Rick, uh, come to mind that, um, your colleagues at Fuqua who are experts in advice taking and giving and like, uh, I can imagine that line of research really informing some of that.

So, uh, thank you for sure, and I encourage, I mean, I, I, I think if there are researchers out there listening to this who want to engage in some of these topics, uh.

Um, you know, we obviously have worked with a bunch of academics already.

We're running, you know, field experiments with firms.

We're, you know, writing papers like the how people use chat GT.

We're doing more practitioner pieces with things coming out in the next year.

So, um, definitely reach out and let us know if, if, if you want to work out on things.

That's great.

I see Mike has some questions from the crowd, right?

I do, and one I wanted to follow up on anyway.

So Ronnie, you and I wrote a book a few years back now called Can This is Save the Earth?

Trying to be provocative here.

We talk about climate change, we talked about clean tech, renewable energy, and the like, and We're at this incredibly interesting moment where there is a massive buildout of compute to meet our AI needs, and we need water, we need energy, we need electricity to, to power these.

And we're sitting here uh in Rosin, we can kind of see Tyson's corner.

Not quite wrestling, but we're, we're almost in the center kind of where a lot of build out has historically taken place.

You, you mentioned it briefly in your comments.

Can you just talk a little bit about how OpenAI is thinking about the need for compute and then of course the energy and everything that needs to come along with that to have sufficient compute power.

Yeah, I mean, this is, I'll, I'll be brief, but this is a huge issue.

We just do another podcast about it, um.

You know, when you think about all the things I've done in my career from the book we did together to the CHIPs Act, this work, um, At OpenAI, it kind of connects all those things.

So like we're talking about the full stack of AI infrastructure, starting from developing the energy from clean and renewable sources, all the way to, you know, powering these data centers that include the GPUs and high bandwidth memory and the things I was working on when I was in government, all the way to sort of the training and inference required on the frontier models that things like OpenAI and then into enterprise applications and the kinds of things that I've been teaching my MBA students about for a long time.

So it's all connected.

I think I see a couple things that are really interesting to focus on.

One is, um, you know, there's been a broader sort of push for electrification that's creating more demands on the grid already and investing in the grid and approving it.

And we were talking about this even in our book, Mike, and it's, it's becoming more and more important.

The other place is sort of like thinking about how quickly we can get energy generation set up and connected.

You guys are, I guess, in the PJM area, right?

We're one of the hot corridors when it comes to this.

And, and, and where there's a lot of data centers.

And so you're gonna see massive investments in the data centers around the country and around the world.

So we're gonna have to think about how we can do these things more quickly, um, because to meet the demand of what's going on.

Secondly, I think because so many hyperscalers and OpenAI are so committed to doing this in a sustainable way, you're gonna have pushes for more innovation in terms of how we do things like pool, data centers, for example, or recycle water.

And so what's interesting to me is when I talked to.

Engineering students at a place like Duke, uh, where they might not have been talking about sort of, you know, the power efficiency of chips or how to save more water at a data center, those are now topics that there's a lot of interest behind, a lot of venture capital money behind, a lot of research dollars behind.

So that's the hope is that we can drive a lot of innovation in this space and actually some of the things that, you know, around the energy transition and beyond are happening because of these big investments.

At the same time, it'll be different in different parts of the world, and so you're going to see a different mix in different parts of the world, different approach to doing this.

And so that's also interesting to remember the US is in isolation as we're thinking about how we develop this infrastructure.

You know, just a quick follow up on that.

Um, one of the ways obviously that we could prove this problem is just the efficiency of the ability to do, uh, LLMs, right?

The, the processing efficiency there.

Can you comment at all about kind of where we see the trajectories there?

Is this a, a case where, you know, uh, we'll have dramatic improvements in our ability to produce the compute we need for a given unit of, of output of, uh, of an LLM?

Yeah, I, I, I can't say that I'm deep in terms of right now the technical work being done around this, but I do think even just routing your queries to the right model could make a lot of sense.

Like you think about, you know, when I used to use deep research in previous iterations of the model, right?

And now, like if it's bundled in together in a GPT 5, you know, when I have a particular query, does it need to go to, you know, which model to really give me the answer I want?

Can I specify, hey, this is the kind of answer I want in a way that's also sort of.

Maximizing efficiency.

These are the things I think are going to be really, really important.

And I think there's gonna be huge opportunities both at the Frontier Labs, but also with a lot of intermediaries.

We're going to build businesses around this.

I think it's a really interesting space, but that, that's what I know based so far.

And what I like, what I like about that vision is that it would require, um, both the organizations and the developers of these models and the users to kind of invest in that this education so that the, the end consumer is aware and knowledgeable so.

That they use the models more efficiently and that that benefits everybody, right?

So I like the idea that if we're efficient, the organization's going to be efficient and then the infrastructure will be better set up to support it, right?

That's so interesting.

Yeah, and it actually relates to a lot of the early work that Mike and people in our community, I guess, did around kind of giving consumers, um, information at the point of sale in those cases about sort of environmental impact, things like that.

It's really interesting.

It sounds like a good class project to to explore, but very exciting.

Yeah, uh, I've got a question for one of our audience members that I thought is provocative, and I'd be curious your, your response to it.

Uh, what are some ways to prevent AI from overshadowing our own authenticity and uniqueness to the work we do and create?

It's, it's a really good question.

Um, I think about it a lot, right?

When the machines are ascendant, we worry somehow that humanity in a zero-sum games become less valuable.

I think it's interesting to think about though, that the machine is also able to do things that can level the playing field and make humans actually more valuable.

You think about how many people, they open up a job listing and it says, hey, you got to learn how to code or know how to code to do this.

And now, Um, somebody can say, well, hey, I might through Bitcoin be able to, you know, do what is needed in that job.

Just like when we remove college degree requirements from job descriptions, we find people to do the job that we there's a chance that actually more people will be able to express themselves and humanity in, in this world.

It's also interesting that people now can complement their skills.

If you think about someone with an idea who wants to make short films, and now they can think about using the tools with Sora rather than.

A lot of other expensive equipment to kind of make their dream into a reality, at the very least creating some, some template and visualize what it would look like before they move on to more sophisticated sort of technology.

So that's really interesting to me.

And I feel like these are places where we can express our humanity more.

Finally, and this is actually what Yaa was kind of saying to the student back in 2022, which I believe too is, and I think Darden and Fuqua are particularly adept at this, which is, You know, just because we have massive open online courses, right?

Mike has taught these, Yellow's probably done it.

I've done it.

Um, it doesn't mean that that human experience at a Gardner Fuqua is less valuable.

I'd argue it might be more valuable.

And I don't think that necessarily that's true for every school, but for the places that are providing the really great in-person human to human experience, people might be craving that and valuing that even more in a world where these things are happening uh with machines at the same time.

So those are the things that made me optimistic, um, but I, but I also, I also understand the concern and skepticism, and that's why, you know, humans have to be in the loop at every step, make these decisions.

Um, in a small democratic way, and it will be different around the world.

I think that's really important to, to remember, you know, I go to a lot, I traveled to 10 countries in 10 months, and, you know, the attitude towards AI differs sharply around the world in ways I think you're going to manifest themselves in interesting patterns going forward.

I can see that.

Yeah, gets to another question.

This might be a more difficult one to answer, but in the kind of geopolitics of, uh, these, these systems now and how there is this kind of mad race, especially between the US and China to develop out this technology.

I was in a conference in Europe recently.

The Europeans are all worried because they don't have their kind of big LLMs being developed per se, um.

Maybe if you just reflect from an open AI perspective, do you, do you talk about this?

Do you think about this, or is it more just like, you know, we, we got our, our job and we're, you know, kind of working on that right now.

I mean, I definitely think that the core job is to build, like, you know, really fantastic tools for our users, right, and making the product work better and of course, developing like the frontier intelligence that that makes it all go.

So that's definitely the core.

I do think though, the geopolitical backdrop is is really important for all of us to remember, you know, if you think about um If you think about the development of AI in a very provincial way, let's suppose just your community or just your country, you may miss that there are other places that are making different decisions that can affect yours.

And so, you think about um issues around safety, it's gonna be really important, right?

To, to make those decisions at the local, state, and national level.

There's also a broader issue about whether the decisions align what people are doing in other parts of the world.

Uh, we might seek to say, hey, we don't want to use AI in this instance or that instance, but it doesn't guarantee that they're going to make those things in other parts of the world.

And the other place that really matters, which is not, you know, my area of deep expertise, but of course, the national security and defense.

And so to the extent that AI is leveraging those applications, um, it really, really matters who's doing what in those, in those, in those capacities, who has access to the, the, the world's leading intelligence.

And so we have to think about as well as navigating the challenges like in our own local context about how we, we choose to use.

AI and I think it just expands the conversation.

There's not an easy answer to it, you know, and I think your conversation in Europe reflects the ones I've had where, you know, they're thinking about, hey, you know, the, the, the most powerful models being developed places outside of Europe.

What does that mean for us?

Um, it's also a really great opportunity in Europe to build on top of the models in really important applications that are key in Europe and build great businesses around those.

I think there's some reason for optimism there too.

But this is the conversation around the world that's happening right now.

I don't know if you, um, have you read this book AI Snake Oil?

Yeah, yeah, yeah, yeah, yeah, yeah, um, so the premise here, uh, uh, by, uh, you know, um, researchers at Princeton, um, uh, part of the premise is that the invention, what you're doing at OpenAI is really the core technology, the, the capability that's being developed by OpenAI and other companies in the space, but the question is like, where is the, the application, right?

Where is the.

Kind of the business kind of specific application gonna come from because that's when it's really gonna take over that that's gonna prevent it from becoming really what you know live up to all the promised unless it's really implemented in in the next level of applications um is it the Walmarts of the world is that, is that where we're gonna see some of that come from or are there new companies that we're gonna see emerging that are gonna kind of make that their.

Like, I wonder how you're viewing that.

I, I see it both ways.

It's a great question.

I think that there'll be some leaders in their industries who are going to really work on this, get this, figure out the right workflows to use AI on, do the kind of change management.

Another reason we still might want to go to business school to learn how to do these things.

Yeah, you need project management for that, right?

You definitely need it.

You can't just drop these things in and expect the whole, you know, company to change.

I think people are finding that out, right?

And so I think it will be some of the leaders in their industries who are doing this, but I also think it's gonna be some really exciting startups who really don't have to re-engineer workflows because they're starting from scratch.

So they can start from the beginning and build the workflow around AI.

This is what you call these AI native startups.

A mix of those companies, of course, it will differ by industry are probably going to be the ones to kind of get to where we need to get first.

And I agree with the idea that, you know, as enterprises adopt AI, that's we're going to really unlock a lot of the value people are talking about.

You're talking about the growth in terms of, let's say, GDP from AI or mostly Thinking about firms becoming more productive using AI.

We're obviously seeing a huge investment from the CapE side that that's also impacting PDP, but what we really think about with growth in AI is firms adopting and using that.

I think it's an important step.

And so that's the space I'm watching.

And there could be particular countries, geographies, um, that are really sort of disseminating AI more quickly and getting more people to use it for useful things, and that could be a huge advantage in the future.

So that's why I think we got to make sure that we get the technology in as many hands as possible so we can figure out what those uses are.

Thank you.

I always like to call when it comes to, uh, digital technologies, whether it's the personal computer or the internet or now, uh, AI, there is often a lack, a significant lag before we really see productivity gains in the economy and a true impact on like GDP in the order of sometimes like a decade plus in some of these examples there.

So it might be too early yet to full, fully see that impact from from chat GPT and these LLMs.

I have one more question.

One more question here from uh one of our audience members.

I really like this.

What do you see as the bottleneck for realizing full impact of uh these systems and AI at this point?

I'll just share that, you know, when I talk about the AI winner of the 1990s when I was a student, we always talked about it, we didn't have either the data or the processing power.

So if you look forward now, is it data?

Is it the building of data centers?

Is it a processing power?

Is it, we need a new algorithm, like, where do you see the, uh, the things that might constrain us moving forward?

So, and I'll, I'll answer this and also pick up on something about your last point about when it shows up in productivity statistics.

And, you know, what's really interesting about AI and our paper kind of documents some of this is I think a lot of the value is in what economists call consumer surplus right now.

You know, a lot of the things like if you think about AI informing the decision that you're making, and now you're going to make a better decision, or you're giving that wedding speech and you're saving time you would have spent on that, and now you can do something else.

That doesn't show up in GDP, but that is improving your life, and that is what economists usually think about in terms of consumer surplus.

And so there's a lot of positive consumer surplus from, from AI.

Erik Mjolfsson and co-authors have written about this and estimate, you know, it could be $100 billion a year from AI just given what people would be willing to pay to get it back, that kind of idea.

So, I think that's interesting.

And I think when you bring that to the enterprise, it's creating value, but maybe not in all the ways that people can see right now.

I think the biggest barrier is probably um sort of Is their organizational willingness to prioritize these projects and go beyond pilots and really re-engineer or reimagine the workflows in their organization?

I think if people are stuck in sort of the SASS model and they think they're going to drop this in and all of a sudden see magic, it's probably not the right approach for AI and I think that's what makes it different.

And so I think that while some companies are unlocking this more over time, and I think that's really the barrier, the friction, are going to figure this out.

And when I join a lot of calls with our folks who are working with our largest enterprise clients.

I always kind of um upgrade my probability of success if I see more senior people on the call, which, which might be interesting, right?

You think it's the junior people who are more tech savvy, but the more senior people are able to really prioritize these AI projects, they're able to pick the right implementations, they're able to scale the pilots, they're able to do the change management that really need to make sure AI implementation sticks.

When you see that in organizations or a leadership commitment, then I think your chances of success are a lot higher.

Uh, there is a lot of work to do, but I'm pretty sure the biggest friction is time.

And that's uh interesting because it's the domain expertise.

Those senior folks have the domain expertise.

When I meet with organizations and talk about AI, uh, you know, I often hear comments around like, yeah, we're just gonna hire a 25 year old and tell them to go and do it.

Like that's not gonna be the way to onboard AI in your organization because they don't know the business and you need to know the business to know how you can change and break the business to to start to leverage AI, right?

I, I agree.

That's been our experience.

Yeah.

I, I always say, you know, we need digital champions, those people who can be translators between the ability of the technology, but understand the business function, you know, exactly to the point you made there.

Well, Ronnie, we're out of time, but any last words, any last observations for us before we end?

Well, first, I just thank you to Mike and Yale for welcoming me to the podcast.

They choose me to be here and exciting.

Also, I've always had such great experiences interacting with Darden and the University of Virginia.

Uh, and having been to the campus that you're seeing at Mike a couple of times, uh, it's a fantastic space.

So I'm, I'm really glad the space you're creating, uh, digitally, the space you're creating physically, and all the amazing things you guys are doing in the world.

Uh, I hope to come back for a future episode and we'll talk about where AI is then.

Hey Mike, it's good to, uh, good to reflect on our wonderful conversation that we had with Ronnie the other day.

Yeah, this is great, you know, we've had a couple of days to reflect.

We're back in our offices now.

Uh, the big question, good, bad, or no disruption?

Yeah, what do you think these LLMs?

Yeah, LLMs, uh, I mean, I think we agree, we are all can agree that there is a disruption, um, happening.

I think we feel the panic in the air, the enthusiasm in the air, the Uh, even the doubt in the air comes hand in hand with big disruptions, right, like that we've seen historically.

Yeah, I totally concur.

I mean, I think it's, it's interesting, like, is it overhyped?

Perhaps, but, you know, as we talked about, I, I, I think there's already enough evidence to, I think, officially declare this a disruption.

Uh, yeah, the, uh, the, you know, amount of adoption.

And even though it's not, you know, penetrating every industry and every sector, there's been enough that are already like widely adopting this, and I think it's, like I said, it's fair to say that the disruption has begun.

And so, you know, I think I was probably already sold before we talked to Ronnie, um, just, uh, cause I am intrigued by the the opportunity and the capabilities of that these models bring, and the imagination, how it gets people excited to think about what is possible.

Um, so while I think like any disruption, any good disruption, there are, of course.

You know, ongoing ethical questions and environmental questions and impact questions, and it's not gonna be purely good across the board, but I think in totality this is a shift for the better.

This extends our, uh, capabilities as humankind, um, in, in many interesting directions and that we're just finding out, um, but for me it's a good disruption overall.

Uh, I'm, I'm aligned as well, um.

You know, it, it, it's challenging whenever we have a new disruption, um, in many ways it's human choice that decides whether it's good or bad.

Um, and, uh, I guess I, I have faith in humanity that will, will navigate this, um, that yeah, it will create some challenges, um, ethical challenges, use cases, you know, the idea of being addicted to your LLM is something we're gonna have to head on, um, but what often happens when we have these disruptions is there's this co-evolution of the institutional environment.

And the technology itself.

And I think we're just at the beginning of that with these generative AI and LLMs that, um, you know, I remain optimistic at this point that we're gonna, that we're gonna get there, that we're gonna get to a more positive space, uh, in the adoption of them.

Yeah, I agree, and I'm hoping that um it will some of the concerns that we see or some of the open questions that we're debating or that we're concerned about, you know, how do we provide energy to uh to data centers, how do we have enough water, how do we figure out some of the infrastructure, um, we talked about those in this, in our podcast in the past and.

Um, other types of disruptions that can have come up, uh, but this is a good excuse, a good reason, a good motivation, a good, um, uh, real enhancer of those types of, uh, solutions, and I'm, I'm looking forward to, to us getting closer and closer to getting our hands on how do we make energy more available, more distributed, cheaper, um, and less, uh, problematic.

So, you know, your, your systems engineering roots are coming out here, like everything is a giant system, and maybe we should do a podcast just on good disruption, everything that's happening right now, you know, how is the system gonna advance here.

Again, I, I'm aligned as well that I think, you know, all of these problems, not to be true Pollyanna, but we will put our attention to them, we will hopefully make some progress on them, um, and it's, it's just, it's, it's a unique time to be living right now with, especially with AI.

Yeah, yeah, great.

Well, thanks again, Mike, for having Ronnie and for allowing me to join your conversation.

He was a great, of course, of course.

Well, as always, I wanna thank our producer Gary, our chief, uh, uh, researcher Becky, and the entire team, uh, for bringing you good disruption.

Thank you all.

Thank you all.

Good Disruption is a podcast from the University of Virginia Darden School of Business.

Darden