Decisions Over Decimals: Professor Netzer

Oded Netzer Transcript

Daniel: We need to talk about Relationships, and we need to talk about your relationship with data. I’m being a little silly here, but listen, you have a lot of data at your fingertips. As a school leader, is it helpful? Is it guiding your decisions? Is it overwhelming? Do you even know what to do with it? Well, today’s guest is sort of a Data expert, and he wrote a book called Decisions over Decimals, and he’s here to unpack how leaders can best leverage and use data. I hope you enjoy today’s conversation.

Hey, it’s Danny, chief Ruckus Maker over at better leaders, better schools. I am a principal development and retention expert, best selling author, and I host. Two of the world’s most downloaded podcasts. This show is for Ruckus Makers, which. Means you’ve made three commitments. You commit to investing in your continuous growth, you’ve committed to challenging the status Quo, and you’ve committed to designing the Future of school right now. We’ll be back after some messages. From our show sponsors.

Hey, Ruckus Maker, I’ll make this quick. If you’re listening to this message right now, you’re missing out. When you subscribe to the Ruckus Maker newsletter on Substack, you get access to micro books focused on how to do school different tools and other resources that will help you make a ruckus and do school different stories and case studies of the world’s most legendary Ruckus Makers of all time. Access to my calendar to schedule coaching sessions, and you’ll also get bonus podcast content that won’t be released on the main podcast feed and podcast episodes without any advertisements. So if you love this show, if it’s helped you grow and you want access to more tools and resources that will help you make a ruckus in school different and become a paid subscriber at ruckusmakerssubstack.com. That’s Ruckusmakerssubstack.com.
I’m sure you’ve heard that energy flows to where attention goes. If you want to get more of what you want, when you want it, as a school leader, I’ve got a tool for you. The secret is to celebrate the positive. Things happening on campus and to go multiple levels deeper to tap into why. It even matters when you do that. Anything is possible on your campus, and I mean anything. And you start to get more of what you want when you want it. If you’d like to spread more positivity and create more value for all stakeholders on your campus, go to betterleadersbetterschools.com/positive and download your free tool today. Over 1 million teachers rely on IXL because it’s empowering.

It helps them make better decisions with reliable data, and it adapts instruction based on student performance. Get started [email protected]. Leaders. That’s ixl.com leaders. How would you like to increase student talk by an average of 40%? More student ownership, more student discourse? Check it out for yourself by trying out TeachFX. Go to teachfx.com betterleaders to pilot their program today. Ruckus Maker if you’re ready to make a real impact schoolwide, get a copy of executive functions for every classroom, for every staff member on campus, and engage in a book study campus wide. Head over to organizebinder.com/book to get your copy of executive functions for every classroom today. That’s organizebinder.com/book.

Danny: Hey, Ruckus Makers. We are here today with Professor Odad Metzer, who is the vice dean of research and Arthur J. Samberg professor of business at Columbia Business School, an. Affiliate of the Columbia Data Science Institute and the author of decisions over decimals. Professor Netzer is a world renowned expert in data driven decision making and extracting meaningful insights from data. He wrote dozens of papers published in the top tier academic journals. His award winning research is broadly read and highly cited. He is an award winning teacher as well at Columbia Business School’s MBA, executive MBA, and executive education programs. Professor Netzer, welcome to the show.

Professor Netzer: Thank you, Danny. And thank you for inviting me with this introduction. It can only go downhill from here.

Danny:I know that the Ruckus Maker is watching or listening is certainly in for a treat. And one of the things that I was excited about in terms of bringing. You onto the show, they like to. Say data rich information, poor school leaders. Are immersed, absolutely immersed in data. And I don’t know that we actually use it all that effectively. I want to hopefully leverage your wisdom and experience. To help out the Ruckus Maker. Watching or listening? But the opening story where I want to take us first is to have you talk about the story of Coke. Introducing a new flavor. That’s a bold move to do and why they were asking the wrong question when they introduced that flavor.

Professor Netzer Yeah, that’s a story that goes way back. I’m a little bit disclosing my age. Knowing this story going back all the way to circa 1985, where Coke. Was introducing a new flavor and in fact, the story backs from the few years before that, where Pepsi repeatedly showed that in blind taste tests, Pepsi tastes better than Coke. Meaning people who did not know what they’re drinking were blindfolded, or they weren’t. really blindfolded, but they were given a cup that just had the drink and didn’t know what they’re drinking. When they were drinking Pepsi or coke, they really preferred Pepsi over Coke. People preferred Pepsi over, people at Coke, they actually replicated this stat. So Pepsi went with this on advertising. And of course, the executives at Coke, at Coca Cola lost sleep, as you could imagine. And it really bugged them because they have run the same experiment themselves, and they found that it’s true. People in giant taste tests did prefer Pepsi over. By the way, they do. These tests are also interesting, maybe for the listeners, the way it is often done. You do a b and a C cut. Two of them are the same, one is different. And you don’t ask people only which one they prefer, but which two are. The same and which one is different. To see that people can actually tell the difference to start. But on average, people preferred Pepsi over Coke in these blind taste tests.
And because that’s bothered the executives at. Coca Cola so much, they went into the R & D labs and started creating a drink that people would beat Pepsi In these blind taste tests that would Taste actually better than Pepsi. And they created the product. It was called New Coke. That’s how they named it, because they. Wanted to be Coke plus. They wanted to keep the brand name. I mean, Coke was a big brand name. So they would keep that, the new Coke, and they went with it to the market. There were literally riots in the street. People went out demonstrating, we want back our old coke and the one that we’ve been drinking for generations over generation, kind of the American heritage drink. We want back the classic Coke. And Coke realized what’s going on. They went back on their decision and Said, okay, we’re going to bring back the classic Coke. After people again literally rioted industry. Now, there were actually millions of dollars in marketing research and research being done to come up with this new flavor. So what really went wrong? I mean, how could the company go so wrong? We’re talking about a multiple million dollar type of decision and the reality is that Coke asked the wrong question. And you’ll see that common thread to the way I think about thinking about being data driven is. Around asking the right questions.

And Coke asked the wrong question. The question that Coke asked was, well, if you go blindfolded, which one tastes better? But we actually do not go to the supermarkets blindfolded. We go to the supermarkets eyes wide open, looking for the product. And the reality is that when our eyes are open, people do want Coke, not just for the taste of it. Again, for the history that goes with it, for the drinks that the generations have been drinking.
And that truly was the difference. Right. The wrong question was to ask, what do people prefer when they’re blindfolded? Because we rarely are blindfolded. Actually, it doesn’t matter what we prefer when we are blindfolded. What truly matters is what we prefer when we get the same product again that existed there for many generations. And eventually again walked back. There is some argument, by the way, that suggests that the whole thing benefited. Both Pepsi and Coke because sales for. Both went up after that. But I don’t think that was intentional. It was really asking the question in a well conducted marketing research. The research itself was well conducted. That’s really interesting. I remember the blind taste test. I remember seeing those commercials and yeah, certainly interesting.

Daniel: Yes, it does date us. That’s okay. It seems to me that context matters a lot. Because you said we walk into the Supermarket, the grocery store, we are not blindfolded. And we’re bringing our history family, we. Look at the marketing, whatever, all this kind of stuff to make this decision. With our eyes wide open. I’m just curious, do you have anything else to add in terms of how. The Ruckus Maker watches or listens when They’re thinking about data and questions too, like the right questions to ask. How does context play into that? And how can we avoid sort of the blind spot that coke had and avoid the wrong question?

Yeah, I think that this is not.

Me saying this is maybe the smartest person ever lived. Albert Einstein said that. It’s not that I’m smart.

I’m just spending more time with the problem.

We should be both as, again, leaders.

In education, but also our students. We should be thinking carefully about the problem. We should be spending more time with a problem.

We tend to have the tendency to jump straight into solution, to go straight.

Into solution mode, to ask ourselves, how can we address it? Right? Because sitting and thinking about the problem doesn’t feel like war looking at data, analyzing things.

Or again, if you go back to.

Our students and we tell them, oh.

Here’S a math question, and just want them to provide the answers.

Right. Spend time thinking about the problem. In fact, if you look at. Again, let’s go back to our students in education, most of the mistakes come.

From people reading wrong the questions, not from really not knowing the answer right.

Not spending enough time with a problem. And as you are thinking through the.

Problem, you are likely to identify different angles, different dimensions to the problem, like.

The one that coke has missed.

Someone would have at some point said, well, that’s fine, but should we also.

Run tests that are not just taste tests, but taste tests that actually are not blind taste tests, and realize, yeah.

We want to taste better for blindfolded.

But let’s make sure that we also.

Taste better for those who actually know what it is they’re drinking.

But more generally, I think we should cherish the skill of asking questions.

In fact, in this book that I’ve written, decisions over decimals together, by the way, with two colleagues of mine, Chris.

Frank from American Express and Paul Manione from Google, one of the things that.

We are arguing is that maybe we.

Are doing job interviews all wrong. The way we do job interviews is we ask questions and we hope that.

Whoever is on the other side going.

To give us really good answers. But if you buy my argument that we need to be good at questioning it, asking questions, then it means that we should be testing for it. When we interview a person, we should see if they’re good question askers. Not just responding to questions. I don’t mean questions such as, well.

Tell me, what is the culture in this school?

Right? I mean, this is an easy question. I’m talking about at least easy question to ask, not necessarily to answer. I think about a question in which.

I give you a scenario, and I.

Want to know what questions you need to know.

We have this problem in the school.

Here is a scenario. Now tell me, what not just solve.

The problem, but what question?

What do you need to know in.

Order to answer this question?

What do you wish in you in order to answer this question?

Yeah.

Well, I think, correct me if I’m wrong.

Please. But it sounds like what you’re asserting.

Here is that by a person demonstrating.

The questions that asked not just how.

To solve the problem, in this scenario.

They are giving you a glimpse on how they think.

Right. And if you know the quality of how they think, then you can make some guesses about the type of value or what they would bring to the.

Table, so to speak, in your building.

Is that correct?

And then how they would inter.

Definitely. And how they would interact with others.

Right.

By asking questions, by being more of.

A part of a community by seeking information, not just by providing answers.

We don’t expect that anyone has all.

Of the answers, but I do expect.

People to know how to find the answers, which often comes with good questions.

Yeah.

How would they figure it out? Perfect. I think you have another story too.

About a firefighter who saved his team from a collapsing.

Yeah, we talk about this in the book, and actually, it’s not even our story.

It’s coming actually from Malcolm Gladwell in his book Blink, where we talk about these blink moments that people have. And this actually story actually relates to.

A framework that we talk about in our book in decisions over decimals, where.

We call it quantitative intuition, which kind.

Of sounds like an oxymoron. How can we take quantitative and put.

It together with intuition? I mean, aren’t people like right brain, left brain?

I’m good in math, I’m good in.

More of humanities and so on.

Our argument is actually, if you want to deal with data, the only way.

To do it, particularly at either a.

Leadership level or I would argue even.

Again, teaching kids in school, is to actually combine the data with a good.

Sense of judgment and intuition.

And we use for that the story of the firefighter, where the story there.

Is a firefighter is rushing.

It’s a true story.

I mean, a firefighting squad was rushing.

Into a burning building, running straight into the kitchen, because that’s where most fires are. And then the leader of the fire squad going into the building, looking around, check that. They look at the ceiling, looks at the walls, collecting information.

Again, data in the world of firefighting.

Is that you are assessing the environment, the context, as you mentioned before, Danny.

And as they are standing there in.

The building, two minutes later, the leader.

Of the firefighter squad calls into the.

Radio and say, everybody out right now. They leave the building. And about a couple minutes later, the building collapsed. So they interviewed him and they say.

Can you tell us, how did you.

Make this February leadership decision? I mean, truly saving the lives of his squad? And he said, I don’t know.

A lot of the things that I’ve seen in this building, I’ve seen before.

And I never called the shot.

But they argued further.

They said, but truly, can you still help us? Can you still think through what was.

It here that caused you to call the shot?

Because we want to teach future leaders to behave in a similar heroic way. And he thought about it, and then he said, well, now that I think.

About it, there are a couple of things that surprised me in that building. The first one was that the level.

Of heat that I felt did not.

Match the fires that I saw with the eyes.

There was less fire than how od the building was. And the second was he said that it was quieter.

Usually you go to a burning building.

And it’s super loud from the fire said here it was quieter, and that was like, I don’t know what’s going on here, but something here is wrong.

So now that I’m standing outside, I can tell you what happened. I didn’t realize that then.

It was much more of a subconscious intuition.

I can tell you exactly what happened. It is the fire was in the basement. Because the fire was in the basement. The fire went up in the wall.

It was hot, but I couldn’t see the fire.

It was also less noisy than usual.

And that’s why the building collapsed inside into the basement.

Usually. Again, fires are in the kitchen.

This one happened to be in the basement.

This is an example of what we call quantitative intuition. Again, you may ask, where is the quantitative? The quantitative come more from the way, again, you collect information in that environment.

But looking for these surprises.

And the argument here is you’re looking at information. Again, information can be data.

Information can be a parent comes and.

Tell you something about their view about the school.

Information could be teachers talking during the break, and you collect information there.

Look for the surprises. Look for the things that don’t match the pattern, not necessarily the things that.

Do match the pattern. And we tend to so often look for the patterns, because we are trained to look for patterns, cherish surprises.

Surprises are super important.

In effect, if anything, surprises are the places where we should be investigating further the pattern. We know the pattern, and that’s exactly what we expect. Spending the time truly at the surprises would help us either find problems or really good insights. Either way, you benefit.

Yeah.

So explore those outliers and get curious about why they exist.

The patterns we’re familiar with to some extent.

I love that you use the word outlier, by the way, because statisticians, right, or mathematicians call these outliers.

And what do we do with outliers? We throw them away or we put them in the appendix.

We need to actually focus on these as opposed to, again, outliers have a negative connotation to them, something we put aside.

Right.

Our argument, my argument is bring them actually to the center, and that’s where.

You spend more time looking at these outliers. Why are they the outliers?

Brilliant.

Well, I use that word for you.

Odette, so I’m glad it landed well, thank you. For sure.

I want to talk about students and.

The student experience with math.

Right.

So one thing that I want to.

Explore with you and I think this would be really helpful for the Ruckus Maker, watching or listening talk to us about just teaching kids to guesstimate.

Right. And why that’s so important.

Yeah.

And I really want to provide maybe.

My kudos to the current education, at.

Least where my kids are going to.

School, that I have seen that the.

Education around math has changed and has.

Changed towards valuing things like guesstimating.

I actually have seen my kids coming.

Back home with homework like how much is 209 and one times 29, right.

Where the idea really is to tell.

Them do not solve the problem but rather guesstimate the number. Right.

So if you again, you give me.

201 times 29, I should be figuring.

Out that I’m in the range of 6000. I don’t need to know the number.

In fact, there are very few occasion.

In life when I ask you that.

Question of 201 times 29 that I.

Actually expect you to solve the math.

As I’ve learned in school, when I need to know the actual answer.

If you really need to fit a.

Closet exactly into the hole in the.

Wall and I need to know exactly.

The distance, yes, decimals may matter. But in most problems we need to solve most places where we do use math, all I need is a ballpark of the number. In fact, I would even argue that.

In many cases, when we do provide.

The exact number, we are not just wrong, we are misleading.

To give you an example, let’s imagine.

That you’re creating now a forecast of the school’s budget for next year.

And because you’re using excel and you’re using multiplication of different things, you’re going to tell me that the school budget.

For next year going to be 800,039.

That’s projection for next year budget. We know it is inaccurate. There is no way you would know that the budget for next year for.

The school at the level of the decimals after these decimals, right. It’s not only wrong in your forecast.

For next year because you know it’s wrong. There is no way you would know.

It to that level.

It’s actually misleading because you give me.

The perception that actually you know the.

Budget for next year down to the.

Decimals, which we know you don.

So when we create even these sophisticated.

Excels, we actually should be rounding up to the nearest thousand, maybe even 10,000.

Depending on your level of confidence with.

These numbers, or maybe even providing a range. And again, the way I’ve learned math.

Unfortunately did not include guesstimating at all. It included math has right and wrong answer, and you better give me the right answer, otherwise I’m going to deduct points.

Right.

Whereas again, if you think about how.

We use it in life, math, most.

Of the time we use it at.

The level of guesstimating. We use it at the level of.

If you give me around the right.

Number, I’m totally happy. And I really think that schools that do that are moving in the right direction of educating the next generation to be feel comfortable with guesstimating to, we call it kind of t shirt sizing. I mean, there is a reason why for t shirts we provide this small, medium, large, extra large and so on.

Because that’s the way it needs to fit the body. I don’t really care. At the level of the small differences, when we buy a dress shirt, it’s not good enough. I need to know it to a finer level. We should teach similarly when we teach.

Math, not just the outcome, but what.

Are we going to use it for?

And therefore, how accurate does it need to be? And again, I want to congratulate these schools that already have taken this step.

Listen, today’s education and encourage those who.

Didn’T to adopt this method of teaching students to guesstimate. I think it’s a truly important 100%.

I love to pad my numbers and.

Always estimate a little higher because then when I typically come in under budget.

I’m always pleased.

Because I’m always padding it a bit. But I don’t know, maybe it’s a.

Bit of a cautious sort of conservative.

Approach to forecasting and budgeting around certain.

Items, but that certainly helps me.

So I’m resonating with what you’re saying.

Well, Odette, I am loving our conversation.

When we get back, I want to talk one more math question and how.

To fear people, or, excuse me, relieve people, right. And free them from the fear of.

I don’t know how to do math and I want to get into AI because that’s a hot topic these days and education and hear your take on that as well.

You know, something that drove me nuts as a Ruckus Maker was hearing teachers.

Say I taught it and the students should have learned it, but really, some teachers just don’t know how to reteach.

So that all kids get it.

That’s where IXL comes in. IXl’s diagnostic automatically identifies knowledge gaps for teachers and provides them with a personalized growth plan for each individual student. Teachers can step into the classroom every day knowing what their students know and what they don’t. IXl’s adaptive platform makes differentiating instruction easy. As students learn, IxL adjusts to the right level of difficulty for each individual. Close knowledge gaps and accelerate learning with IXL. Get started [email protected]. Slash leaders that’s ixl.com slash leaders. What do you see in your classrooms.

And how did you see it?

As a principal, you can’t be everywhere.

At once, so how can you help support every teacher in the building? With TeachFX, teachers can gather their own feedback without relying on classroom observations. The TeachFX instructional coaching app is like giving every teacher their own instructional coach.

Whenever they want it.

Ruckus Makers can pilot teachfx with their teachers. Visit teachfx.com betterleaders to learn how. That’s Teachfx.com betterleaders. Have you read executive functions for every classroom yet? It’s a game changing resource that can help you transform every classroom on your campus and your students lives. If you have any teachers who are struggling to engage their students, then executive functions for every classroom is the book for you. This groundbreaking book will equip you and your teachers with the tools to help students develop foundational executive functioning skills, enabling them to plan, focus attention, remembering instructions, and juggle multiple tasks successfully. Executive functions for every classroom is a game changer, and it’s here to help you take your school to the next level. Visit organizebinder.com book and click on the link to order a copy for everybody on your staff. Let’s do school different together.

Head over to organizebinder.com slash book and get your copies today.

All right, we’re back with Oded netser.

Professor who is also an author.

He wrote decisions over decimals.

You could go over to dodthebook.com and.

Pick up your copy, which we highly recommend. And were talking math. We were talking students and their ability.

Or inability to guesstimate.

And we also want to free people. We want to free Ruckus Makers and.

Their students from the fear of, I don’t know, math. Right? A lot of kids might feel that way, maybe even adults.

So, Odette, what would you say? How can we?

I mean, to be honest, it’s one of the reasons that we wrote the book was exactly this myth.

There is this myth that people are afraid of using data because they’re saying, well, I wasn’t top of my class in math.

I’m not an excel.

I’m not this type of person. I’m more of the right brain, not the left brain type person.

Sure.

And we believe it’s a myth. It’s a real myth.

Meaning there are skills needed to be.

Data driven, to be able to rely on data. In fact, I often say that I have good news and bad news. They start with the bad news.

The bad news is it’s not a.

Choice whether to make decisions with data.

I mean, us as rockets makers.

Our students eventually will need to use data.

This is reality. The good news is you don’t need.

To be top of a glass in.

Math to do it. You don’t need to be real to have really good math skills to do it. One approach we already talked about guesstimating.

Learning how to guesstimate does not require solving logarithms in your head.

Does not require solving square roots in your head. It requires really simple way of operations, which our brain is often wired for. So these guesstimating type of tools help free us from the need to be correct, from having, again, doing fancy math.

In our head, to figure out how.

Much is 2001 times 201 times 29. The other thing with numbers or with quantitative skills, in order to evaluate a number that you see, you rarely need to know math.

What you do need to have is something you mentioned before, Danny, which is context.

And context we learn from being in.

The environment, from seeing things around us.

So that when we look at a number, we are saying, I don’t know what you’ve done wrong, but this number is absolutely wrong. And the reason why I know this.

Number is absolutely wrong because I’ve seen.

A few other numbers, I’ve been in.

That area, and this is just not correct.

And this requires very different skills. In fact, I would argue that it’s the skills that those who spend most of their time with math, I’m thinking data scientists or statisticians are actually lacking.

Context because they’re focusing more on how.

To calculate the numbers and less so about the environment.

Evaluate being what I like to call.

Fierce interrogator of data.

Fierce interrogator of data are not those who interrogate it more from a mathematical.

Point of view, but rather from the context point of view and saying, I.

Know that this number is wrong. Again, I can’t tell you what mistake.

You’Ve made in your calculations, but I.

Look at this number and I can.

Tell you it’s wrong. And that’s a very important skill that the people who believe, oh, I can’t.

Do math, are often afraid to employ because they say, well, I don’t know.

How you calculated it. It shouldn’t matter. In fact, what should matter is that.

You know that this number is wrong.

Or at least it hurts in the stomach.

Maybe the number is right and maybe they’ll bring you a whole explanation of.

Why that is right.

It’s exactly what I mentioned before.

Look for the surprises. Bring your, and by the way, surprises.

Are by definition quantitative intuition.

Because what is a surprise? You have information, you have priors or intuition, and the two do not match.

Look for these cases, you’ll become a.

Really good interrogator without being a math based.

Okay, so looking for the surprises, outliers.

Have seemed to popped up again and it’s just certainly very important. I’m curious, do you have any quick tips? I guess in terms of teaching leaders how to look for those.

I mean.

It’S not a quick tip, that’s a.

Very long tip is experience. But Henry Kine once said that experience is a great school, but the fees are high. So yeah, having experience helps because then you look at the number and you say, look, I’ve been here for a.

Long time, is this wrong?

So what are the shortcuts to experience?

Right, because again, we can’t always afford.

To have the experience because from one place to another and so on, learning a few numbers at the tip of your fingers.

So consultants often do this before they go into a new environment. They often spend a day or two in the office just learning a few numbers.

And again, you don’t need to be.

Top of class in math in order.

To just look at numbers. Ask yourself, what is the top school doing? What is the war school doing? Now I know the range.

So now when I see a number, I can say it falls within the.

Range or not, right. These type of things that allow you.

To jump start the experience, if you.

Will, by reading a little bit and learning the environment. So spend some time looking for reports, for example, about the industry or about the topic. So when you see a number, you.

Can actually compare it to that and.

Say, does it fall within or outside any reasonable range?

Yeah, that makes a lot of sense.

Cool.

Let me ask you, I got to ask about AI.

I know it’s a topic you’re very.

Interested in and obviously it’s in a.

Lot of school leaders minds these days especially.

Well, everybody loves to talk about Chat GPT.

I know there’s a lot more AI.

Than just that, but yeah, in terms.

Of generative AI and your research implications.

For school, what are you thinking?

Yeah, I know that I’ve been in.

A meeting that didn’t discuss Chat GPT.

In the past three months or four months.

I actually do a lot of my research on this topic, on topic of.

AI and specifically on language type models. And I have to say that what.

We have seen since November of 2022.

When OpenAI introduced Chat GPT, is a true leap in technology and innovation in terms of where we are with generative AI.

And it also started a little bit.

Of an arm race across the big giants in that industry to, and we’ve seen even since then, a huge progress in what these tools can do, which.

Means we truly live in a different world. I think we better adapt to this.

New world we live in. Right.

One question that people often ask is, well, what do we do with it in the classroom?

Right. And I think a good analogy for us is to go back to calculators. And when they came about and the analogy was, well, what’s now?

I mean, our kids will never learn.

How to do math now that calculators are there.

And the approach we’ve taken eventually was, well, if you’re in primary school, we definitely are going to not allow you to use these calculators. So you actually learn to do the.

Things I talked about before this guesstimating to do math in your head. But then as you go into high.

School and beyond, we actually are going to allow to use calculators because that’s what you’re going to use when you.

Go out there to the wild.

I think we are likely to see something similar to that. We’re definitely in primary school and so on. We do want to make sure that.

Kids learn how to write, they do.

Learn how to express themselves.

And whether we’re going to fully prohibit.

These in schools or ban them in.

Schools, or are we going to maybe.

Conduct enough exercises in class where students cannot use them to make sure that.

They learn how to write.

But this way or the other, we.

Definitely want to make sure that people.

Know how to write independent of having.

Machines that help us write. But eventually, as we again go to.

High school and beyond, I think we.

Will need prepare people to a world in which we have now tools that.

Help us to ride. Maybe to give you another example, I mean, I’m an immigrant to this country.

From Israel and non english speaking country came to this country in the era of spell checkers. Am I spelling worse than I would.

Have if spell checker did not exist?

For sure, if I did not have.

Spell checker, I would have been a better speller.

These days, on the other hand, are.

My documents better spell check than they.

Would have been at spell checker did not exist.

They’re absolutely better spell check than if I didn’t have spell checkers and had.

Just to learn how to spell right.

So eventually, I think, again, another analogy.

Here, that eventually our documents will become better, but some of it will be.

Us, some of it will be the generative AI. I do think that there are several.

Important issues that we need to worry about.

One of them is currently misinformation and.

The fact that they are doing something.

That is called hallucinations.

In the language of these language models. They make up stuff because it’s a.

Predictive model that predicts the next word.

And the next word may not be correct, or the next sentence or the source, for example, that they provide may not be correct. So they do tend to hallucinate. So we need to find a solution for it.

I am optimistic that the industry will.

Find solution for it because it’s an important problem.

But maybe the more worrisome situation is.

That we now have the ability to generate a lot of text that creates by machine, by people with ill intent who will use it for the wrong purposes.

It could be very persuasive language.

Right.

It’s very easy now to create misinformation.

With a very persuasive language using your model. So I’m very sympathetic to the industry call to I don’t think pause, because I don’t think pause is possible at this point.

But for regulators to already think about.

How do we, for example, authenticate something.

That comes from a machine so people know that whether they’re speaking to a.

Machine or to a human and so on. I think we definitely need to already.

Be working on regulations, maybe not mimicking the mistakes that we’ve made with social.

Media, letting the industry regulate itself.

Yeah, that’s worked out fantastically. I like the idea, especially in political ads.

And they talked about having a watermark and saying, hey, this was generated by.

AI or something like that.

But one story, quick story I’ll share.

That’s fun.

I need to send a letter.

I was mentioning the Denver live event.

At least on the live, it won’t be on the recorded version of the podcast, but everybody that listens, they know.

I did a live event in Denver where I’m planning to do so, and.

I needed to write an email inviting more people to attend.

Right. And at this point, let’s say there was eight tickets left.

Okay.

But Odette, I was tired, so what.

Am I going to do?

How am I going to get this email written?

I decided, well, Chat GPT four is a perfect solution. I had already written an eight page.

Document myself that explained the event, the benefits. What you’ll learn, the outcomes, deliverables, so on and so forth. And I asked the robots to, hey, read this eight page document and write a concise email that invites people to attend. And it did it while I slept.

I took a nap, a 20 minutes.

Nap, and when I woke up it was there.

I would say it’s about 80% of.

Where I wanted it.

So I revised and massaged to 20%.

To get it to my voice. And then I hit send and a.

Bunch of people applied and they came.

So that’s a teammate of mine.

Now it’s a colleague. It’s a tool just like a calculator.

And it helped out. And the best thing, those leaders that.

Come to Denver, they’re going to get a great result too, you know what I mean?

So it’s a win, triple win scenario for. Yeah. So appreciate you sharing that.

Let’s relate to the word, to the phrase you use, which is a colleague.

I mean, think about having a colleague.

Who just read the entire Internet.

Think about, you are programmer and now.

I have a colleague sitting next to.

Me who just read every single code ever written and is helping me. Writing my code could be a tremendous, of tremendous value with dealing with the risks of understanding what is really human.

Created and how right 100%. There’s a white paper I wanted to.

Put out around principal development retention forever.

And to be honest, personally me, I’m.

Not making the time, I don’t have.

The time for it.

So that means I’d either have to delegate to somebody on my team or.

Find somebody out there, freelancer sort of.

Interview, see if they’re the right quality, so on and so forth, and then.

The cost and all of that.

Or what I chose to do is at least draft the first version.

I asked chat, GPT, here’s the audience, superintendents, assistant superintendents, here’s the solution.

The leadership community that we support and.

Some information around it, and write me.

2500 to 5000 words with and include.

At least ten pieces of research right cited and it spit something out. There’s work that needs to be done for sure on it. But man, did that get me really.

Closer to done a lot faster.

I think I would advise situation is, check the sources because it’s a predictive model.

The sources are not necessarily the real sources.

They will be around the right source.

But not always the exact one.

Well, to your point, I wasn’t going to record it, but now I will.

Since you’re bringing it up.

I think it hallucinated a few. Right?

Because it did say one that was.

Like an unpublished article and I was.

Just like that thing just totally made something up and cited it as valid. And I know that personally.

So that’s an issue that I have.

To change if I do anything with this report.

To date, I haven’t done anything.

I just did this last night while I was watching Barry on HBO. Max. So anyways, okay, cool.

Let’s get to the last three questions.

That I asked all my guests, and thanks again for being here.

And Ruckus Makers, definitely check out decisions over decimals.

Get [email protected].

So odette, if you could put a message on all school marquees around the.

World for a single day, what would your message be?

I would focus on it’s not about.

The destination, enjoy the journey. I think we are way over focusing on the outcome on the metric on the destination.

It’s not about the destination. Enjoy the journey 100%.

Now let’s talk about your dream school.

Odd.

You’re building your dream school.

You’re not constrained by resources. Your only limitation is your ability to imagine. How would you build this dream school.

And what would be the three guiding principles?

Maybe it relates to things we already talked about, but different places, but focus on understanding. Not necessarily on knowledge, but the ability to understand.

I mean, again, relative to my own.

Education when I was young, I think the schools are much better in teaching my kids to understand math as opposed.

To know math, to think through the.

Processes of how did you come up with the answer?

Don’t just give me the answer, focus on questioning skills.

I would have a guiding principle on.

Kids that are teaching kids to be curious to ask questions less.

Focus again on the typical. Definitely in the entry into school on.

Testing and particularly testing around time period. Right? We teach kids to think fast on in an exam type.

Where is that useful in life apart.

From the next test that they need to do in order to get to other places in life.

But if you really want to change.

The way education is being done, we.

Need to teach less for testing and much more for critical thinking.

Again, through understanding and through learning how to ask questions.

And the third one would be around what I just mentioned as what I.

Would put on every school, which is I would focus the school on the.

Journey as opposed to on the destination.

Making sure that kids love to come to school.

Enjoy the journey, the process of learning.

And not just this exam or the.

Other exam, of getting to this school.

Or the other school after this school.

Really making sure that the journey matters at every moment. We have the kids.

We need to make sure that the.

Journey matters, even if they do, for at that point in time, something they may not enjoy.

Because not everybody enjoy every part of school, but that they appreciate that it’s.

About the journey and it’s not about.

Just getting again this grade or the other grade.

Then brilliant.

We covered a lot of ground today, Odette, of everything we discussed. What’s the one thing you want a Ruckus Maker to remember?

Look for the surprises. Trust your doubts.

Thanks for listening to the better leaders Better Schools podcast Ruckus Maker how would you like to lead with confidence, swap exhaustion for energy, turn your critics into cheerleaders and so much more? The Ruckus Maker Mastermind is a world class leadership program designed for growth minded school leaders just like you. Go to betterleadersbetterschools.com Mastermind, learn more about our program and fill out the application. We’ll be in touch within 48 hours to talk how we can help you be even more effective. And by the way, we have cohorts that are diverse and mixed up. We also have cohorts just for women in leadership and a BIPOC only cohort as well. When you’re ready to level up, go to betterleadersbetterschools.com mastermind and fill out the application. Thanks again for listening to the show. Bye for now and go o make a ruckus.