PRACTICE AND POLICY LECTURE SERIES, NOVEMBER 2013 - IMPROVING CHILD PROTECTION WITH SAFETY SCIENCE

No Audio: Michael Cull, PhD, MSN - Office of Quality, Safety and Risk Prevention, Center for Excellence for Children in State Custody

Dr. Michael Cull: Thanks for letting me come talk to you today. I work in, I kind of share my role at Vanderbilt I'm kind of 50/50. I spend about half of my time working in the Center of Excellence for Children in State Custody, which I've been involved in that program since its inception in about 2000 - so we're going on 14 years. That's a program that was funded out of one of our big lawsuits in Tennessee to work specifically with the child welfare system. So clinically I'm trained to work with kids, but I've been working in the child welfare system now for many, many years. 

Most recently though, I've, in the Medical Center, started doing some work with our Office of Patient Safety and Risk Prevention. And what I do in that office is I'm responsible for our safety culture survey. I do a lot of work and build a lot of programs working with teams around how we communicate to provide safer care, how we understand our system better from that safety perspective. So what I've been doing is bridging those two worlds recently. I've been working with our child welfare system around kind of, how do we take these patient safety concepts and apply them in a child welfare setting? Because it became apparent really quickly when I started doing this new work that it's very, very applicable. 

So what I hope to do then today is cover some of these topics, just define some of this stuff for you so you have a context for it. We'll talk about what safety science is, what safety culture is, what it means to be a high-reliability organization, talk about how those things might apply in child welfare and then I'll give you some concrete examples of what we're doing in Tennessee, some of the actual sort of strategies and tactics that we're trying to translate into that environment. 

The setting's not conducive. I don't do a lot of standing-up-at-podium kind of lectures. I do a lot of team training, small groups... so would love it if anyone wanted to interact with me while I stood up here and stared at this big crowd. But I understand that it's easy for you to sit silently. I'm comfortable with silence so if I stand here for a long time, please speak up. 

So this is actually the only slide I think that I have that says Safety Science on it, so I'm going to define it for you and tell you what I mean and then we'll talk about what we're really going to talk about today. Safety Science is a big bucket term. It applies to all those sciences that we draw on and apply to systems to make them safer. So it pulls from engineering, it's very inter-professional. It pulls from psychology heavily. It's in the world of human factors. There are programs beginning to emerge that are specifically called Safety Science or Human Factors in Safety Science, but really it's how do we use the... what we know of, how to best engineer a system, how to understand how teams communicate with one another, how people exist within complicated systems. 

It's using all of that information and applying it to making systems more safe is what we call "Safety Science." And the way we do that, we know it's really important to engineer better systems, but what we really need to do is build cultures that sustain that kind of activity. So what I'll talk about a lot more and I may not even mention the word safety science again - it's safety culture because that's what we're trying to build. 

We're trying to build a culture that's values, its principles, the things that they teach people as they come into the organization are focused on those behaviors that support an engaged workforce and reliable, relatively error-free care delivery in our context over a long period of time. And we know a really important aspect of that is how leadership interacts with the organization. And I'll talk in a lot of detail about this, we know that leaders in these organizations have to be able to balance, when is an individual accountable? When is the system accountable? And it's often at both ends and there's an interplay. It's an understanding of the complexities of how humans exist within systems or organizations. 

We also know that some of the really important qualities of these organizations is that they value communication, open communication, in their workforce, at all levels. They value transparency. And maybe more important than anything else I've said so far, is that they're learning systems, that they're systems that value above all else learning from mistakes, errors and good things that happen. But that they're learning systems. 

So I'm going to start with a couple of kind of cases, although that's kind of an overstatement. One of them actually comes from your system, and I'll say that I read some about it, but I don't have all the context or all the details, so I'm going to set this up now and then I'm going to come back to it a little later and we'll talk. So my question, if anybody does want to speak up, is what, aside from the obvious, what things do these two tragedies have in common? 

A Turkish air flight, they receive erroneous information from the plane's radio altimeter. The plane crashes and it results in the death of four crew members and five passengers. And then a child welfare context, a two-year-old girl is left unattended by her foster family and subsequently drowns in the family's pool. What things are similar about these two cases? Again, aside from the tragic deaths that occur. 

Audience member: Better communication.

Dr. Cull: Communication issues. 

Audience member: (inaudible)

Dr. Cull: They will both be on TV, there's no question about it. 

Audience member: (inaudible)

Dr. Cull: They're preventable, that's a good one, I like... at some level, the risk that these organizations were at that resulted ultimately in these things happening probably fall in the bucket of preventable, if we knew enough about them ahead of time. 

Audience member: (inaudible)

Dr. Cull: A system failure. 

Audience member: (inaudible)

Dr. Cull: So, I'm going to go ahead and stop there. Does anybody else have a really good one they want to share? Because that one segues me well. 

Alright, so I think about these at the system level. I think the similarities are at the system level. And what I hope to do by the end of this talk is make the case that child welfare... Can I stop for a second? So I can know the audience a little bit better. Who here works in child welfare, and is at the sort of the supervisor, frontline worker level? Just hands. So are other people from other agencies? In leadership roles? I'm focused on child welfare so I hope this is all relevant, but that's kind of the audience that I'm looking at, is that work. So, what I hope to convince you of is that child welfare shares a lot of similarities with some of these other organizations like aviation that we talk about a lot when we talk about in the context of safety. 

So, I think both systems have... there's this element of public trust. The public thinks that somewhere there in the background people are looking out for their best interest, right? When we get on airplanes, we expect that at some level, the airline is concerned about not crashing and killing a lot of citizens. Same thing in child welfare. 

There's also active, sometimes aggressive public oversight. So both systems are highly regulated at the state level, at the federal level. They're highly, highly, regulated systems. Like pilots, child welfare workers are making high-risk decisions minute by minute, day by day. And the results of their decisions often times have really high consequences, so crashing planes, children dying, children being exposed to subsequent abuse. So we're making high-risk, high-consequence decisions, again daily if not minute by minute in both these systems. 

And to your point, they're very high-profile. When things happen, it's a big deal. The news loves to write about fatalities in the child welfare system. And it happens in every state. I know that you guys have been through it here in Oklahoma. We've been through it in Tennessee at a level that I've never experienced before, and again I've been working in this system for 14 years. But it ultimately resulted in our Commissioner last year, a little longer ago maybe, resigning. We were just in Denver at a meeting two weeks ago and I was amazed at what they've been going through there with the media coverage that they've had. 

So like aviation, like some of the other industries I'm going to talk about in a moment, they're very, very high-profile. So these are the similarities that I think exist, and there are many probably more. So, aside from the obvious, the tragedies that occurred, the systems that the tragedies occurred within share a lot of similarities that I think bear further attention. 

So, the organizations that we know successfully navigate this have been termed high-reliability organizations. There's a pretty large literature base now growing around this notion of high-reliability organizations. The organizations that work in these high-risk, high-complexity situations, they're able to do their job relatively error-free for extended periods of time. And they have kind of less than their share of bad incidents. And I'm going to show you some examples of where these organizations are and what they look like in a minute. So, I'll say this probably a lot. 

This is how I frame the world of patient safety and how now we're applying it to child welfare. Is this idea of, can we make child welfare a high-reliability organization? Can we operate in a child welfare setting, which I just showed you what was similar is arguably very, very different than these other organizations too. Can we make it highly reliable? Can we do our work for extended period of time without big errors? 

So high-reliability organizations, and one way to say this is sort of that, the thing, what that senator said about pornography, "You kind of know it when you see it." We've done a good job of defining what high-reliability organizations look like. They do two things really well: they anticipate and they contain. So, they're constantly wary of risk. They're constantly challenging the workforce within. They're constantly challenging their own assumptions. So that's this notion of being preoccupied with failure. They're always wary of when their next opportunity to fail is. 

And they're always challenging their assumptions. They're always combatting this notion of confirmation bias. It's one of the biases that Human Factors folks tell us that exist in systems that we're wired as humans to seek out information, seek out data that confirms our own assumptions. So if I think there's no risk in a situation, I'm going to seek out the data that confirms that. That's just how we're wired. And you can think that we shouldn't be wired that way, but it doesn't mean we're not. And so understanding that, and building that, and kind of baking it into your system that you're constantly challenging your assumptions about data is a principle aspect of these high-reliability organizations. 

Sensitivity of operations means that sort of at all levels of the organization, they know what's going on down at the frontline. They know what's happening. They know that at any given work day or work setting, caseloads are X and we have so many people on FMLA, and we know that we have to understand those things because they're going to impact the risk of the system. We can't assume normal operations if all these things are actually impacted and the front's sensitive to that. 

And the containment about when unexpected events happen - and they will - because we, we're looking for them, how do we contain them? How do we mitigate the impact of those unexpected events? One way is this notion of deference to expertise. We put decision-making down at the point of care. We, without respect to title or degrees, we put decision-making closest to the point of risk. 

And then we build a system through learning because that's the number one thing about these things, is they're learning systems, that's resilient. They can expand and come back to normal operations when unexpected events happen. So here are some examples. These are the kind of most-studied high-reliability organizations. 

Air traffic control or aviation really more generally, the power grid, international banking, nuclear power. You've probably, when something happens it's a big deal, but actually they do a lot of work and nothing ever happens. And then when it does, it's a big story. But generally speaking, they're really good at delivering power, and they're really good at landing airplanes. 

And they do it... we tell a story, or make a little joke kind of about aircraft carriers. Aircraft carriers, the Navy takes relatively young seamen, train them, put them on this big floating parking lot, splash it with seawater and jet fuel, it's really loud so they have to use hand signals, and they land airplanes. And they don't crash them. 

So, that's what it means to be a highly reliable organization. So, where's healthcare in this? It's not on my list. Hospitals? Not there. And that's where I work. That's the work I do for at least half my life is... how do we make our system at Vanderbilt more safe, more reliable?

No Audio: ôAdverse Event rates: US Hospitalsö showing
 
So this chart, let me orient you to it quickly. On the vertical axis is lives lost per year, and across the bottom are number of encounters per each fatality. So low number of encounters, low number of lost lives, high number of encounters. So over here, this is where aviation lives and nuclear power. They're considered ultra-safe, so they're having one fatality, or less than one fatality per 100,000 interactions. 

We're over here. So we're having more than one fatality per every 1,000 encounters in healthcare. Lucian Leape, who you're actually going to see a video clip of in a moment, wrote a paper in '98 where for the first time, it was in JAMA, the Journal of American Medical Association, first time sort of called out healthcare and said we need to shift the way we think about error in medicine from one of a system that shames and blames to one that understands that complexity and understands that balance between sort of human and system accountability. 

And then follow down that with an IOM report that named this 100,000 lives number. So that was about, I think that was 2001, report came out from the IOM that said there are about 100,000 unnecessary, preventable deaths in the hospital system in the United States annually. And that number was ahead of deaths, and this is again, this is 15 years ago, so we weren't as good at treating it as we are now, but it was ahead of HIV, it was ahead of motor vehicle accidents. 

Hospitals are not safe places. Hospitals are not as safe as the deck of an aircraft carrier. We're down here with bungee jumping and mountain climbing. And it's probably an underreported number. That 100,000 lives has kind of made it into our consciousness in healthcare. It's probably underreported. And we haven't moved the needle. Again, we've know about it for a little over a decade, and we have not moved the needle. 

No Audio: Video title ôMistakes in Healthcare.ö

Alright so, I'm going to show you about an eight-minute clip. I hope it will stimulate a little bit of conversation. The first speaker, just to give you a little bit of setup, is Lucian Leape. He's considered sort of the father of, or when you see him kind of the grand wizard, he's got big white eyebrows, he's about 6'2, of the Patient Safety Movement. 

Wrote that first article, was involved in the IOM and he's been an active advocate for patient safety in hospitals since. He and two other people are going to kind of tell stories about mistakes in our healthcare system, and the mistakes that result in, not just the fatalities, that result in all the other bad things that happen in our healthcare system. (laughs) We have a hand signal for the audio.

(VIDEO PLAYS)

No Audio: IHI Open School for Health Professionals. Perspectives: The Mistake, Part I. What is one error that youÆve made? What did you learn from it? What can others learn from it?

(MUSIC PLAYS)

Lucian Leape: Like most doctors, I recall every one of them vividly, the serious ones. You never forget them. I mean, I've been out of practice for 22 years, but I clearly remember the serious mistakes I made. I didn't make a whole lot of them, but you remember them. The one that would be helpful to talk about I think was... I was a pediatric surgeon so I operated on children. And one of the serious mistakes I made was failing to operate on a child who had bleeding from the intestinal tract. And I delayed surgery so long that she had ended up dying from this. And as far as I was concerned, it was an entirely a judgmental error on my part. 

The teaching and my experience, I was fairly experienced at that time. I'd been out of residency for ten years or more. The teaching and my experience was that bleeding from the stomach in children always stops. You don't need to operate on them and, in fact, there's a fair amount in the literature about how it was wrong to operate on these people because you didn't need to. And so I believed that and so we treated the child and gave transfusions and so forth, and finally became apparent that it wasn't stopping and by that time, there'd been enough damage that we weren't able to save her. 

I, of course was devastated by this, as was the family. And we all cried together. I mean, no question about it. And I did, you know, I did explain it to them, and I apologized and so forth, but no apology's going to bring back your child. I just thought it was about the worst thing I'd ever done in my life. 

What do you learn from it? Well, I think a very simple lesson. That is decisions that are really critical to life and death, or nearby, near that, should never be made by one person. If we work in meaningful teams, I wouldn't have been able to get away with that. I would have had people challenge, nobody challenged my judgment. No, no resident, or other person around ever asked any question about it because it was my decision. I was respected and so it wasn't challenged. It should have been challenged. It should have been challenged. It should have been discussed in an open manner. And if we would've had another pair of eyes looking at that child, she might be alive today. 

Kathy Duncan: As a new nurse, about eight or ten months out of nursing school, I was working in the critical care unit working on three to eleven. Patient went into VTAC, and at the time the crash cart was in use across the hall. And I rushed in with the defibrillator only and shocked a patient without using the gel pads. Now, fortunately, the patient came out of that horrible rhythm, which I mean, a life-threatening rhythm, but he had these horrible burns on his chest because of what I had done, in my haste and not thinking to do the right thing. 

That patient stayed with us for three or four days. I still remember his name, I still remember his face, and it really, he was very upset about the burns on his chest. He didn't care that he was in a horrible rhythm. He was very upset with me personally. He was awake and knew what was going on. And it really weighed on me for a long time. It still does. 

That was hard for me as a 23-year-old kid to, you know, you just want to say, "Well golly, you know, you're alive. You know, get over it." But you know, fortunately I didn't say that to him. In code situations, staff get so flustered, and myself included, that you just start grabbing things and saying, "I think epi's good. I think giving the lidocaine's good. I think we need to start some fluids" without really thinking through what the protocol is, and thinking though what we learned in ACLS and how do we get the airway open first, how do we do this, how do we give things appropriately? Because you really can hurt somebody in an emergency situation like that. 

Michael Leonard: Well, having been a cardiac anesthesiologist for 20 years, I can tell you several personal mistakes. The one that really jumps to mind was four, five years ago in the operating room, doing a quick, complicated surgical case, which I've done 10,000 times, with a lot of interruptions, surprises, distractions and the surgeon turns to me without any warning and says, "I'm done." 

And I'm thinking I have this patient asleep, they've given paralyzing agents, I've got a lot of things to do to wake this patient up and the rate-limiting step is really giving the medicine to reverse the curare muscle relaxant. And then I go on my drug cassette like I've done day after day, several thousand times before to grab this yellow vial of glycopyrrolate and some neostigmine and I grabbed the wrong vial. And so I ended up re-paralyzing the patient, which I recognized, you know, several minutes later. And we did, you know, it's hard. You kind of lean over the ether screen to the surgeon. You say, "Well, we're going to be here for a while because I just made a mistake," which is not a fun conversation. 

But we sat in the operating room for an hour, waited for it to wear off. And what struck me is when I went back and looked at the drug cassette, I had four yellow vials in the same drawer among 20 different drugs. They looked almost identical and I thought, "I'm the Chief of the Department. I'm the Chairman of Surgery. I'm expert. I've done this for 20 years and yet in a quick, complicated situation, I pick up the wrong one and gave it." 

So what do you do? Well you go and tell the truth to the patient, the family. You know, I talked to the patient's wife because this guy's going to come in seven or eight times that was cancer (?) before it's fatal. I said, "This is why we're in there for two hours, not one. You can see I was using both these medications. They look very similar. At the end of the case, I accidentally picked up the wrong vial, gave another dose." 

There's no residual harm, which is important because they're going to come back several times and the last thing you need is more concern or anxiety that something's going to go wrong, or something did go wrong that they don't know about. And what was interesting was talking to this woman, I mean, I'm apologizing to her and like 30 seconds later she gets up and puts her arm around me and says, "It's okay, doctor. We're human. We make mistakes." And I'm thinking, "She's kind of way ahead of me on this." But what was important to her was, "What are you going to do to fix it?ö 

And I said, "Well the good news is I'm kind of the boss." And I said, "When I leave here, I'm walking to the OR pharmacy, and we're going to very quickly re-label these drugs, put red sticky labels over the top of all the muscle relaxants, and we're going to put, physically separate them into different drawers." System Safety 101. So it's very hard to make this mistake. 

I think the real cultural learning was walking down the hallway to the OR pharmacy, running into five of my anesthesia partners and showing them the two vials and saying, "Look what I just did," and having four of the five of them look me in the face and say, "Oh I've done that. In fact, once I did that last week." And I'm thinking that's a pretty bad answer. I'm the Chief of the Department. I have no idea this is going on, and not only are we continuing to put patients at risk, but we're letting our friends and colleagues all go step in the same potholes. So that was one of many memorable adventures, shall we say? 

(MUSIC PLAYS)

Dr. Cull (continues): So these three cases I think all point out something a little bit different. Lucian Leape, the first speaker talks about something that we talk about a lot in patient safety. This notion not only of teams, but of people on teams feeling safe to speak up and challenge authority. So, again we try to push down decision-making closest to the point of risk as we can, deference to expertise. We also need people who feel safe to work in their environments, speak up, take in personal risk, and challenge decision makers when they know things aren't going the way they're supposed to be going. 

I'm going to come back to the nurse because I think actually she's a bad example, and I want to come back and explain that. And I do like the way she uses "golly" in a sentence because it doesn't get used enough. 

(laughter) 

Dr. Cull (continues): The last guy, it's kind of the classic, it's two things: it's kind of the classic examples of a human factors error, and we have lots of examples of this in healthcare. Two vials look too similar. Two concentrations of heparin, 1 to 1,000, 1 to 10,000, somebody grabs the wrong concentration, gives it to a kid, kid dies or has a bad outcome. We have nurses that have been fired, charged, committed suicide over events exactly like that. We have pharmacists sitting in jail because of, right now, because of errors like that. 

And what we do unfortunately is stop there, find someone to blame and don't learn from it. And that's the point he really made was other people in his department had made this exact same mistake and there was no system in place to harvest that information, feed it back into the system so you could build a more resilient workforce and support a learning system. That was the point he was really making. 

Now the nurse, her story to me is about, "If I had just thought in the moment better about what I was doing, if I had just known more about my protocols, if I had just slowed down, realized I didn't have..." All those things assume, they give too much credit to sort of individuals in the moment. And what you really need to focus on is, how do you build a system that supports that new nurse, doesn't let her make those bad decisions? 

In healthcare we talk about, just one example, hard stops. So as I said in the beginning this is a little about engineering, finding those places where you don't want two vials that look different. But it's also about building a culture. One of the classic hard stop examples is in ORs, we don't, we now have anesthesia tubing that only connects to an anesthesia port in the wall, and oxygen tubing that will only connect to the oxygen port. 

People have successfully worked around that because that's what we do. There's a big sign right here that says "Absolutely no drinks allowed." So I have a stool with my water on it. It's my work around. It's created new risk. I won't spill water maybe on a laptop, but I'm probably going to trip over this stool in a minute. So, people can try hard to work around these hard stops, but that's the kind of ways we engineer our system. 

So do you think we face similar challenges in child welfare? I've just exposed healthcare to you. We kill 100,000 people - probably an underestimate. And that doesn't count all the medication errors we make, and it certainly doesn't account for all the bad things that happen to staff who are involved in these fatalities because we all - people in this sort of this serving industry, the healthcare, the health services - we get up every day. We're all smart and well-trained. We want to go in and take care of people. So when these things happen, it takes a toll on our workforce. 

Do you think we have similar problems in child welfare? Does any of this resonate with you? I mean I think we do, or obviously I wouldn't be doing this talk. I also think that it's not a child welfare issue. I think it's societal. I'd have to take child welfare agencies off the hook a little bit because ultimately child welfare agencies are the result of public policy. Public policy is what societies choose to do or not do. So we choose to put an agency in charge of responding to child abuse and maltreatment. 

No Audio: ôDeaths from Abuse and Neglect.ö Nationally estimated 1,560 children died from abuse and neglect in 2010 or 2.07 children per 100,000 children in the general population. This is an average of four children dying every day from abuse or neglect.

And so here's some of the numbers. About 1,500 kids die every year. I'm sure that's an underreporting, too. We know we have definitional issues in states. We don't have good data, but this is the best we have right now. That's an average of four kids dying every year, I mean sorry, every day. So to put that in context. It's about 72 Kindergarten classes that we lose every year. It's about 120 little league baseball teams. And it's a little less than the 2012 graduating class in Oklahoma City Public Schools last year. So it's a big number. It's a lot of kids dying every year. 

The reason I take child welfare off the hook a little bit is because we react. We're, this is, we were at a conference a couple weeks ago. This is a public health issue. This is a societal issue, abuse and neglect and like deaths in hospitals, before the report came out that got everybody stirred up, because we thought we were pretty darn good at taking care of people before that. 

And then this report comes out from the Institute of Medicine and it says, "You're killing more people than car accidents, HIV, a whole host of other things." And it's an underreported number. So before that came out, we weren't riled up enough about it. And it's taken us awhile to get there. But these are real numbers, real lives. 

So how do we advance what I'm calling a safety culture, what the literature calls a safety culture in child welfare settings? What are those specific things that we're going to go do? The first thing we have to do is think differently. You have to pursue a different lens. You have to understand bias in a system and human behavior. You have to start thinking about fallibility and system interactions, the things that I've been talking about a little bit. 

You have to ask new questions, and I'll talk a little more about all these things. Because we want to know ôwhat and why,ö not ôhow and who.ö It's not about finding someone to blame. It's about understanding why something happened, and what was the context when it happened? So we can learn from it, keep it from happening the next time. 

And then, I think we're not very good at this in healthcare at all, and I think there's huge opportunities in the child welfare workforce. Safe, engaged workforce that has the tools they need to do their job should be an absolute precondition, not a priority, a precondition. A system shouldn't function without that, first and foremost. And I think we have a long way to go in healthcare, we certainly have a long way to go in child welfare. 

So we talk about this contrast between an old view and a new view when we talk about this work. And so an old view of error in systems or mistakes in systems would say professionals are supposed to get it right. They've been trained, they're just supposed to get it right so let's find out why they didn't and we'll move on from there. 

Bad things happen because people make mistakes. This is a huge fallacy because often times bad things don't happen when people make mistakes. Actually the majority of the time, people make mistakes all day long in their job and there's no bad outcome. And similarly, bad outcomes happen when people don't make mistakes. And so a view that says bad outcomes occur because mistakes happen, it's too linear. It overstates kind of the causal reactions in our systems. And it kind of misses the point because people make mistakes all the time without bad outcomes. 

People and organizations that fail are bad. Again, not necessarily true. Often times, these things are out of the control of the system, the system didn't understand well enough how to prevent what went wrong. And I think really important in the child welfare system, this notion that blame will motivate people. That heads are going to roll. If we find the person that did this, we figure out what went wrong, and we make very public our response as an organization to their failure, then other people, that nurse, if we had fired her for shocking that patient, other nurses, they would have thought better about delivering care. Does that make sense on its face? 

I think in that situation, nine out of ten nurses might have made a similar mistake. And none of them would have been helped in their work by her being fired or blamed. So blame doesn't motivate people. People need to be held accountable. That happens. There's certainly a balance when people do things that are malicious, do things that are outside of practice that they need to be held accountable for. But it doesn't help our systems learn. 

So a new way of thinking about it would say we know we're going to fail. It's inherent in complex systems and we've built very, very complex systems. We've built systems that are too complex for humans to function within them. There's no question about that. 

Risk is always emerging. So we can't pat ourselves on the back for understanding that risks are in front of us today because there's going to be new ones coming. So we have to be, like when I was listing those sort of, the principles of high-reliability organizations that being constantly wary of our next opportunity to fail. 

In healthcare we do a thing called safety rounds. It's something we want to try in Tennessee where leaders at whatever level go out into the unit, so we'll have sort of director-level people going out into the units, going up to nurses and in a kind of scripted way asking explicitly, "When's our next opportunity to hurt a patient?" And having that conversation on the frontline. And the first time it happens, they freak out. They don't want to have that conversation, but generally, over time the culture changes and this is one of those opportunities. I was talking to Director Lake a minute ago, in this work, sometimes you have to sedate the lawyers. And this is one of those things that lawyers aren't terribly comfortable with. 

Not all risk is foreseeable, so you have to build a resilient workforce, because when those unexpected events happen, you have to be able to expand and come back to normal operations. People and systems are fallible. And again we have to have well-prepared professionals. They're crucial. It has to be a precondition. 

Alright, so let's revisit the two cases. So we have the Turkish Air flight that died. Four crew members. I mean they crashed. Four crew members, five passengers died. And then we have the two-year-old girl left unattended by foster parents, drowns in the family. The - did I say this earlier? - this is a quick summary of an event that actually happened in Oklahoma. This is the expert findings. This is an accident analysis report. The findings from that of the crash. And this is pretty typical of an accident analysis report in aviation. First bullet, and I didn't cherry pick the passages I'm going to use very much. If, they're on the Internet. You can read them. If you read both reports in total, you'd get the same feel that I think I'm going to describe in a minute. 

So, Sidney Dekker, whose actually a real prominent figure in Safety Science and Accident Analysis... you know, he writes about sort of how the training matches with industry standard, the experience of the pilot, and how that may or may not have contributed to, or it actually didn't protect this crew from this tragedy. But I want you to focus on this one and then I'm going to come back to it. He writes, "The documentation and training available for flight crews in the Boeing 737NG leaves important gaps in the mental model that a crew may build up about which systems and sensor inputs are responsible for what during an automatically flown approach." And so it's really these "leaves important gaps in the mental model that a crew may build up about blah, blah, blah," that's what I want you to focus on and I'm going to come back to it. 

And this is the report about the incident with the two-year-old that happened in Oklahoma. And in this report, I want you to focus on, he says things like, or the team says things like, "indisputable," "should never have been approved," "the pool should have been removed," "no children should ever," and "by failing to ensure that this hazard was..." And so the word "should" I'll just say, I'll contrast it, too, always makes us kind of cringe. We have a visceral kind of reaction in our work to the word "should." 

So look at these two, kind of summaries or just pulling out these quick words, and tell me, which one of these two approaches to accident analysis supports a learning system? Does anyone think the first one? Does anyone not think the first one? And then which one focuses on identifying someone to blame, and would sort of not be supportive of learning? The second one, right? So the word "should," we don't care so much about "should." We want to know why. We want to know what decisions were going on. 

It's another bias. I talked about confirmation bias. Another bias that plagues accident analysis or going back and looking at these cases is hindsight bias. It's that bias of, "I knew it all along," because it's easy to understand, or it's easy to think you understand the decision path when you already know the outcome. But you can't possibly know all the things that were impacting the decisions, right? 

So starting from there, starting from understanding of hindsight bias we'd say that I don't want to know about shoulds, I want to know about what was going on in that case manager's decision-making that led to not responding to that risk, because there may have been very good reasons. There may have been very bad reasons, but it may have been influenced by her excessive caseload and the number of hours she worked that week. And you need to understand those things if you really want to make your system safer. 

So we talk about a new view. You can't stop at what I was just describing. So when we understand the cause, we aren't done. So, in the two-year-old's example, the report says that basically not taking the pool out or not putting a fence around the pool was the problem. If those, if one of those two things would've happened, this child wouldn't have died. It literally says it was a preventable death and if one of those two things hadn't happened, or had happened rather, the pool removed, the fence around the pool. That's an oversimplistic view of risk. 

It's plausible that a fence around, like my stool, could've created a different risk. It could've made the family less vigilant and put the child in a less unsupervised sort of state. It just isn't, it's not the place to stop. It's the symptom in the system when you understand cause. It's not the end of your inquiry, it's the beginning. So when you identify these things, you dig more deeply. 

And we know that another thing about humans is that we're wired to see patterns where they don't exist, and miss patterns where they do, and so that's the benefit of understanding data. So, these events aren't random or isolated. They can be understood. Risk in our system can be understood. 

No Audio: Video title ôHuman Limitations.ö

Alright, so I'm going to try to explain this in a different way, try to make my point a little differently. Video's going to come on in a minute, and the narrator's going to give, it's only two minutes long. Narrator's going to give you some instructions. If you've seen this video before or one like it, I'm going to give you a different set of instructions. So he'll come on, you'll get it in a minute and give you instructions. He's going to tell you to watch the players wearing the white t-shirts. And they're going to be on the stage passing a basketball. And you need to count the number of times they successfully pass the basketball. 

If you've seen this before or a video like it, then I want you to also count the number of bounce passes that happen, and I'm going to try to pause it. Sometimes I can't pause it. And we'll talk about the number you guys come up with, okay? Did I confuse anyone? Well done, nice. I didn't even have to use my hand signal.

(VIDEO PLAYS) 

Narrator: Count how many times the players wearing white pass the ball. The correct answer...

(VIDEO PAUSES)

Dr. Cull (continues): 16. Did anybody get a different number? Did anybody get a number other than 16? What other numbers did you come up with?

Audience member: 13

Dr. Cull: 13. Usually people are around the 13 to 16 or 17 range. Did anybody see anything else odd about this video? Don't say it. Don't say it! You have to cooperate. Alright, let's roll it.

(VIDEO RESUMES)

Narrator: 16 passes. Did you spot the gorilla? For people who haven't seen or heard about a video like this before, about half miss the gorilla. If you knew about the gorilla, you probably saw it. But did you notice the curtain changing color or the player on the black team leaving the game? Let's rewind and watch it again. 

(Rewinding sound) 

Here comes the gorilla and there goes a player, and the curtain is changing from red to gold. When you're looking for a gorilla, you often miss other unexpected events. 

(VIDEO ENDS)

Dr. Cull: So the point here isn't whether anybody in the room, and I'm sure that some of you missed the gorilla, and more of you missed the curtain and the person leaving. It doesn't matter. The evidence is very clear, about half of you will. And so what's important to know is that people on your teams will miss these things. 

And so, does it matter, you know, Director Lake, it's your responsibility to count. You're going to be paid based on your ability to count those passes. And you should've seen that gorilla because policy 7.975 says when you're counting basketball passes, pay attention to gorillas and curtains changing colors, and you missed it. It does no good for us to understand that in that way. What we need to know is what prevented him from seeing the gorilla? Because we know it's coming. We can predict it. 

So how do we develop this new way of thinking and learning? Well, here's what we're doing in Tennessee. And I'm going to try to get through these slides in about ten minutes or so. These are the specific things that we're doing that kind of fall into three buckets. We're enhancing our ability to do surveillance about these things, and that's about doing, right now, we're not doing all the things that I list yet. We want to do some and we are doing some. 

A different kind of accident analysis, which I'll talk about in our child death reviews. We've applied a different model to child death reviews that I'll talk about. And, what I'll talk even more about is that we're doing a safety culture survey, which is that we're measuring our system to understand those behaviors that we know are important to support a safe culture. We're going to be working on better communication at all levels of the organization. And then we're going to be looking for those opportunities to better engineer our system. And that's the one that we're not quite to yet. 

The one thing that probably falls in this bucket is that we've implemented a safe sleep bundle, sort of a package of services that everybody, every kid gets when, all infants get, and we're piloting it. They get it before we leave the home. So in the same way that we wouldn't let, at the children's hospital in Vanderbilt, a kid leave in a car without a car seat, we've decided that we're not going to leave a home and let a kid sleep there until we know there's safe sleep furniture in the house, so a Pack-and-Play. 

And it was much more complicated than we thought when we started down that path to actually get something in the home in that way. But, I think the analog is appropriate. We would never let a kid get in a car without a car seat. We shouldn't leave a home without them having a safe place to sleep because we know they're at risk. 

So this is what our new accident model looks like. We've really only done two things, although it's been a very complicated process. We've hired people and made them professional accident analysts. We've trained them to think this new way about the role of humans in systems and error and mistake, and we've applied an accident model called Accimap. 

And so this is what we, this is the product, sort of, of our accident analysis and what we're doing is it's a model that builds up so we start from what happened, child fatality and we build up through these levels all the way up to what governance policies, regulations, and legislation actually impacted this decision. We connect all the dots, we look for themes and we write our findings in the way that Dekker wrote his findings about that analysis. 

And what we avoid doing is coming out of these child death reviews with a list of new improvement projects, new training initiatives, and new policies because they're really ineffective tools for improving our system. So we collect them up as findings... we're just getting involved in this work so we're, I don't know, six months in. We're going to collect them up as findings, we're going to meet and really focus on the things that we know we can go fix. 

We're very careful about not letting this be, you know, just a way to produce 100 different things that we know we don't have the resources to go fix right now anyway. And believe me being a part of the child welfare system in the role that I was over the last 12 or 14 years, I've been a part of reports that recommended training and policies, because it's just how we used to think about things. 

But I drive the same route home from Vanderbilt every single day. I've been doing it for 16 years. Invariably once a year or so, I'm going to drive right past my street. I don't need a new policy to get me home. I don't need to be trained on the route. So, they're not very effective tools at making our systems better. 

The other thing we're doing is measuring our safety culture. So, like all measurement, measurement of our safety culture is about communication. We measure temperature outside so that you and I can go outside and agree upon what 75 degrees feels like, or today what 31 degrees feels like. That's why we're measuring our safety culture. We're measuring it so we can talk about it, so we can develop a language. And, we're also doing it because there's an intervention effect that happens when you do these kinds of assessments in a workforce. 

When you ask these kinds of questions that I'm going to show you in a minute of a workforce they actually respond in it, and there's some evidence that it affects their behavior. And then we're going to hopefully use this data to track change over time. We're, of course, all living in fear that our local newspaper, The Tennessean, will request these data from us before we have the opportunity to understand them and publish them because they ask hard questions. 

We put together a 41-item survey. It's five dimensions that were previously validated in the healthcare organizational theory literature. We adapted them for child welfare environment. We've already sort of revalidated the psychometrics, we'll be publishing that. This tool's widely available, or freely available. I'll send it to anybody who wants to use it. 

We had a target of case managers and their two levels of leadership. And we got a 70% statewide response rate after I kind of cleaned up the data and took out the invalid surveys. We were really at about 86%. So we had an unbelievable response from our staff. 

No Audio: ôYears of Experienceö bar graph showing 

Not unexpected, we only asked a couple of demographic questions. So not unexpected, our workforce skews towards about 45% or so, or five years or younger, and then as they shift over to these bars, our leadership, as they begin to make that shift, the experience levels go up. 

We asked them about the number of hours they work. This is self-report because we know our actual work hour data is really unreliable because they are under so much pressure not to use overtime, that they just work and don't report it. I have a sense that fatigue is way under-understood. I didn't say that right. It's not understood well in the child welfare system, at least in our systems. 

Some systems are set up differently where their case workers are not on call or out at night in a way that ours are, but our case workers are working really long hours and we don't spend very much time thinking about how that impacts their decision-making. And we have a specific question about how they understand that on the survey, or set of questions. So they're all working more than, I put 37 1/2 hours on there because there's something cultural about it in our system. When I go out and meet with staff and talk about these results, they talk about 37 1/2 hours like it's sacred. In part because they're told they can't get overtime. And so what they'll say is things like, "I can't manage my 38 cases in 37 1/2 hours." They say it all the time, so it's a big deal for them. And they're all working obviously many more hours than that as I'm sure your workers are. 

No Audio: ôAll Dimensions û All Regions, All Staffö a bar graph showing all metrics under 60% (fail threshold): Stress Recognition û 44%, Safety Organizing û 39%, Psychological Safety û 42%, Leader/Member Exchange û 54%, Safety Climate û 56%.

This is what our data looked like. Just to orient you, this line, the red line at 60%, below that line is considered sort of problematic in this work in healthcare. When I do these surveys at Vanderbilt, most of our bars come above that line, but below the green line, which would be considered a high-reliability organization. As you can see, all of ours fall well below that line in child welfare. We don't know what that means. Nobody's done this in child welfare yet. It might be that the lines should be set at a different spot, but I don't think so. 

Because what it says to me is on these measures, the stress recognition, which is one of our scales, that's the notion that your workforce understands that fatigue and stress impacts their decision-making. You want a workforce that's not heroic, that understands working long hours, having the flu, having... those things affect your ability to make good decisions. And what this tells us is that really only about four out of ten people feel that way, or they understand that that impacts their decision-making. 

Psychological safety is about feeling respected, accepted as part of a team, and free to take interpersonal risks. This is what Dr. Leape was talking about. This is speaking-up behaviors. "Will I challenge authority?" It's only about four out of ten on that one, too, in our workforce, and this is consistent between our team leaders and our staff. Some of the other ways we're looking at the data, variability across regions is important so when you see gaps like this, and they are statistically significant, we want to know why. Why are some regions looking different? 

None are looking great, but why do some regions look different than others? We want to look at the differences between our supervisors and our case workers because what's emerging from the healthcare literature is that while all the scores are important, you see a disparity like that, regardless of the order - and this would come from like surveys of operating rooms, when surgeons think they work in a very safe operating room, and the nurses think that it's not, or reverse that, they have higher adverse events in fatality. Those are the ORs that you don't want to be in. 

So, regardless of where the score falls, when there's that discrepancy, we want to know about it. So these are our findings... great staff engagement. We were excited by that. I have about 49 words when I put it into a text, 49,000 words of text associated with stress and fatigue. We know that we haven't organized teams well to anticipate and learn from unexpected events. 

Six out of ten people don't feel psychologically safe. They don't feel accepted, respected, safe to take interpersonal risks in our setting. And it's no surprise, right? Because when they make a mistake they... these child welfare workers work in this space of unmitigated risk, and when they make a mistake, it's going to be on the news, if it's a big enough mistake. So it's really no surprise that only six out of ten of them feel like they work in sort of a psychologically safe environment. 

And we know we have a lot of work to do on our supervision models, and what's interesting for us about this, too, is that psychological safety and supervision are some of the strongest predictors of case managers leaving the field. So, we know we need to keep people in these jobs so they can gain the experience they need to be good at it, but we know we have models that don't support that, and I think I'm getting told now... Sorry. 

So, here's what we're focusing on. Psychological safety, our supervision model, our teamwork and communication. Why are we looking at teams in child welfare? I have this sense that child welfare workers are expected to do the work of police officers, pediatricians, teachers and a whole host of other professionals and they don't have the tools and resources they need to do it. 

And they, unlike some of these other disciplines, don't yet understand the value of organizing into teams. In our state, at least, they're individual actors. And we're also focusing on it because it's one of the areas where there's a lot of emerging literature, at least in healthcare, that you can improve outcomes by focusing on teams, focusing on a safety culture as it were. Alright, I got there, sorry. 

Any questions in the 45 seconds that we have left? And I'll hang around, I'm happy... Yes sir.

Audience member: There are some cases where, for reasons of public accountability, you would want to hold a particular worker accountable for reasons of public safety. So how do you avoid the bias, the hindsight bias, and how do you avoid differentiating system failures and focusing on outcome from holding people accountable?

Dr. Cull: So you're absolutely right. I hope no one takes away from this that I say that people aren't accountable. I think it is this a small, small percentage of the time though, that that's where we should stop. One disadvantage that you have in child welfare and all these sort of human services agencies is you don't have the protections that we have in healthcare or aviation around reporting, around peer review. We do a lot inside hospitals. 

I mean, you never hear, you rarely will hear about the surgeon that made the mistake that led to the fatality. Doesn't mean we didn't sanction them, we just didn't sanction them in the media, and we didn't hang them out there to be the person that was responsible for this death. And it's just, it's an incredible disadvantage for the system, for child welfare systems. And so I think we fundamentally have to somehow change the way people think about this work and people have to step up, legislators need to step up and figure out ways to better protect child welfare workers if they want to build a better, safer system. 

Because they assume they're going to be held accountable, and the response I believe is disproportionate when it happens. And so when it happens - this is really important - when it happens and it happens in all states. We go through it, it happens in all states. When it happens, when, you know, subcommittee XYZ says heads are going to roll and the newspaper runs a series on the fatalities and why are you killing kids in our system and then you find the people who are accountable, it drives, and then their names, it drives the rest of your workforce underground. 

It makes them that much less likely to report those errors when they see them. And we have to know about those opportunities when people see mistakes happening, or they've made them themselves so we can learn from them. And so if we can't get past it, then it's going to be very, very challenging to do. I think your question is a very, very good one and it's very challenging. 

No Audio: Questions? Michael.cull@vanderbilt.edu  www.mc.vanderbilt.edu/coe