In looking over Tom Tango's experiment with fielding grades, I couldn't stop thinking about one thing... confirmation bias.
Very simply, confirmation bias describes the act of accepting only those facts that buttress a pre-existing opinion while discarding those facts that run contrary to one's opinion. In short, we're much more comfortable continuing to believe what we already believe.
Here's the bad news: this affects every single one of us.
One example in the baseball world where confirmation bias bites us is during cut meetings in Major League spring training. In this setting the Major League coaching staff, a few front office members, and possibly a scout or two sit in a room to discuss all of the players in camp and decide which ones are going to back to the minor leagues.
First, some background: there are generally 50-60 players in Major League camp and all of these guys have survived an incredibly rigorous screening process over the previous five or ten years (high school, college, rookie ball, A ball, etc) in order to be invited to camp. Let's face it - every one of these players does something well enough to merit both the invitation and some enthusiasm from people in the room. On the flip side, there has never been a perfect player.
So, there we sit discussing the skills of a highly qualified and tested group where the distinction between players is very, very thin. However, what becomes clear is that for the players we want to keep in big league camp, we generally talk about what they can do. For the players we want to send down, we tend to focus on what they can't do, so the decisions seem obvious (which they're not). Understand, I keep using "we" because every one of us in the room is guilty - we can't help ourselves!
So, why do we do this?
I remember a time when Bill Parcells was in the midst of a so-called "quarterback controversy" where every week he was being asked about his quarterback. Week after week he had to answer the same questions in the same way, further committing to a certain QB. Then in one game in which they were losing, Parcells changed his QB in the second half, and they went on to win the game. Afterwards the press was grilling him about the QB change, attempting to get him to comment on the controversy, and saying, "I thought you said player x was your quarterback." Bill leaned into the microphone, probably as only Parcells can, and enunciating slowly, said:
"I changed my mind."
The fact is that it can be really difficult to change your mind, especially when you've taken a public stand on an issue. Nobody wants to be seen as a flip-flopper (I'll stay away from any partisan jabs) or someone without conviction. However, that mindset can often handicap us in making the best decision.
Circumstances change, rules change, new information becomes available. Many things can happen that should alter our position on a topic, but that's simply tough for us to do.
21 comments:
I think most people always have a sense of pride in being right, so it's no wonder confirmation bias exists in every walk of life.
The most obvious examples are seen in politics, from both sides. People's beliefs are a very hard thing to change, and the pursuit of being correct is one of the most American ideals.
In terms of baseball and the Tango fielding grades, I think reputations play a big factor. If you hear player X is a horrible defender, all it takes is one defensive miscue for a person to confirm that belief.
I have always felt Derek Jeter had this reputation amongst baseball of being some sort of acrobat at SS. This belief was brought on by his many web gems and the constant praise journalists gave him (especially those Gold Gloves). The defensive rep gets built into every fans mind, so when they see him make a particular play well, it just gets put into the mind as another piece of evidence into the legacy of Jeter. That belief gets so ingrained in a lot of people's minds, that they ignore a lot of the evidence that suggests Jeter might not be a very good, or even average, defensive SS.
Then again, who knows, maybe I have a confirmation bias AGAINST Jeter's defense and I choose to overly weigh articles I read that detract to his abilities and choose to ignore a lot of the praise he gets from some, but just like any other American, I'm going to believe I'm right at the end of the day, until better proof can prove me wrong
Conformation Bias is everything isn't it? That's one reason to deal with pure numbers instead of people, the numbers don't have bias.
This is a known decision making problem. Usually you want to counter it by basing your decisions on data and information. Naturally, statistics are the bread and butter of baseball, and analyses are based on it, but what other type of data and information do you use?
Also, the perceptions and intuition of the people can sometimes give you better resolutions (and quite often fail you, as you just mention). How do you balance between the two? Do you prepare a decision making strategy, principles, methods and tools before making decisions?
Confirmation bias is hard to avoid. It is why people hold on to bad investments for so long. They look for information that confirms their original thesis and miss the bad news/events that causes an investment to do poorly. A possibly solution, suggested by much smarter people than I, is to put probabilities on each side.
For example, player X has X% of succeeding in the majors for these reasons. He also has X% of failing for these reasons. The goal being to keep the players with the highest expected probability and cut those with the lowest.
Whenever you evaluate the player, you can look back at the odds of success and recalculate based on improvement (or lack of) in certain attributes.
The problem however is that it implies a false level of certainty. You end up feeling like you have made a science out of it, and in some cases it is still an art.
Everyone in baseball fails - it's a question of who fails less when everyone is watching!
Closely related - the need to lower cognitive dissonance, and the only solution is to realized that you can't be completely free from bias!
If anyone is interested, I highly reccommend "Mistakes Were Made, But Not By Me." Very eye opening although more about life in general than sports.
Very nice post! Totally agree, great observation, never realized that about Tango's fan survey.
"the numbers don't have bias"
No, because it is a person picking the numbers to look at. Do you look at the hitter's BA or OPS? His hitting road vs. home? His batting against LHP and RHP? Is HR more important or ISO? Playing at Coors or playing in Petco? Playing in a pitcher's league like the Eastern League or a hitter's league like the PCL?
There is a lot of bias in baseball stats, and you need to look at the right set of stats depending on the situation.
Obsessive is exactly right. Baseball stats can be really, really dangerous tools when it comes to confirmation bias.
Paul,
Just out of curiosity when are the Padres going to name their minor league players of the year?
Ah, Confirmation Bias... The main culprit in the "James Skelton is too skinny to catch" movement.
Paul,
What's the status on Steven Garrison? He was shut down in August with a bad shoulder. Did he require surgery? How long will he need to rehab?
Thanks
People always bring up Jeter, but that is a terrible example. In the Fans' Scouting Report, Jeter always rates as average for a SS, yearin and year out. And this year, he rates as below average.
The ones who believe that Jeter is an above average fielder are the broadcasters that drink the Jeter kool-aid.
Paul brought up confirmation bias, and I suspect this is true for a couple of players on each team. But as he notes, this is true not only with the Fans Scouting Report, but other decision-making processes among professionals.
The only way out of it is to rely on data, like UZR. But, if the data goes against your biases too much, you will simply ignore that data, thereby reaffirming the initial bias you wanted to avoid!
It's like taking Psych research writing class all over again =)
Wouldn't peer review possibly aid in diminishing the effect confirmation bias has? I don't know if this would work in baseball, but in other fields, it seems this is some of the motivation behind peer reviews. Obviously the reviewers will also have confirmation bias, but the less attached they are to the situation, or the less they have at stake, the more objective they could be.
I guess this would be tougher with baseball, as opposed to science, where things are made more transparent.
When I clicked over to Tom's research page and started filling out the report for my favorite team, I found myself experiencing this phenomenon. I didn't know that the technical term for it was "confirmation bias".
I found myself rating players based on gold gloves, age and related physical ability, and the comments I frequently hear "experts" state about the players defensive ability, as opposed to objective quantifiable numbers.
I'll give an example: Russell Martin. Guy won a gold glove last year and is regarded as one of the best young catchers in the game.
Reactions / Instincts - Very Good (did anyone watch the All-Star game?)
Acceleration & Velocity - stolen bases at Catcher are a premium stat so, positionally, Very Good
Hands - I regularly watch the guy make what I think are outstanding digs out of the dirt, so Very Good
Now here is where it gets difficult...
Release / footwork, throwing strength, and accuracy - Last year's gold glove tells me he's solid in these departments, but my eyes tell me he needs to improve. "Experts" are so quick to give him acolades that I think he is likely Very Good in all three, but his CS% to date is .233 (69 SB's allowed, 21 CS) which is in the lower third of MLB. Obviously, I do not have access to a radar gun, so I have no clue where his throwing velocity ranks...it looks good, but how can one tell? Accuracy is pretty fair to judge subjectively, and I think he is Good in that department, so I end up ranking his release/footwork as Fair and attribute the CS% to that.
The survey was much more difficult to complete than initially thought.
You should have seen me ranking Andruw Jones...given the past accolades as compared to current performance, I was REALLY stumped there!
Tango,
I'm forgetting the exact term but isn't the Fans' Scouting Report also a case of population bias.
Most of the people that follow your work and participated in the poll and indeed most of the people who follow Paul's blog tend be the more knowledgable and well spoken fans.
Case in point, I just got back from Shea Stadium today to hear 50,000 Mets' fan chant Carlos Delgado was the league MVP and that Carlos Beltran can't hack it in the clutch. Defense, baserunning, and overall year stats be damned.
If you handed out your poll at Yankee game, like they do with All Star ballots, I have no doubt Jeter's defense would get A's across the board and 50,000 drunks would be screaming at you for even thinking otherwise.
Mr Depodesta,
I know this is slightly off topic but could you please right a post explaining what is going on. With the resignation of Joyner, it seems that there is almost an anarchy going on between the front office and the players. And then it seems that Alderson's interview made it even worse and more apparent.
I know that a lot of the things you can not comment on, but could you please write something and explain what you can. Because from where I and a lot of other passionate Padres fans are standing the team is a complete mess. (And 2 blow out losses to LA doesnt help at all)
padman42,
I will. In fact, I'll try to do it either later today or tomorrow. I wanted to make sure that both Sandy and KT had spoken on the issue first, as it's not really my place.
Paul.
With hindsight being 20/20. If you could go back upto 3 years and make all the FA signings you want but:
1) You have to take the deal they signed
2) You have a steady $85M payroll
3) You couldn't do any elaborate trades
4) Unrestricted FA only, no "we should have given X his draft day demands"
What signings would YOU have done to make the 2008 Padres favorites for the WS.
jc: sure, the survey has population bias in that the responders are not random baseball fans. They are hardcore fans. And, that is the only intent of the survey: what do hardcore baseball fans think they see?
Whether that has value needs to be studied. I get a high correlation in responses when I have just 15 ballots for a team.
I get a decent correlation to UZR and PMR, even if most people who respond don't know those terms, or at the least won't know the actual numbers for the players being judged.
I do find bias among the old guys who used to be great when young (think Junior). It takes a good two years for the halo effect to go away, but so does this apply to the coaches who vote in the gold glove awards. This is a bias that can be addressed.
There is probably bias with guys having a good or bad year at the plate. Again, coaches get into that bias as well.
What can be no question is that the results from this survey is more relevant than a Padres blogger or commenter saying "Khalil is great!" You will have a tough time trying to make the case that one top Padres blogger has more value than two or three dozen hardcore Padres fans. This may or may not include Paul here. We don't know that yet.
And at the very least, I will likely trust the evaluations of the other 29 teams fans on the other 400 players, over a Padres' scout's evaluation of said players. Maybe the scout will give a better evaluation on a hundred of those players... maybe 200? Certainly not all 400. I'd bet on the Fans over the non-team scout in that case.
I see another value in this post in addition to the discussion of "Confirmation Bias" Tangotiger says, "The only way out of it is to rely on data, like UZR. But, if the data goes against your biases too much, you will simply ignore that data, thereby reaffirming the initial bias you wanted to avoid!" but that insists that confirmation bias is something evil. It's not. It's just something that should be recognized.
The essential opposing sentance in the post was "Let's face it - every one of these players does something well enough to merit both the invitation and some enthusiasm from people in the room."
Personally, I didn't like Jim Leyritz. His stats were nothing special, yet in the spotlight that guy was something else. If only every game for him was the seventh game of the World Series he'd be in the Hall of Fame.
Stats aren't taken on a players mood and motivation. Sure there are guys who excel after a "change of scenery", so we know it occurs, but then you have to ask what was wrong with our scenery?
In a lot of ways, these players have been the top dog whereever they've played, and they get up in the higher levels and they find that they're not and their faults are apt to be exposed in front of large audiences.
Is the player a boy or a man? A mench or a prima donna? Selfish or a team player? When the coach talks, does he listen? Can he make adjustments? Stats can show that over time, but a small market club can't really rely on players maturing on their own and only taking the players with the consistantly awesome stats. They have to have coaches and management who are great at communication and evaluation.
The Chargers recently saw the result of a team without that clubhouse atmosphere. I have enjoyed Mr. DePodesta's recent posts because they expose the process and the thinking in Padre-ville, and it is a winning plan.
As a senior in highschool, I always think I know whats best for me, including the huge decision of where I am going to further my education. At the begining of summer, I visited a school six hours away and fell in love with it. I told my dad that this was it. I was done searching. He told me to wait and look at other schools before he hands them a check. Two weeks ago, I was accepted into the school of my dreams - two hours away. My parents were thrilled. I realized that I needed to wait and tell myself I was wrong, then soon following was the reward.
Post a Comment