ORACLE

By Doug Samuelson

The little group of analysts continued sharing their reactions to the Japanese tsunami. Now they were gathered again, a couple of months later, still admiring the effectiveness of the Japanese preparations and response, praising the courage and commitment of the nuclear plant workers, lamenting the scale of loss, and trying both to make some sense of the tragedy and to extract lessons for analysts. They had agreed that sometimes extraordinary events can overwhelm even the best preparations and defy even the best predictive methods – and sometimes highly improbable good things can happen, too.

“Somehow it seems as if we’re just restating the obvious,” Dan shrugged. “We’ve all heard about ‘the black swan.’ Events that seem highly improbable sometimes do happen. And, Ed, your story about the kamikaze pilot who was miraculously spared to tell his story was amazing, but it’s still the same idea, isn’t it?”

“Well, maybe,” Ed conceded, “but for an obvious idea, it sure seems to take a while to sink in. How many nice refereed journal articles did you keep seeing, after the financial crash, about systems of differential equations, Black-Scholes derivative pricing, linear programming portfolio analyses – all still over-fitting the little movements between big shocks, and missing the big shocks that are the real events of interest?”

“You’re right,” Gary chimed in. “I remember a huge military manpower planning model I worked on some years back. The data set was organized by cohorts, by month when they joined. After a while we figured out that something that happened with the first cohort in, say, their 15th month, the second cohort in their 14th month, the third cohort in their 13th month, and so on, really was a simultaneous event for all the cohorts. We had a number of ‘jumps’ or ‘shocks’ like that. The data structure had made them a bit harder to spot. We’d go back and ask the sergeant-major, who was our guru for the history of this service branch, and it would always turn out to be something like, ‘That’s when the new reenlistment bonus structure came out.’ So we realized that those annoying ‘jumps’ or ‘discontinuities’ were actually the events of interest, and all that other stuff we’d been elaborately modeling was basically noise between events! Same phenomenon, isn’t it?”

“So it seems,” Alan responded, “Here’s another form of it. I had an interesting experience when I wrote about this idea, including Ed’s story. My editor objected to using ‘kamikaze’ in the title. He explained, ‘People who read the whole story carefully will get your point, but I’m afraid some people will read just that word and think you’re being insensitive to Japanese readers.’ At first I thought he was being oversensitive, but I guess I can see why he was concerned.”

“Social scientists call this ‘framing bias,’ Ed explained. “A seemingly minor choice of a loaded word can send some readers or listeners down a different track from what you intended. A friend of mine loves to startle audiences by announcing, ‘Let’s talk about the race question!’ Then he describes how tricky it is to ask what race or ethnicity someone is, and how what you want to study changes how and whom you ask. Of course nearly no one in the audience thinks at first that this was what he was about to discuss. He uses that reaction to illustrate how the way you pose a question can influence the answers you get, both from the respondents to the question and from the readers of the analysis.”

Carl, an editor himself, had joined the group. “That’s a fancy term for a simple but powerful idea,” he said. “Any good editor or reporter knows that the most insidious and powerful source of bias is story selection. More than any choices about content, follow-up questions or presentation, you can make or break a cause, or embarrass yourself in the effort, by persistently focusing on just one aspect of a complicated story, or by ignoring some points of view altogether.”

“So if you’re Fox News and you openly dislike just about everything about Obama, you concentrate on the events that are going badly for the administration, and if you’re, say, MSNBC, you emphasize whatever seems to be going well,” Dan growled.

“Or you could focus on whatever’s going badly for Obama’s opponents,” Carl added. “And the trouble is, the effects go beyond the immediate issue. Too much scandal-seeking and fault-finding leave people viewing government as a whole so negatively that it becomes much more difficult to govern. We’re seeing quite a bit of that now, I’d say. On the other hand, too little critical coverage can help a strong leader with bad intentions get away with more. We’ve seen that a number of times, too!”

“It can happen to the best of us,” Alan chuckled ruefully. “Last weekend, meeting a friend for lunch in an unfamiliar part of town, I drove right past a huge shopping mall. It was well hidden by trees, with no big signs, to blend into the neighborhood. I thought my friend had told me it was on the left, so I was looking there, and it was on the right! What you start out looking for really does alter what you end up observing, doesn’t it?”

“Right,” Carl agreed. “Which makes you wonder about models and modelers without much sanity checking on their assumptions, wouldn’t you say?”

Doug Samuelson (samuelsondoug@yahoo.com) is president and chief scientist of InfoLogix, Inc., a small R&D and consulting firm in Annandale, Va.

References

1. “The Black Swan’s Parable,” OR/MS Today, April 2011
2. “The Race Question,” OR/MS Today, February 2009