Tell Me Something I Already Know (Or Want to be True)!

19 December 2008 by Sanjay Saigal.

As a member of the advisory council for the upcoming INFORMS Practice Meeting in Phoenix, I am assembling what I hope will be a boffo slate of speakers for the track on Managing Risk & Uncertainty. Researching recent work in the area, I encountered an excellent 1997 paper by Dick Barr and Tom Siems titled Bank Failure Prediction Using DEA to Measure Management Quality. Early warning indicators (called Key Risk Indicators, KRIs, in the Risk Management community) of bank failure include one that is difficult to directly extract from balance sheets: Management Quality. Barr and Siems used Data Envelopment Analysis, or DEA (a linear programming-based efficiency measure, look here for a tutorial) to identify an analytically meaningful surrogate for Management Quality. The resulting multi-factorial risk model was remarkably predictive: it could correctly label a bank as strong or an incipient failure with 96% accuracy, a year to 18 months out.

I wonder whether the early warning system was used. Siems is listed as employed by the Federal Reserve Bank of Dallas, a regulatory body. So there appears a prima facie opportunity to apply the model. On the other hand, I would not be surprised to hear that the paper’s publication was its terminal “development milestone”.

Speaking later with Doug Smith, another financial risk estimation guru, I commiserated how often models such as Barr & Siems are left on the shelf. As Doug characterized the unfortunate imperatives facing his technical collaborators employed in finance: “The pressure to create profits meant that the results of risk models were ignored”.

Doug’s lament brought to mind a frequent problem for the analytics practitioner: the unfortunate habit of our “customers” (whether line managers within our own organizations or external consulting clients) to cherry-pick which analysis to use, and which to ignore. The situation becomes especially tricky when analytical findings collide with political winds. The latest such example comes from the British Isles, where a pay-as-you-go congestion charge system pushed by Newcastle University researchers was deep-sixed by Manchester voters. Feasibility, it turns out, is in the eyes of the customer. In this case, the citizenry decided that the traffic smoothing benefits of the congestion charge were trumped by the nefariousness of the new “tax”. Never mind the value of time!

While there are good estimates for project acceptance/failure (e.g., here) I have not come across estimates on how often analytics projects fail, or are shelved, for extraneous reasons. Do you have data, or even stories, to share?

Posted in Risk | 3 Comments »