27 February 2009 by Sanjay Saigal.

While upgrading a portfolio planning model implemented in Microsoft Excel, the need arose to validate a complex simulation involving log-normal distributions. The log-normal probability distribution is obtained by taking the logarithm of exponentiating (UPDATED: 3/1/09) the ubiquitous normal distribution. Its applications are common enough for some researchers to propose that the log-normal deserves at least equal billing, if not center stage.

The need was to back-calculate the mean and variance of the log-normal distribution using certain other known measures. This required non-trivial nonlinear manipulation, an activity that reminded me of the time lapsed since I last engaged in open-ended mathematical discovery. Which brought me to the question: mathematical sophistication is clearly necessary for Analytics, but is it significant? And if not, what skills are required in an Analytics practitioner?

The business pundit Peter Drucker wrote (h/t to J. D. Meier):

When “operations research” first came in, several of the brilliant young practitioners published their prescription for the operations researcher of tomorrow.  They always came out asking for a polymath knowing everything and capable of doing superior and original work in every area of human knowledge.  According to one of these studies, operations researchers need to have advanced knowledge in sixty-two or so major scientific and humanistic disciplines.  If such a person could be found, he would, I am afraid, be totally wasted on studies of inventory levels or on the programming of production schedules.

Though exaggerating for effect, Drucker is asking a good question about feasibility. Few of us can be Feynman! As Analytics moves from being a “nice to have” to a “must have”, the appropriateness of the training of its frontline soldiers becomes increasingly critical. As described elsewhere here, and elsewhere (the issue is elegantly covered at a high level here by Jim Orlin), the INFORMS professional society is stepping to the challenge.

Like many, perhaps most, Analytics professionals I add value through the modeling and implementation of decision-support techniques, not by creating new mathematics. However, my training was almost entirely mathematical. Even now, graduate training tends to over-focus on mathematical technologies. But for professional success, the Analytics practitioner requires a broad set of divergent skills beyond math.

Some skills – facility with optimization and simulation techniques, statistics, certain aspects of software engineering – can be effectively channeled through higher education’s delivery model. But other, equally important, skills cannot. For instance, project management cannot be taught, only learned through the process of doing. (It especially cannot be taught by academics with zero IT project experience.) Critical skills such as business communication, spreadsheet-based analysis and team effectiveness are best left to the in-service setting. (See this online version of the 2007 Interfaces article by Sodhi et al for a list of commonly expected OR skills.)

The academic part of the profession has focused on core technical skills. Is it the role of Analytics professionals to fill in the picture?

Posted in Education, Operations Research, Intechné | 6 Comments »

9 January 2009 by Sanjay Saigal.

Joe Nocera opens his recent NYT Magazine article on the failure of Financial Risk Management by referring to “a persistent tension between those who assert that the best decisions are based on quantification and numbers, determined by the patterns of the past, and those who base their decisions on more subjective degrees of belief about the uncertain future.” Like many of the statements in the long (7600 word), this will give pause to anyone who makes a living in Analytics. While it is true that in Finance, quantitative vs. “soft” approaches inspire quasi-religious passions, mainstream enterprises have long incorporated quantitative analysis as an indispensible tool. Note that word – tool. Analytic methods typically help along decision-making, they don’t replace human judgment.

Consider forecasting. It’s an inevitable function in a manufacturing or service enterprise. Forecasting systems typically include a statistical core – time-series trend analysis, regression-based “causal” models, or combinations thereof. But even in the CPG sector, where large volumes allow statistical forecasting to shine, demand planners inevitably perfect the forecast by injecting intelligence that the most advanced technology cannot replicate. For instance, most forecasts systems do not automatically incorporate adverse weather, or holiday-induced long weekends, or competitor actions. In fact, in my observation the more sophisticated the statistical forecasting system, the more sophisticated the planning staff’s analytic role.

But I digress. My point is simple – with experience, the conflict between Quants and traditional analysts (Quals?) becomes more notional than actual. I bring it up, however, to point out the serious lag between actual practice, and its reporting. But perhaps the schism has a more insidious side-effect.

Nocera’s article fingers the risk measure known as Value at Risk (VAR) as “what led to the financial meltdown”. (A good technical introduction to VAR is available here.) In a nutshell, for any financial instrument, VAR measures the maximum expected loss over a given time horizon. Since loss is uncertain, a probability or comfort factor is also involved. Institutions typically look for 99% or 95% certainty, depending on application. A number of “experts” are quoted denouncing or defending the role of VAR (including  innocent electrons sacrificed to Nassim Taleb’s predictable fire-bombing of practically all quantitative methods; this slow-unfolding but inevitable crisis is hardly a Black Swan, but to a man with a nuclear finger, all battles look like WWII).

Nocera’s indictment is unveiled as he recounts the gradual institutionalization of VAR:

• Instead of one of many risk indicators, VAR was treated as the risk measure. Speaking of investors and CEOs of financial institutions, Nocera says: ”In the bubble, with easy profits being made and risk having been transformed into mathematical conceit, the real meaning of risk had been forgotten. Instead of scrutinizing VaR for signs of impending trouble, they took comfort in a number and doubled down, putting more money at risk in the expectation of bigger gains. “It has to do with the human condition,” said one former risk manager. “People like to have one number they can believe in.”

  The tendency to rely on point estimates is a frequent cause of the so- called Flaw of Averages, which Probability Management is designed to combat. That the estimate is produced by sophisticated analytical computation doesn’t suffice. Understanding risk requires one to look at multiple alternative futures. That is best captured by a probability distribution (or a histogram, which is a granulated distribution). Or, minimally, by a ranking of scenarios.

• VAR can be non-additive over time: VAR computations are often based on the market at calculation time. But they aren’t updated as market conditions change, possibly dramatically in volatile times. When it comes to making new investments, that’s not a problem. But added to existing investments evaluated under different regimes, VAR may seriously misstate portfolio risk.
  
  
• VAR is super-additive over the market: It’s hard to understand how letting each institution rely on its own VAR measure could have sounded like a bright idea to Clinton-era regulators: ”The Securities and Exchange Commission, for instance, worried about the amount of risk that derivatives posed to the system, mandated that financial firms would have to disclose that risk to investors, and VaR became the de facto measure. If the VaR number increased from year to year in a company’s annual report, it meant the firm was taking more risk. Rather than doing anything to limit the growth of derivatives, the agency concluded that disclosure, via VaR, was sufficient.”

  Even if each market participant accurately managed its own risk, it’s stunningly obvious that given the proliferation of incestuous instruments such as credit swaps, the risk to the market was many orders of magnitude higher than any summation of individual risks.

Rereading the article, I am astounded by how many basic principles of quantitative analysis were violated by supposedly sophisticated actors. Perhaps it does come down to the Quant/non-Quant disconnect that opens the article: executives and regulators usually come through the latter door.

Posted in Intechné | 1 Comment »

26 December 2008 by Sanjay Saigal.

The BBC recently reported on the long-simmering power struggle in Christiandom’s holiest site: the Church of the Holy Sepulchre in Jerusalem. This site – by tradition the place of Jesus Christ’s burial and resurrection – is contested by six denominations that occupy tiny bits of it: Roman Catholic, Greek Orthodox, Armenian Orthodox, Syrian Orthodox, Egyptian Copt and Ethiopian Orthodox. Each faction constantly maneuvers to improve its territory, their shenanigans (real and perceived) often boiling over in eruptions of violence.

The church itself is in a state of near-collapse. But since its ownership is in dispute, all attempts at repair are stymied. The situation is dismally Pareto optimal: any effort at improving matters makes at least one other party unhappy. The following heartrending yet funny anecdote captures the hopelessness of the situation:

The intractable nature of the territorial arguments over the site are epitomised by the short wooden ladder that rests on a ledge above the church’s main entrance.

It has been there since the 19th Century because rival groups cannot agree who has the right to take it down.

Under the Status Quo agreement, rights to the windows reached by the ladder belong to the Armenians, but the ledge below is controlled by the Greeks.

So the messiness persists, despite its ongoing cost to the Christian community. (Not to mention the potential catastrophic downside – complete collapse of the complex’s badly deteriorating roof.)

I was reminded of the Church of the Holy Sepulchre while navigating the recent brouhaha in the Analytics blogonook created by a post in the Intelligent Enterprise blog. Doug Henschen quizzes IBM’s Ambuj Goyal on Big Blue’s “analytics strategy” following its recent acquisition of Ilog, a leader in decision technology components (and my past employer). Henschen’s didactic frame is the notional question: “Will IBM Add Analytics to its Toolbelt?”

Henschen summarizes the interview’s takeaway in his lede as “[IBM contends that] predictive and statistical modeling — key offerings for the likes of SAS and SPSS — are overrated. IBM has what Goyal describes as better, cheaper alternatives in a mix of techniques developed for industry- and domain-specific challenges.” This startling conclusion has been met with (I’ll use a polite word) skepticism by Analytics-oriented blogs. James Taylor at the EDM blog smells the ghost of sales campaigns past:

Sadly this reminded me of the old days of IBM - when FUD (fear, uncertainty and doubt) was IBM’s reponse (sic!) to anything they did not do well. Predictive analytics are not overrated, at least not by anyone who understands them. It is true that predictive analytics, like all good technologies, are sometimes overused by over-enthusiastic supporters and that they can’t do everything. IBM’s lack of this technology is a mistake as without it their solution set is incomplete and no amount of FUD will change that.

Anne Milley at SAS’s company blog SAScom is predictably indignant in light of Goyal’s presumed attack on SAS’s knitting. Milley labels Goyal’s comments “dizzying spin” and suggests that IBM execs “deride the value [of Analytics] because they haven’t been able to monetize the analytics in their research labs even as others achieve significant returns”.

The BBC report came to mind because, as in Jerusalem, the disagreement occasioned by Goyal’s interview is fundamentally doctrinal. It centers on the existential (and ungrammatical) question – what is Analytics?

I have previously mentioned Tom Davenport’s HBR article called “Competing on Analytics”. (Davenport has also published a subsequent book with this title.) Davenport never quite defines Analytics, but his view is obviously expansive; his keystone success story is Marriott Corporation, whose Total Hotel Optimization system relies, at its core on Linear Programming (i.e., Optimization). Davenport sows confusion by using interchangeably using Analytics and labels like “statistical masters”. Analytics is not Statistics. Statistics is a tool in the arsenal of the Analytics professional. But it doesn’t describe the category.  

The SAS Institute has adopted the branding framework of Analytics to compete in the Decision Management space. Historically the purveyor of a statistical toolkit (which later morphed into a Data Warehouse platform,) SAS had weak or non-existent offerings in Optimization and Inferencing/Rules, and came late to the Business Intelligence and Predictive Analytics (BI/PA) wave. So its stance of statistical methods as somehow defining Analytics is natural.

Conversely, prior to the Cognos acquisition, IBM had negligible footprint in the BI/PA space. In terms of Optimization, while boasting incredibly qualified R&D and Consulting groups, it has been unable to develop a profitable software offering. (In the ‘80s IBM attempted to market an optimization toolkit called OSL, made many sub-optimal decisions along the way, and finally killed the product in the late ‘90s. More recently, IBM has tried to leverage an open source toolkit called COIN-OR as a brand-builder, in my opinion essentially futilely.) The acquisition of Ilog provides IBM immediately with best-of-breed offerings in Optimization and Inferencing (or Business Rules). However, it still lacks anything like SAS’s statistical platform. Goyal’s pitch suggests that they know it. And they are trying to advance the notion that what they have is what the market needs.

As is not uncommon, differences born of necessity are being painted in Marketing’s primary colors to manipulate each vendor’s “rightness”. While this makes perfect sense to advance each party’s immediate business imperatives, it does not help the cause of Analytics in the Enterprise. James Taylor correctly observes that:

The important thing is to focus on the decision and then figure out how to solve it… Business rules, optimization, data mining, predictive analytics and adaptive control [are all] necessary ingredients for Enterprise Decision Management and business success.

Indeed.  Let me illustrate with an Analytics success story. In a highly successful Space and Assortment Planning project at a greeting card manufacturer, most of the techniques in Taylor’s list were invoked. SAS was used to derive the space elasticity curves that drove revenue forecasts, CPLEX was used to optimized a complex sequence of layout models, and JRules was used to manage scenarios, preferences and exceptions. As the then Ilog project lead, I cannot recall a discussion where we, or any technology partisan on the large project team, attempted to push a technology agenda. The team uniformly focused on creating the fastest-performing, most accurate and most efficiently usable application.

Along similarly cooperative lines, instead of advocating the relative importance of one’s own bag of tricks and denigrating the competition’s, actors in the Analytics ecosystem need to adopt Davenport’s catholic approach. Let’s focus on expanding the impact of the field, on making it as pervasive a function in business as Accounting or Human Resources. To mash up the rallying cry of Bill Clinton’s 1992 campaign with the verbal stylings of the logorrheic Sarah Palin, “It’s all of the above, stupid”!

Posted in Business Intelligence, Operations Research, Vision, Intechné | 1 Comment »

12 December 2008 by Sanjay Saigal.

In recent weeks I have been working with Sam Savage, well-known OR personality and a consulting professor at Stanford. We’re focusing on developing a practice framework for Probability Management. Whazzat, you ask? In sum, Probability Management is all about robust decision-making in the presence of uncertainty. (Pretty much the vision for Intechné!)

Since real world decision problems are almost always ill-structured and fuzzy, our tools of choice belong to the worlds of simulation, and statistical visualization. Stochastic optimization plays a role too, but in a very different form than typically understood, say, in Operations Research circles. In general, we are not interested in creating IT systems that generate “best possible” recommendations. Rather, we enable managers to interactively explore the decision space of good solutions, using something similar to a business intelligence (BI) approach. The key difference between BI and Probability Management is that while BI is essentially descriptive (identifying multi-factorial relationships, typically for historical data) Probability Management is prescriptive: our clients learn what to do better.

I intend to write further on this topic, but for now let me point interested readers to our newly redesigned web site. The organization is a loose consortium of academic and commercial folks involved in the field, as vendors, users, and advisors. Check out the Interact! tab. It contains illustrative Excel models that describe the relevant concepts far better than long-winded descriptions. If you find it interesting and wish to discuss further, contact me.

Posted in Probability Management, Business Intelligence, Intechné, Risk | 1 Comment »