4 February 2009 by Sanjay Saigal.

Exiting graduate school in ‘91, I interviewed with the Thinking Machines Corporation (TMC). It was a job for which I was woefully unqualified. No surprise, I didn’t get the chance to find out. And anyway, in two years, the company had gone under, victimized by being ahead of its time. But for the ten years it was in existence, TMC burned a spectacular trail through applied computation science. Inarguably its brightest light was the physicist Richard Feynman. (TMC founder Daniel Hillis affectionately recounted their collaboration in Physics Today, and you can read it here.)

Feynman’s legend as a Javert-like pursuer of enlightenment is well-established. (See, for instance, the Feynman tribute site.) Sadly, set next to his spectacular Physics work, Feynman’s contributions to analytic decision-making are less well-known. Hillis’ article described some of Feynman’s pioneering work on the numerics of the Connection Machine. But more interesting to me is Feynman’s work on the Challenger disaster commission.
Alone among the “wise men” on the commission, Feynman went to the source – interviewing the shuttle engineers whose 1 in 100 failure estimate magically inflated to 1 in 100,000 in the hands of NASA management! His appendix to the final report is worth reading as a primer on (desirable) structured design and (self-deluding) risk management. For instance, here is Feynman talking about how shuttle uniquely engines were built:

The usual way that such engines are designed (for military or civilian aircraft) may be called the component system, or bottom-up design. First it is necessary to thoroughly understand the properties and limitations of the materials to be used (for turbine blades, for example), and tests are begun in experimental rigs to determine those. With this knowledge larger component parts (such as bearings) are designed and tested individually. As deficiencies and design errors are noted they are corrected and verified with further testing. Since one tests only parts at a time these tests and modifications are not overly expensive. Finally one works up to the final design of the entire engine, to the necessary specifications. There is a good chance, by this time that the engine will generally succeed, or that any failures are easily isolated and analyzed because the failure modes, limitations of materials, etc., are so well understood. There is a very good chance that the modifications to the engine to get around the final difficulties are not very hard to make, for most of the serious problems have already been discovered and dealt with in the earlier, less expensive, stages of the process.

The Space Shuttle Main Engine was handled in a different manner, top down, we might say. The engine was designed and put together all at once with relatively little detailed preliminary study of the material and components. Then when troubles are found in the bearings, turbine blades, coolant pipes, etc., it is more expensive and difficult to discover the causes and make changes. For example, cracks have been found in the turbine blades of the high pressure oxygen turbopump. Are they caused by flaws in the material, the effect of the oxygen atmosphere on the properties of the material, the thermal stresses of startup or shutdown, the vibration and stresses of steady running, or mainly at some resonance at certain speeds, etc.? How long can we run from crack initiation to crack failure, and how does this depend on power level? Using the completed engine as a test bed to resolve such questions is extremely expensive. One does not wish to lose an entire engine in order to find out where and how failure occurs. Yet, an accurate knowledge of this information is essential to acquire a confidence in the engine reliability in use. Without detailed understanding, confidence can not be attained.

A bottom-up approach is equally essential for building robust and sustainable Analytics-driven decision-support systems. Without the benefit of well-tested atomic models (e.g., the peculiar demand for fundamentally different types of products), building a large-scale model ab initio (in this instance, a manufacturing company’s demand forecast) invites shelfware-hood. I have seen exactly this sort of failure occur at a multi-billion dollar world-wide ERP roll-out, where the demand planning model template built by HQ analysts was so inapplicable to specific business units that local planners took to over-writing official forecasts produced by the ERP with their own hand-computed numbers!

As I’ve previously described (e.g., here), the practice of Analytics inevitably occurs in the shadow of business imperatives. Business goals give shape to the analysis. But they also have the power to corrupt the process. Feynman pungently captures that corruption at NASA, and its consequences in his final words:

For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.

One wonders about Feynman’s analysis of the current financial crises, which appears at least as much a product of self-delusion on high as the Challenger disaster!

Posted in Methodology, Operations Research, Risk | 3 Comments »

29 May 2008 by Sanjay Saigal.

In 2001 INFORMS, the Operations Research professional society, hosted its first national conference on OR Practice in San Diego. Some 300 OR professional attended. Since that tentative beginning, the conference has become a “can’t miss” for OR professionals and practice-minded academics, nearly 500 of whom attended the 2008 conference in Baltimore, MD.

Among its other benefits, the Practice conference has been significant in triggering a wholesale appraisal of OR methodology. The discussion can be said to have been kicked off by Karl Kempf at the 2003 meeting in Phoenix, AZ. Karl’s talk in Phoenix, titled Optimization of a Semiconductor Supply Chain: Technical & Cultural Issues, identified numerous maxims to speed adoption of OR at Intel, his employer. He ended with the declaration that while technical issues (modeling, algorithms, software implementation, etc.) were – in his words - “enough to keep us busy for the foreseeable future”, they “pale in comparison with the organizational dynamic issues”.

Karl returned to his theme the following year in Cambridge, MA, where he focused specifically on Organizational Barriers to Applying Optimization in Business Operations. The key take-away from that presentation was the need for the OR professional to focus on the algorithm to solve the business problem, without being bound by a specific software tool or approach. At the same conference, Jeff Winters from UPS spoke on Why OR Projects Fail, a crab-wise approach to identifying what is not good methodology!

Over the next four years, presentations on the theory of OR practice became among the best-attended and most talked-about part of the Practice conference. Many highly respected thinkers of the field, both practitioners and academics, have taken their turn at bat. Selected highlights:

• Palm Springs 2005
  
  • Harlan Crowder in Stuff Happens: Why Practical OR Projects Fail, fingered “cognitive friction” in the difficulty of implementing OR because of its opacity for the non-technical user.
  • The late Rick Rosenthal in Secrets for Success with Optimization, suggested that solution persistence should be the default choice in optimization modeling.
• Miami 2006
  
  • Marshall Fisher in Eight Habits of Highly Effective OR Implementers laid out his best practices, while pointing skeptically to the goal of creating an actual theory of OR practice.
  • Glenn Wegryn in Sustaining a Vibrant OR Practice at Procter & Gamble, focused on the essentiality of communicating the value created by OR methods.
  • Gary Cross in OR Based Client Engagements, spoke of the need for gradual expansion of scope, and of project-integrated learning, while acknowledging the higher risk inherent in OR-based IT.
  • Harlan expanded on his rules of thumb for successful implementation in Practical OR.
• Vancouver 2007
  
  • Peter Kolesar in Creating a Theory of OR Practice, focused on consultative OR rather than IT implementation, with recommendations drawn from the broader world of business consulting.
  • Tom Baker in OR’s Curse & How to Deal with it, warned against narrow technique-orientation and reiterated the importance of communication, technical demystification, and risk management.
  • Karl Kempf drilling down on Collaborating with IT, hammered on the need for structured development methodology and iterated development.
• Baltimore 2008
  
  • Jean Pommier described the philosophy and artifacts underlying ILOG’s Methodology Implementation of Decision-Support Systems and how Ilog uses its methodology framework to better sell and deliver optimization technology.

(I also spoke in the “theory of OR practice” track in Baltimore on Towards a Theory of OR Practice: Are We There Yet? Material from that talk forms the basis of this and other recent blogposts on methodology.)

Over the years, the discussion on Theory of Practice has emphasized both types of value delivery: consultative (one-off analysis) and systems-based (IT-embedded). However the tenor of the discussion has trended from the general to the specific. Increasingly actionable best practices have been proposed by speakers such as Crowder, Rosenthal, Fisher and Kolesar, among others. Modes of integrating into broader IT implementation currents have been proposed by Cross, Kempf, Pommier, etc. Summarizing brutally, the main lessons from seven years of discussion are:

1. Customer-focused orientation: This can best be described by the directive – “get over your expertness!” It’s seductively easy for an OR professional to fall back on technology that is, to the non-technical end-user, indistinguishable from magic. It is not enough to conjure up an “optimal” solution. The solution also needs to be explicable to the planner who generates them today using decades of first-hand experience of the overlying process and after days of hacking with spreadsheets. The focus of the OR professional needs to be on the problem, not on her technical toolbag. (An especially difficult chore if the practitioner works for a software vendor or a consulting firm that closely identifies with a narrow technical approach!)
2. Communication: Nearly every speaker places the need to clear, regular, and structured client communication near the top of the OR practitioner’s checklist. Is OR special in this regard, different from, say, mainstream IT? The consensus is yes, based on its unique combination of technology “magic” and the expected unfamiliarity of the typical end-user with OR. Client buy-in comes from developing a shared vision for the project, at the managerial and the end-user levels. Maintaining that buy-in requires ensuring that ongoing end-user education is not sacrificed to the tyranny of delivery dates and feature commitments.
3. Methodology: While the notion of a development methodology is not native to the OR community, a consensus on the need for a formal framework appears to be developing. Given the early stage of evolution of an OR-specific methodology, one finds almost as many specific recommendations as recommenders. The need to rigorously assess data requirements and availability tops many lists. As does the inevitability of an iterative implementation framework. Some “technical” recommendations, such as Rosenthal’s emphasis on modeling solution persistence, also appear. (An interesting recommendation from more than one speaker, to “avoid failure”, appears to be a bromide. But it makes perfect sense in the context of the perception of OR as “black magic”; unfamiliar and disruptive technology is rarely allowed more than one strike. Perhaps this will change once – if? - OR captures the mainstream.)

Conclusion

The INFORMS Practice conference has been the locus of methodology-related discussion in the OR community. Based on the popularity of theory of practice-related talks, the need for a usable delivery methodology for OR – presumably based on a descriptive theory – is widely shared. As clearly, there is neither an existing theory of practice, nor a methodology based on a proto-theory. There are good-sounding suggestions for best practice frameworks, but nothing validated except by personal (”expert”) experience.

In the next post in this series, we will examine possible next steps for methodology development.

Posted in Methodology, Operations Research | 1 Comment »

15 May 2008 by Sanjay Saigal.

In a recent article, I examined the need for the practice of Operations Research to be driven by a formal methodology. The primary motivating factor is the increasing mainstreaming of OR (or more general Advanced Analytics) techniques in Enterprise IT. This in turn is forcing system developers and consultants to proactively manage scalability and management of risk by “industrializing” OR delivery.

The typical use of this word in OR-related discussion concerns specific techniques – usually mathematical or statistical in nature – for solving reasonably well-posed problems. (Fairly typical of this usage is the paper titled “A methodology for integrating cell formation and production planning in cellular manufacturing”, published in Annals of Operations Research.) This OR is replete with (so-called) methodologies to solve problems fitting into structured classes. However, in most cases, “methodology” is just a nickel cigar encased in a five dollar label. What is under discussion is essentially a “method” or “technique” (or perhaps a class of such methods.). When speaking of an OR Practice Methodology, we’re interested not in the tools but in the practical principles and artifacts governing the deployment of OR techniques and methods.

The focus of this series of articles is on embedding OR in Enterprise IT. (While OR value is often delivered through one-off analyses, the role of a practice methodology in that purely consultative model is less clear.) In the Enterprise IT context, software engineering provides a sound basis for exploration. Methodologies such as Agile Programming, eXtreme Programming (XP), CMMI and Rational Unified Process (RUP) provide the IT project manager with a variety of risk control frameworks and usable artifacts for standard software development. However, they do not directly address the needs of the OR practitioner. Neither are they well-understood or used in the OR community.

Attempts on OR-specific methodologies have failed to gain traction in the practice community. The CHIC methodology for constraint programming was extended in the late nineties to large-scale combinatorial optimization problems and named CHIC2. However, I do not believe that CHIC2 was ever taught or used anywhere outside its academic seedbed in Europe.

More recently, French software vendor Ilog has extended its proprietary methodology for rule-based systems – ISIS – to optimization. Being a proprietary system that is viewed as a competitive advantage by Ilog, a list of OR-specific ISIS features has not been released. I do believe that ISIS has not, in any meaningful way, been used to develop a large-scale OR application. Ilog’s view of OR is limited to optimization, and thus ISIS is unlikely to be applicable to OR as a whole. But if opened to public view, it is the most promising methodological effort that I am aware of.

In the next article in this series, I will discuss practice methodology from the perspective of INFORMS, the leading professional society for OR.

Posted in Methodology, Operations Research, Risk | 1 Comment »

26 April 2008 by Sanjay Saigal.

Over the past few years, as IT has grown to become a key enabler across business functions, software engineering practice has focused on implementation methodology. Frameworks such as Agile Programming, Rational Unified Process (RUP), and more recently, Eclipse Process Framework Project (EPF), promise to improve IT implementation by distilling industry best practices into usable tools and artifacts. The ultimate goal for these methodology frameworks is to industrialize IT project work, to make it scalable, repeatable and predictable. There is some evidence of success: the Standish Group estimates that over the 1994-2004 period, US IT project success rates more than doubled, to where one in three of all projects are deemed successful. They explicitly credit new IT methodologies: “Doing projects with iterative processing as opposed to the waterfall method, which called for all project requirements to be defined up front, is a major step forward.”

Figure 1: Standish Group 2006 US IT Survey

Note that even armed with full-featured methodologies such as RUP, IT projects are high-risk. As Figure 1 (above) shows, though the US success rate was up to 35% by 2006, one in five project still failed outright. The remaining 46% are condemned to the purgatory of “late, over budget, or below requirements”, presumably enroute to shelfware status.

In Competing on Analytics, a 2006 Harvard Business Review article (available here in research report form, and here as a teaching aid) Tom Davenport argues for a cross-Enterprise approach to integrating Operations Research (OR, also sometimes referred to as Advanced Analytics, Management Science or Quantitative Analysis) into a company’s decision-making. As organizational decision-making becomes increasing IT-based under competitive pressures, more tactical and operational decision-support systems (DSS) have cores built on integer programming, advanced forecasting, Monte Carlo simulation, ant colony optimization, and similarly complex techniques. Moving from piston aircraft to turbines to jets to rockets requires increasingly complex power plant engineering and operating regimes. Similarly, going from simpler rule-based systems that automate decision trees to optimization-driven systems that implement OR technology require a re-examination of IT development processes.

As Davenport documents with examples from companies such as Marriott and Procter & Gamble, OR can maximize value extraction from a company’s resource base. But it also throws up new risks and challenges in implementation. For instance, the major challenges in implementing a database query-driven system (for instance, a standard CRM) involve business process mapping, user adoption, server and bandwidth sizing, etc. However, turbo-charge that application with an OR engine (for example, by embedding a constraint programming-based product configurator,) and you now need to worry about reliable search performance, solution persistence, and other optimization artifacts. Though high-value in concept, OR-powered IT implementations increase project risks along new dimensions. As OR capabilities become checklist items on RFPs, the need for a robust, scalable, and practical OR-aware implementation methodology becomes more urgent.

In upcoming articles, I will discuss practice methodology development for OR-powered systems, starting with a review of how practitioners view practical OR.

Posted in Methodology, Operations Research | 1 Comment »