Among its other benefits, the Practice conference has been significant in triggering a wholesale appraisal of OR methodology. The discussion can be said to have been kicked off by Karl Kempf at the 2003 meeting in Phoenix, AZ. Karl’s talk in Phoenix, titled Optimization of a Semiconductor Supply Chain: Technical & Cultural Issues, identified numerous maxims to speed adoption of OR at Intel, his employer. He ended with the declaration that while technical issues (modeling, algorithms, software implementation, etc.) were – in his words - “enough to keep us busy for the foreseeable future”, they “pale in comparison with the organizational dynamic issues”.

Karl returned to his theme the following year in Cambridge, MA, where he focused specifically on Organizational Barriers to Applying Optimization in Business Operations. The key take-away from that presentation was the need for the OR professional to focus on the algorithm to solve the business problem, without being bound by a specific software tool or approach. At the same conference, Jeff Winters from UPS spoke on Why OR Projects Fail, a crab-wise approach to identifying what is not good methodology!

Over the next four years, presentations on the theory of OR practice became among the best-attended and most talked-about part of the Practice conference. Many highly respected thinkers of the field, both practitioners and academics, have taken their turn at bat. Selected highlights:

• Palm Springs 2005
  
  • Harlan Crowder in Stuff Happens: Why Practical OR Projects Fail, fingered “cognitive friction” in the difficulty of implementing OR because of its opacity for the non-technical user.
  • The late Rick Rosenthal in Secrets for Success with Optimization, suggested that solution persistence should be the default choice in optimization modeling.
• Miami 2006
  
  • Marshall Fisher in Eight Habits of Highly Effective OR Implementers laid out his best practices, while pointing skeptically to the goal of creating an actual theory of OR practice.
  • Glenn Wegryn in Sustaining a Vibrant OR Practice at Procter & Gamble, focused on the essentiality of communicating the value created by OR methods.
  • Gary Cross in OR Based Client Engagements, spoke of the need for gradual expansion of scope, and of project-integrated learning, while acknowledging the higher risk inherent in OR-based IT.
  • Harlan expanded on his rules of thumb for successful implementation in Practical OR.
• Vancouver 2007
  
  • Peter Kolesar in Creating a Theory of OR Practice, focused on consultative OR rather than IT implementation, with recommendations drawn from the broader world of business consulting.
  • Tom Baker in OR’s Curse & How to Deal with it, warned against narrow technique-orientation and reiterated the importance of communication, technical demystification, and risk management.
  • Karl Kempf drilling down on Collaborating with IT, hammered on the need for structured development methodology and iterated development.
• Baltimore 2008
  
  • Jean Pommier described the philosophy and artifacts underlying ILOG’s Methodology Implementation of Decision-Support Systems and how Ilog uses its methodology framework to better sell and deliver optimization technology.

(I also spoke in the “theory of OR practice” track in Baltimore on Towards a Theory of OR Practice: Are We There Yet? Material from that talk forms the basis of this and other recent blogposts on methodology.)

Over the years, the discussion on Theory of Practice has emphasized both types of value delivery: consultative (one-off analysis) and systems-based (IT-embedded). However the tenor of the discussion has trended from the general to the specific. Increasingly actionable best practices have been proposed by speakers such as Crowder, Rosenthal, Fisher and Kolesar, among others. Modes of integrating into broader IT implementation currents have been proposed by Cross, Kempf, Pommier, etc. Summarizing brutally, the main lessons from seven years of discussion are:

1. Customer-focused orientation: This can best be described by the directive – “get over your expertness!” It’s seductively easy for an OR professional to fall back on technology that is, to the non-technical end-user, indistinguishable from magic. It is not enough to conjure up an “optimal” solution. The solution also needs to be explicable to the planner who generates them today using decades of first-hand experience of the overlying process and after days of hacking with spreadsheets. The focus of the OR professional needs to be on the problem, not on her technical toolbag. (An especially difficult chore if the practitioner works for a software vendor or a consulting firm that closely identifies with a narrow technical approach!)
2. Communication: Nearly every speaker places the need to clear, regular, and structured client communication near the top of the OR practitioner’s checklist. Is OR special in this regard, different from, say, mainstream IT? The consensus is yes, based on its unique combination of technology “magic” and the expected unfamiliarity of the typical end-user with OR. Client buy-in comes from developing a shared vision for the project, at the managerial and the end-user levels. Maintaining that buy-in requires ensuring that ongoing end-user education is not sacrificed to the tyranny of delivery dates and feature commitments.
3. Methodology: While the notion of a development methodology is not native to the OR community, a consensus on the need for a formal framework appears to be developing. Given the early stage of evolution of an OR-specific methodology, one finds almost as many specific recommendations as recommenders. The need to rigorously assess data requirements and availability tops many lists. As does the inevitability of an iterative implementation framework. Some “technical” recommendations, such as Rosenthal’s emphasis on modeling solution persistence, also appear. (An interesting recommendation from more than one speaker, to “avoid failure”, appears to be a bromide. But it makes perfect sense in the context of the perception of OR as “black magic”; unfamiliar and disruptive technology is rarely allowed more than one strike. Perhaps this will change once – if? - OR captures the mainstream.)

Conclusion

The INFORMS Practice conference has been the locus of methodology-related discussion in the OR community. Based on the popularity of theory of practice-related talks, the need for a usable delivery methodology for OR – presumably based on a descriptive theory – is widely shared. As clearly, there is neither an existing theory of practice, nor a methodology based on a proto-theory. There are good-sounding suggestions for best practice frameworks, but nothing validated except by personal (”expert”) experience.

In the next post in this series, we will examine possible next steps for methodology development.

The typical use of this word in OR-related discussion concerns specific techniques – usually mathematical or statistical in nature – for solving reasonably well-posed problems. (Fairly typical of this usage is the paper titled “A methodology for integrating cell formation and production planning in cellular manufacturing”, published in Annals of Operations Research.) This OR is replete with (so-called) methodologies to solve problems fitting into structured classes. However, in most cases, “methodology” is just a nickel cigar encased in a five dollar label. What is under discussion is essentially a “method” or “technique” (or perhaps a class of such methods.). When speaking of an OR Practice Methodology, we’re interested not in the tools but in the practical principles and artifacts governing the deployment of OR techniques and methods.

The focus of this series of articles is on embedding OR in Enterprise IT. (While OR value is often delivered through one-off analyses, the role of a practice methodology in that purely consultative model is less clear.) In the Enterprise IT context, software engineering provides a sound basis for exploration. Methodologies such as Agile Programming, eXtreme Programming (XP), CMMI and Rational Unified Process (RUP) provide the IT project manager with a variety of risk control frameworks and usable artifacts for standard software development. However, they do not directly address the needs of the OR practitioner. Neither are they well-understood or used in the OR community.

Attempts on OR-specific methodologies have failed to gain traction in the practice community. The CHIC methodology for constraint programming was extended in the late nineties to large-scale combinatorial optimization problems and named CHIC2. However, I do not believe that CHIC2 was ever taught or used anywhere outside its academic seedbed in Europe.

More recently, French software vendor Ilog has extended its proprietary methodology for rule-based systems – ISIS – to optimization. Being a proprietary system that is viewed as a competitive advantage by Ilog, a list of OR-specific ISIS features has not been released. I do believe that ISIS has not, in any meaningful way, been used to develop a large-scale OR application. Ilog’s view of OR is limited to optimization, and thus ISIS is unlikely to be applicable to OR as a whole. But if opened to public view, it is the most promising methodological effort that I am aware of.

In the next article in this series, I will discuss practice methodology from the perspective of INFORMS, the leading professional society for OR.

…OR people should see the BI community as a great source of problems and inspiration (and should make an effort to learn their language). But BI will inevitably use non-OR methods for some of their issues, so is rightly not “the same as” OR. But we as a field should know more about what they are doing if we are going to be part of this business direction.

Can BI and OR cooperate for mutual benefit? I look at the issue from the perspective of two technology experiences.

A traveling salesman story

The baffling non-communication between BI and OR immediately reminds me of Traveling Salesman Problem (TSP) research back in the late 80s. The OR (Groestchel, Cook, and others using polyhedral combinatorics), Computer Science (Bell Labs folks using Lin-Kernighan type heuristics) and Artificial Intelligence (using ANN, Simulated Annealing and related “bio-physical” approaches) communities were all working on the same problem, with little cross-pollination. I was fortunate to attend a conference at Rice University that invited leading lights from camp to develop a holistic vision of TSP research.

As a graduate student, I recall sharp surprise by how the limitations and strengths of each school drove its research. The AI folks armed with elegant learning-oriented methods, focused on small – say a few dozen cities – instances, often drawn directly from the real world. Their algorithms would solve series of iteratively changing instances, reliably and fast. Though there was little emphasis on mathematical optimality, their results were accessible to non-expert readers.

The CS folks were at the other end of the size spectrum. They were interested in instances drawn from telecom networks, with millions of cities, which needed to be solved in seconds. Their algorithms were essentially heuristics, so their results focused on empirical performance measures – solve time and distance from optimality. The application areas often involved real-time setting, so narrowing the range of outcome variability was the key research driver.

The OR approach – based largely on cutting plane methods for integer programming – implicitly assumed that provable optimality was the goal. The focus was on solving larger and larger problems to optimality ever faster. At that time, if I recall correctly, the largest solved involved a few thousand cities. However, the methods were extremely non-robust. There were many small “deviant” instances, for which known cutting planes were insufficient. Even worse, within any instance class and size, there could be 10x (or more) variability in solution times.

I recall that by the end of the conference, each research school far better appreciated the others’ methods. For example, in my own research (on a different but related problem) I discovered that a hybrid technique combining a randomized Lin-Kernighan type “kick” heuristic with a cutting plane method worked best. In the broader research community however, the TSP dialog did not lead to a common “business direction”. Perhaps this was due to the mechanics of the research enterprise, which can be quite territorial.
However, it is possible for commonality of goals to induce a commonality of trajectory. This is evident from the CPLEX experience at Ilog.

CPLEX et Ilog

In the late 90s, Ilog was an emerging vendor of optimization tools to the business community. Faced with the challenge of expanding beyond its European base, it found that its bread-and-butter CS-based optimization technology – Constraint Programming – was little known and unappreciated in the US. There, OR techniques such as linear and integer programming were pervasive. A small company called CPLEX Optimization was the leading US vendor. Though the initial idea was for Ilog to license and embed CPLEX technology, Ilog ended up acquiring the company.

The CPLEX acquisition created a host of integration issues for Ilog. Most of them are irrelevant to our story, but a key problem was the mix of mutual incomprehension and suspicion between the two technical groups. The Ilog folks, hailing from a CS background and flush with a string of European successes based on Constraint Programming (CP), viewed OR methods as non-robust and old school. (Constraint Programming gained prominence in the 1980s in the AI community. The simplex method in OR dates all the way back to 1948!) CPLEX folks had a hard time moving beyond the fact that CP offered neither provable optimality nor guaranteed solve times.

As CPLEX rose to become Ilog’s most successful product over the years, an interesting convergence occurred. CPLEX increasingly incorporated CP-like heuristics into its core engine. And CP product development increasingly became driven by CPLEX’s strengths: fast performance and large-scale numerical robustness. Ilog also developed products incorporating CPLEX and CP, creating offerings still without competition in the marketplace. Today, tensions between the two product lines continue to generate tectonic rumbles. But ten years down the road, ILOG has surmounted the original mutual incomprehension between the two teams and grown to the premier player in its space.

BI and OR: Quo Vadis?

In TSP research, though all parties shared goals, convergence did not occur in any meaningful sense. Within Ilog, CP and CPLEX were able to cross-pollinate and cooperate to mutual benefit. What are we to make of these two cases?

One difference, perhaps the key difference, is that the cooperative motivations for TSP researchers were weak. Research is a Darwinian enterprise; schools of knowledge play in a zero-sum sandbox. At Ilog, it was clear that CPLEX needed to succeed for Ilog to grow. Shared purpose – helped along by managerial directives – drove the cooperation of two technologies.

On the face of it, the BI/OR situation is more akin to the TSP case: there is (in Al Gore’s evocative phrase) no controlling authority forcing cooperation. However, from the OR perspective, the situation is far more fraught. The OR profession is in a period of existential self-reflection. What is the proper domain of OR? Is OR mainly a research area or is it a profession (or both)? If it’s a profession, should it include the usual attributes such as licensure, professional closure, and a professional code of conduct?

In contrast, BI has captured much of the “intelligent decision-making” mind-share in the business world in a very short period of time. And all that, on the basis of market-ready IT-aware technologies, with a less established research base. The strengths and weaknesses of the two communities are starkly complementary. And as the need for smart IT expands, the multiplicative impact of cooperation (in whatever form) has inescapable logic.

Figure 1: Standish Group 2006 US IT Survey

Note that even armed with full-featured methodologies such as RUP, IT projects are high-risk. As Figure 1 (above) shows, though the US success rate was up to 35% by 2006, one in five project still failed outright. The remaining 46% are condemned to the purgatory of “late, over budget, or below requirements”, presumably enroute to shelfware status.

In Competing on Analytics, a 2006 Harvard Business Review article (available here in research report form, and here as a teaching aid) Tom Davenport argues for a cross-Enterprise approach to integrating Operations Research (OR, also sometimes referred to as Advanced Analytics, Management Science or Quantitative Analysis) into a company’s decision-making. As organizational decision-making becomes increasing IT-based under competitive pressures, more tactical and operational decision-support systems (DSS) have cores built on integer programming, advanced forecasting, Monte Carlo simulation, ant colony optimization, and similarly complex techniques. Moving from piston aircraft to turbines to jets to rockets requires increasingly complex power plant engineering and operating regimes. Similarly, going from simpler rule-based systems that automate decision trees to optimization-driven systems that implement OR technology require a re-examination of IT development processes.

As Davenport documents with examples from companies such as Marriott and Procter & Gamble, OR can maximize value extraction from a company’s resource base. But it also throws up new risks and challenges in implementation. For instance, the major challenges in implementing a database query-driven system (for instance, a standard CRM) involve business process mapping, user adoption, server and bandwidth sizing, etc. However, turbo-charge that application with an OR engine (for example, by embedding a constraint programming-based product configurator,) and you now need to worry about reliable search performance, solution persistence, and other optimization artifacts. Though high-value in concept, OR-powered IT implementations increase project risks along new dimensions. As OR capabilities become checklist items on RFPs, the need for a robust, scalable, and practical OR-aware implementation methodology becomes more urgent.

In upcoming articles, I will discuss practice methodology development for OR-powered systems, starting with a review of how practitioners view practical OR.

It goes without saying that smart business decision-making involves advanced analytical techniques from the fields of Operations Research, Statistics, and Artificial Intelligence. These include Predictive Analytics and Data Mining (to detect correlative, possibly causal, relationships in historical data), Monte Carlo Simulation and Decision Analysis (to simulate the impact of such relationships and tease out key sensitivities in anticipative decision-making) and various flavors of Optimization (both mathematically-driven algorithms and less-structured, heuristic approaches). But the application of a wide spectrum of techiques does not necessarily guarantee smart decisions. Intechné differentiates itself by explicitly focusing on an often overlooked issue in applying advanced analytics in the enterprise: Risk.

Viewed in the context of applying advanced analytics to business improvement, risk is like the weather: everybody talks about it, but nobody does anything about it. Or, nothing systematic at any rate. At the purely technical level, approaches such as mathematical optimization produce “brittle” decisions; very small changes in input can produce dramatically different recommendations. Perhaps that’s not of concern in the few situations where human operators are not in the associated decision chain. But in general, analytics are used to support decision, not execute them. Unexplainable, non-intuitive, or volatile decisions often force operators to work around their decision-support systems, or even completely ignore them. For instance, we found a sophisticated SAP/APO installation essentially ignored by its users (demand planners at a Food & Beverage company) because it couldn’t auto-profile different product types. While the overall MAPE was ok, sales forecasts for individual products diverged from reality in unexpected ways.

When it comes to delivering decision-support technology based on advanced analytics, a host of implementation risks arise beyond standard IT development risks. For instance, the response times of constraint programming models can decay exponentially with input size. (This is quite different from, as an example, rule-based decision engines.) Encountered unexpectedly, such non-responsiveness leads to expensive and disruptive modeling/algorithmic rework at an advanced project stage.

All in all, the business of delivering smart business decision-making is characterized by these, and many other risks. In informal feedback from colleagues and clients, we find that project mortality in this area is unacceptably high: about one in three advanced analytics projects fails to perform to expectation.

An active risk management orientation lies at the core of our vision for Intechné. In forthcoming communications we describe how this orientation is incorporated into our practice culture, and how it has been shown to improve client results.