6 PerspectivesApril 28, 2010

Avoiding the Limitations of Decision Trees: A Few Tips from Mediators Who Use Them

No tool is perfect, and decision trees are no exception.  A few of the comments on prior posts in this series have explored some of the problems mediators and advocates have with decision trees and what we can do about them.  Today we’ll explore both the problems some mediators see in decision tree analysis and how those mediators make the tool more effective for parties and their counsel.

Garbage In, Garbage Out

Garbage in, garbage out is a problem in all forms of data analysis.  In decision tree analysis every input — from numerical values to probabilities to the construct of the diagram itself — affects the output, or the expected monetary value of your case.  Los Angeles mediator Joseph C. Markowitz summed it up nicely in Quantifying Uncertainty:

One [kind of uncertainty that decision trees can never resolve] is the “garbage in, garbage out” kind of uncertainty. When a lawyer says he has a 60% chance of prevailing on a claim, all that represents is a seat of the pants feeling about the case. That is not to say that the lawyer’s assessment is wrong — it could be based on years of experience and some pretty good hunches about what juries might do with a case. But it is also not a very firm number to start with. And when you start with a very unscientific probability number as a basis for calculating the value of a case, you are conveying a degree of certainty about the ultimate value that is probably not warranted. Add in the uncertainties about things like appeals over issues that have not even materialized yet, and you are dealing with a whole lot of uncertainty.

How do you avoid the GIGO problem?   As Geoff Sharp told us in Risk analysis in mediation, “[g]ut instinct, sloppy guesswork and grey hair no longer seem to be enough in complex, high stakes mediation.”  This is the time, as Montreal’s Brian Daley reminds us, for client and counsel to “deconstruct a complex lawsuit into discrete steps and possible outcomes that can pave the way for appropriate decision-making.”  There’s no shortcut to rigorous analysis and candid evaluation, and I can’t make up one here.

Avoiding the “Black Box Syndrome”

Don Philbin, a Texas mediator and negotiation consultant (whose post ADR Decision Tree — Fit the Forum to the Specific Issues is one of the most creative and useful tools out there), tells us by way of our recent discussion on LinkedIn’s Commercial and Industry Arbitration and Mediation Group:

[Decision trees] are interesting graphics that help engage the frontal cortex. The chief criticism seems to be that they take a number of wild guesses and roll them back to a very precise number that often is not one of the remedies available in the case. So I’ll usually start with a hand drawn version on tear sheets and then put it in the computer later so we don’t have the black box syndrome. Since the most valuable part of adding this science to the art of negotiation is that it breaks the psychological link to the number “we like,” I prefer animated outcome curves that move with various adjustments for costs, cognitive errors, etc. They graphically display the difference between possibilities (y-axis) and probabilities (x-axis) without overly focusing on the one net expected value that a decision tree produces.

Don is right that computer-generated decision trees can produce the black box syndrome, and starting with a hand-drawn map is an easy fix. (For those who want more on how to actually use decision trees, see Don’s article in the Harvard Negotiation Law Review styled The One Minute Manager Prepares for Mediation:  A Multidisciplinary Approach to Negotiation Preparation.)

Math Isn’t Enough

In addition to his point about garbage in/garbage out, Los Angeles mediator Joe Markowitz explored a second decision tree problem on Mediation’s Place:

[Another] uncertainty you cannot eliminate is the uncertainty of predicting how people will deal with the choice between the mathematical probabilities of the decision tree analysis and the concrete offer on the table. So if you tell the plaintiff that they have the choice between the defendant’s $50,000 offer and a 30% chance of scoring a million dollar verdict at trial (or you tell the defendant that they can pay the plaintiff $200,000 or face a 10% chance that the plaintiff will get a million dollar judgment), you would think that taking your chances at trial would be the obviously better option in both cases, but a lot of people will take the offer rather than risk getting nothing (or pay the unreasonable demand even if they are very unlikely to lose at trial). Their choice will depend on how much they like to gamble and a lot of other psychological factors that cannot be very easily quantified. Remember how Monty Hall used to offer people the choice between something like $500 in an envelope or a one in three chance of winning a new car? A surprising number of people chose the envelope. . . .

So yes doing the decision tree exercise can be very useful, but mainly to demonstrate to people just how much uncertainty remains in front of them if they want to continue to litigate, and perhaps as a means of making people comfortable with the fairness of the settlement offer. That kind of analysis can’t really give you a precise indication of what a case is “worth,” but it might help some people decide if they want to settle or not.

And even if clients could be persuaded to follow the math, Michael Webster has commented here previously that decision trees can’t produce a true expected value of a case:

A complex decision tree might help you find outcomes that you had not thought about, but it is highly unlikely that decision analysis is ever going to progress to give you an expected value of a case.

While Marc Victor has a persuasive response to this last point, Philip J. Loree Jr., who writes at the Loree Reinsurance and Arbitration Law Forum, reminds us that “[t]he art here is predicting how the decision maker will analyze the case.”  Phil and Joe are right — standing alone, the expected monetary value at the end of the decision tree doesn’t settle the case.  But a decision tree, and the analysis required to get it done right, can highlight unforeseen contingencies, uncertainties and opportunities in the case.  With a little help from a good neutral, this may help your decisionmaker decide if she wants to settle or not — and that’s plenty to ask for.

Some Lawyers Aren’t Good at Math

For a final tip, Phil Loree implicitly acknowledges what many of us already know — lawyers aren’t always good at math — and proposes a solution:

[I]f you have someone on your team who is an actuary, statistician or mathematician — or simply someone with a solid quantitative background like an engineer or skilled accountant — you might want to enlist that person’s assistance to be sure that at least the quantitative aspects of the analysis are on the mark.

We always get smart, practical advice from Phil.

Try some of the tips above to make your next decision tree more effective.  You’ll be glad you did.

[Note:  For for more on Decision Tree Analysis, Settlement Perspectives' series on decision trees includes:

1. Decision Tree Analysis in Litigation:  The Basics
   
2. Why Should You Try a Decision Tree in Your Next Dispute?
3. Advanced Decision Tree Analysis in Litigation:  An Interview with Marc Victor, Part I
   
4. Advanced Decision Tree Analysis in Litigation:  An Interview With Marc Victor, Part II
   
5. Decision Trees in Mediation:  A Few Examples, and
6. Avoiding the Limitations of Decision Trees:  A Few Tips from Mediators Who Use Them (this post)]

Categories: Decision Trees,Mediation,Negotiation,Settlement,Tactics

I welcome comments on this or any post. Please feel free to comment in the “Add Your Own Perspective” area below, whether you use your actual name or pseudonym. If you don’t feel comfortable commenting directly on this site, I can easily be reached by email at jd@johndegroote.com.