6 PerspectivesApril 28, 2010
Avoiding the Limitations of Decision Trees: A Few Tips from Mediators Who Use Them
No tool is perfect, and decision trees are no exception. A few of the comments on prior posts in this series have explored some of the problems mediators and advocates have with decision trees and what we can do about them. Today we’ll explore both the problems some mediators see in decision tree analysis and how those mediators make the tool more effective for parties and their counsel.
Garbage In, Garbage Out
Garbage in, garbage out is a problem in all forms of data analysis. In decision tree analysis every input — from numerical values to probabilities to the construct of the diagram itself — affects the output, or the expected monetary value of your case. Los Angeles mediator Joseph C. Markowitz summed it up nicely in Quantifying Uncertainty:
One [kind of uncertainty that decision trees can never resolve] is the “garbage in, garbage out” kind of uncertainty. When a lawyer says he has a 60% chance of prevailing on a claim, all that represents is a seat of the pants feeling about the case. That is not to say that the lawyer’s assessment is wrong — it could be based on years of experience and some pretty good hunches about what juries might do with a case. But it is also not a very firm number to start with. And when you start with a very unscientific probability number as a basis for calculating the value of a case, you are conveying a degree of certainty about the ultimate value that is probably not warranted. Add in the uncertainties about things like appeals over issues that have not even materialized yet, and you are dealing with a whole lot of uncertainty.
How do you avoid the GIGO problem? As Geoff Sharp told us in Risk analysis in mediation, “[g]ut instinct, sloppy guesswork and grey hair no longer seem to be enough in complex, high stakes mediation.” This is the time, as Montreal’s Brian Daley reminds us, for client and counsel to “deconstruct a complex lawsuit into discrete steps and possible outcomes that can pave the way for appropriate decision-making.” There’s no shortcut to rigorous analysis and candid evaluation, and I can’t make up one here.
Avoiding the “Black Box Syndrome”
Don Philbin, a Texas mediator and negotiation consultant (whose post ADR Decision Tree — Fit the Forum to the Specific Issues is one of the most creative and useful tools out there), tells us by way of our recent discussion on LinkedIn’s Commercial and Industry Arbitration and Mediation Group:
[Decision trees] are interesting graphics that help engage the frontal cortex. The chief criticism seems to be that they take a number of wild guesses and roll them back to a very precise number that often is not one of the remedies available in the case. So I’ll usually start with a hand drawn version on tear sheets and then put it in the computer later so we don’t have the black box syndrome. Since the most valuable part of adding this science to the art of negotiation is that it breaks the psychological link to the number “we like,” I prefer animated outcome curves that move with various adjustments for costs, cognitive errors, etc. They graphically display the difference between possibilities (y-axis) and probabilities (x-axis) without overly focusing on the one net expected value that a decision tree produces.
Don is right that computer-generated decision trees can produce the black box syndrome, and starting with a hand-drawn map is an easy fix. (For those who want more on how to actually use decision trees, see Don’s article in the Harvard Negotiation Law Review styled The One Minute Manager Prepares for Mediation: A Multidisciplinary Approach to Negotiation Preparation.)
Math Isn’t Enough
In addition to his point about garbage in/garbage out, Los Angeles mediator Joe Markowitz explored a second decision tree problem on Mediation’s Place:
[Another] uncertainty you cannot eliminate is the uncertainty of predicting how people will deal with the choice between the mathematical probabilities of the decision tree analysis and the concrete offer on the table. So if you tell the plaintiff that they have the choice between the defendant’s $50,000 offer and a 30% chance of scoring a million dollar verdict at trial (or you tell the defendant that they can pay the plaintiff $200,000 or face a 10% chance that the plaintiff will get a million dollar judgment), you would think that taking your chances at trial would be the obviously better option in both cases, but a lot of people will take the offer rather than risk getting nothing (or pay the unreasonable demand even if they are very unlikely to lose at trial). Their choice will depend on how much they like to gamble and a lot of other psychological factors that cannot be very easily quantified. Remember how Monty Hall used to offer people the choice between something like $500 in an envelope or a one in three chance of winning a new car? A surprising number of people chose the envelope. . . .
So yes doing the decision tree exercise can be very useful, but mainly to demonstrate to people just how much uncertainty remains in front of them if they want to continue to litigate, and perhaps as a means of making people comfortable with the fairness of the settlement offer. That kind of analysis can’t really give you a precise indication of what a case is “worth,” but it might help some people decide if they want to settle or not.
And even if clients could be persuaded to follow the math, Michael Webster has commented here previously that decision trees can’t produce a true expected value of a case:
A complex decision tree might help you find outcomes that you had not thought about, but it is highly unlikely that decision analysis is ever going to progress to give you an expected value of a case.
While Marc Victor has a persuasive response to this last point, Philip J. Loree Jr., who writes at the Loree Reinsurance and Arbitration Law Forum, reminds us that “[t]he art here is predicting how the decision maker will analyze the case.” Phil and Joe are right — standing alone, the expected monetary value at the end of the decision tree doesn’t settle the case. But a decision tree, and the analysis required to get it done right, can highlight unforeseen contingencies, uncertainties and opportunities in the case. With a little help from a good neutral, this may help your decisionmaker decide if she wants to settle or not — and that’s plenty to ask for.
Some Lawyers Aren’t Good at Math
For a final tip, Phil Loree implicitly acknowledges what many of us already know — lawyers aren’t always good at math — and proposes a solution:
[I]f you have someone on your team who is an actuary, statistician or mathematician — or simply someone with a solid quantitative background like an engineer or skilled accountant — you might want to enlist that person’s assistance to be sure that at least the quantitative aspects of the analysis are on the mark.
We always get smart, practical advice from Phil.
Try some of the tips above to make your next decision tree more effective. You’ll be glad you did.
[Note: For for more on Decision Tree Analysis, Settlement Perspectives' series on decision trees includes:
- Decision Tree Analysis in Litigation: The Basics
- Why Should You Try a Decision Tree in Your Next Dispute?
- Advanced Decision Tree Analysis in Litigation: An Interview with Marc Victor, Part I
- Advanced Decision Tree Analysis in Litigation: An Interview With Marc Victor, Part II
- Decision Trees in Mediation: A Few Examples, and
- Avoiding the Limitations of Decision Trees: A Few Tips from Mediators Who Use Them (this post)]
Categories: Decision Trees,Mediation,Negotiation,Settlement,Tactics
6 Perspectives:
Philip J. Loree Jr. — Thursday, April 29, 2010 4:46 am
John,
Congratulations on another well-written and very useful post. And thanks for the mentions and kind words!
These decision trees really are an important tool when you are dealing with complex cases. Sure we could all predict a settlement outcome by pulling a number out of the air based on gut feeling, but that estimated outcome would not be grounded in any quasi-objective reality.
Decision trees are to negotiators what anchors are to boats. If you throw down your anchor in the right place, set it properly and feed out enough line, then it will probably ensure that you will not drift too far from where you want to be. But as most boaters know, to be effective, anchors require a good deal of line to be paid out relative to the depth of the water (known as “scope”). And that means you have to accept that, absent a prevailing wind, you are going to drift over a fairly large perimeter of water relative to where you dropped the anchor. So there’s certainly security but only a mild degree of precision in terms of your location.
The same holds true for decision trees: If you use them correctly they can identify a range of realistic settlement outcomes. That range however, can be pretty broad, and just as an anchor won’t necessarily keep you in the same — let alone the right — place at all times, so too a decision tree won’t precisely identify what might be a perfect settlement outcome. Security, yes…precision, no.
That may not be the tightest analogy you’ve ever heard, but I hope you get my drift (no pun intended…:)).
Phil
John DeGroote — Thursday, April 29, 2010 5:27 am
Phil–
Your anchoring analogy really works here. You give us quite the sound byte with “If you throw down your anchor in the right place, set it properly and feed out enough line, then it will probably ensure that you will not drift too far from where you want to be.” Thanks for that, Phil, and thank you for helping me learn something new about a topic other than negotiation today, too.
As I said yesterday, “[w]e always get smart, practical advice from Phil.”
John DeGroote
Philip J. Loree Jr. — Saturday, May 1, 2010 10:58 am
John,
You’re way too kind!
I suspect you’d enjoy boating, although I must say that it consumes large amounts of both time and money. I’m actually selling my current boat (28′ sportfisherman) and swearing off boating for a bit. When I do get back into it, I will probably downsize — say, 23′-26′.
Phil
Jacob Ruytenbeek — Wednesday, May 5, 2010 11:11 pm
Philip,
What a great analogy! I hadn’t thought of that one, but it’s perfect.
John,
Thanks for addressing some of the limitations of decision trees. More education about decision trees and their inherent limitations is needed so that the mediation community understands that they are only a single tool in the mediator’s toolbox and not a panacea for settlement. They should be applied at the right time, in the right context, by a mediator who has a defined purpose for using the decision tree. In such a scenario, it’s a very valuable tool to have.
Joe Markowitz — Thursday, May 6, 2010 10:17 pm
I admire the creativity of the anchor analogy, but if I had to go with a boating metaphor, I would probably say that decision trees are more like the boat’s navigation system. Especially in the old days before GPS, which makes it all too easy, in order to get your position and plot your course, you have to factor in a lot of variables like current and windspeed and average velocity and course heading, etc. And depending on those changing factors, you might make your harbor by nightfall or you might not. Maybe you can tell I’m a sailor and not a powerboater.
I say that any metaphor is good in a storm, and any kind of analysis that helps people appreciate all of the vagaries of their position is useful. And thanks again for all the mentions.
Deirdre — Saturday, September 4, 2010 1:12 pm
John – I first ran across “decision trees” analysis when the U.S. Army was using it for priority target risk analysis. It’s amazing how far the process has come since then and the diverse applications it has generated. Thanks. Deirdre