My research in the 1960s and 70s compared decisions made by the participants in laboratory experiments with an ideal standard described by decision theory. My first discovery, with Ward Edwards, found that as people obtain more information about an uncertain situation they usually don't revise their probability assessments sufficiently, a phenomenon we described as 'conservatism', described below. My second discovery, with George Wright, showed differences in dealing with uncertainty by Eastern and Western participants in our experiments, listed below under cultural differences.
Recent research describes the harms from the misuse of psychoactive drugs, opioids and nicotine. My research for the European Medicines Agency and the European Community's IMI-PROTECT project, established the feasibility and desirability of creating explicit, quantitative models of the benefit-risk of prescription drugs. I'm now exploring ways to apply that research for the betterment of individual and public health.
Listed here are my 10 most-cited research publications, plus a few more.
Recent research describes the harms from the misuse of psychoactive drugs, opioids and nicotine. My research for the European Medicines Agency and the European Community's IMI-PROTECT project, established the feasibility and desirability of creating explicit, quantitative models of the benefit-risk of prescription drugs. I'm now exploring ways to apply that research for the betterment of individual and public health.
Listed here are my 10 most-cited research publications, plus a few more.
Overconfidence
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In D. Kahneman & P. Slovic & A. Tversky (Eds.), Judgment Under Uncertainty: Heuristics and Biases. Cambridge: Cambridge University Press.
One of the biases exhibited by the vast majority of people studied, under all sorts of circumstances, with many types of problems, is overconfidence: people are excessively sure of themselves in situations of uncertainty. With several colleagues, I carried out many studies on overconfidence, but stopped after Sarah Lichtenstein, Baruch Fischhoff and I reviewed the literature on the topic up to 1980. The phenomenon continues!
One of the biases exhibited by the vast majority of people studied, under all sorts of circumstances, with many types of problems, is overconfidence: people are excessively sure of themselves in situations of uncertainty. With several colleagues, I carried out many studies on overconfidence, but stopped after Sarah Lichtenstein, Baruch Fischhoff and I reviewed the literature on the topic up to 1980. The phenomenon continues!
Harm of psychoactive drugs
Nutt, D. J., King, L. A., Phillips, L. D., & on behalf of the Independent Scientific Committee on Drugs. (2010). Drug harms in the UK: a multicriteria decision analysis. The Lancet, 376(1558-65).
A team of over 20 experts drawn from different disciplines collectively assessed 20 psychoactive drugs, legal and illegal in the UK, on 16 physical, psychological and social harms. Alcohol emerged as the most harmful, followed by heroin, crack, methyamphetamine, cocaine and tobacco. Later papers focused on opioids and nicotine.
A team of over 20 experts drawn from different disciplines collectively assessed 20 psychoactive drugs, legal and illegal in the UK, on 16 physical, psychological and social harms. Alcohol emerged as the most harmful, followed by heroin, crack, methyamphetamine, cocaine and tobacco. Later papers focused on opioids and nicotine.
van Amsterdam, J., Nutt, D., Phillips, L., & van den Brink, W. (2015). European rating of drug harms. Journal of Psychopharmacology, 29(6), 655-660. doi: DOI: 10.1177/0269881115581980
This isn't in the top-10, but I've included it because it is a replication of the above Lancet study. With a different group of participants, the results correlated 0.993 with the UK study; a rare validation of expert judgement (which is required because there is hardly any scientific data about the illegal drugs). The shared definitions of harms between the two studies, on-the-spot peer review of expert judgement, and impartial facilitation of the decision conferencing process contributed to the similarity of final weighted scores for the two groups.
This isn't in the top-10, but I've included it because it is a replication of the above Lancet study. With a different group of participants, the results correlated 0.993 with the UK study; a rare validation of expert judgement (which is required because there is hardly any scientific data about the illegal drugs). The shared definitions of harms between the two studies, on-the-spot peer review of expert judgement, and impartial facilitation of the decision conferencing process contributed to the similarity of final weighted scores for the two groups.
Rogeberg O, Bergsvik D, Phillips LD, et al. A new approach to formulating and appraising drug policy. International Journal of Drug Policy 2018; 56: 144-52.
This isn't in the top-10 either, but is included because I believe it to be an important approach to building and testing drug policies about psychoactive drugs. It's an MCDA model of the seven key features to be addressed in any drug policy regime and includes 27 criteria reflecting ethical and normative concerns against which any existing or proposed policy regime can be evaluated. In testing the model for the current UK government's policies for alcohol and cannabis, we found that for each substance a policy regime of state control fared better than prohibition, free market or decriminalisation.
This isn't in the top-10 either, but is included because I believe it to be an important approach to building and testing drug policies about psychoactive drugs. It's an MCDA model of the seven key features to be addressed in any drug policy regime and includes 27 criteria reflecting ethical and normative concerns against which any existing or proposed policy regime can be evaluated. In testing the model for the current UK government's policies for alcohol and cannabis, we found that for each substance a policy regime of state control fared better than prohibition, free market or decriminalisation.
Conservatism in probability revision
Phillips, L. D., Hays, W. L., & Edwards, W. (1966). Conservatism in complex probabilistic inference. IEEE Transactions on Human Factors in Electronics, HFE-7, 7-18.
This paper reports the experiment in which Ward Edwards and I discovered conservatism, the failure to revise judged probabilities as much as is justified by Bayes’ theorem when more information is provided.
Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72, 346-355.
Worried that the above task was too complex (4 hypotheses and 12 kinds of data), we constructed the simplest task possible, (2 hypotheses and binomial data). Conservatism emerged again, and continues to do so in the more than 100 experiments that followed.
This paper reports the experiment in which Ward Edwards and I discovered conservatism, the failure to revise judged probabilities as much as is justified by Bayes’ theorem when more information is provided.
Phillips, L. D., & Edwards, W. (1966). Conservatism in a simple probability inference task. Journal of Experimental Psychology, 72, 346-355.
Worried that the above task was too complex (4 hypotheses and 12 kinds of data), we constructed the simplest task possible, (2 hypotheses and binomial data). Conservatism emerged again, and continues to do so in the more than 100 experiments that followed.
Requisite decision models
Phillips, L. D. (1982). Requisite decision modelling: A case study. The Journal of the Operational Research Society, 33, 303-311.
An application for a Midlands manufacturing company of a decision analysis that examined possible courses of action in light of considerable uncertainty about the outcomes which were multi-attributed. This experience led to an elaboration of 'requisite' in the next paper.
Phillips, L. D. (1984). A theory of requisite decision models. Acta Psychologica, 56, 29-48.
This launched the theory of doing decision analysis with groups of key players in which a decision model that is ‘good enough’ helps a decision maker resolve the issues that make it difficult to decide the way forward. Reports a real multi-attributed case which motivated my development of this approach.
An application for a Midlands manufacturing company of a decision analysis that examined possible courses of action in light of considerable uncertainty about the outcomes which were multi-attributed. This experience led to an elaboration of 'requisite' in the next paper.
Phillips, L. D. (1984). A theory of requisite decision models. Acta Psychologica, 56, 29-48.
This launched the theory of doing decision analysis with groups of key players in which a decision model that is ‘good enough’ helps a decision maker resolve the issues that make it difficult to decide the way forward. Reports a real multi-attributed case which motivated my development of this approach.
Decision support systems
Edwards, W., Lindman, H., & Phillips, L. D. (1965). Emerging technologies for making decisions, New Directions in Psychology, Volume 2 (pp. 261-325). New York: Holt, Rinehart and Winston.
The first hint of how what we now know as Bayesian Belief Networks could improve the speed of making valid inferences in situations of great uncertainty. Now the basis for many AI systems.
Edwards, W., Phillips, L. D., Hays, W. L., & Goodman, B. (1968). Probabilistic information processing systems: Design and evaluation. IEEE Transactions on Systems Science and Cybernetics, SSR-4, 248-265.
A large, complex experiment that confirmed the system design from the 1965 paper. We showed that people could make the required judgements of likelihoods about evidence, which when combined by a computer using Bayes’ theorem, provided a definitive intelligence assessment of the true state of affairs quicker and with less data than any other system. Intelligence analysts take note: we’ve known how to deal with unreliable evidence for over 50 years!
The first hint of how what we now know as Bayesian Belief Networks could improve the speed of making valid inferences in situations of great uncertainty. Now the basis for many AI systems.
Edwards, W., Phillips, L. D., Hays, W. L., & Goodman, B. (1968). Probabilistic information processing systems: Design and evaluation. IEEE Transactions on Systems Science and Cybernetics, SSR-4, 248-265.
A large, complex experiment that confirmed the system design from the 1965 paper. We showed that people could make the required judgements of likelihoods about evidence, which when combined by a computer using Bayes’ theorem, provided a definitive intelligence assessment of the true state of affairs quicker and with less data than any other system. Intelligence analysts take note: we’ve known how to deal with unreliable evidence for over 50 years!
Cultural differences in dealing with uncertainty and risk
Wright, G. N., & Phillips, L. D. (1980). Cultural variation in probabilistic thinking: Alternative ways of dealing with uncertainty. International Journal of Psychology, 15, 239-257.
Wright, G. N., Phillips, L. D., Whalley, P. C., Choo, G. T., Ng, K.-O., & Tan, I. (1978). Cultural differences in probabilistic thinking. Journal of Cross-Cultural Psychology, 9, 285-299.
When I was a PhD candidate at the University of Michigan in the early 1960s, I noticed that some of the Eastern students in our experiments on probability assessment were almost random generators of data. In my early years at Brunel University, George Wright and I, helped by our Eastern undergraduates, devised a series of experiments whose basic finding was that our Chinese, Malaysian and Indonesian participants seemed to think more in ‘black-and-white’ terms than Western participants, who were inclined to consider degrees of uncertainty. Later research showed that the Japanese are more like Westerners, and that the Chinese and others are perfectly capable of assessing accurate probabilities, but unless the task calls for probabilistic thinking, they aren’t inclined to do so. Many Westerners are like this, too. Some important implications here for East-West communication.
Wright, G. N., Phillips, L. D., Whalley, P. C., Choo, G. T., Ng, K.-O., & Tan, I. (1978). Cultural differences in probabilistic thinking. Journal of Cross-Cultural Psychology, 9, 285-299.
When I was a PhD candidate at the University of Michigan in the early 1960s, I noticed that some of the Eastern students in our experiments on probability assessment were almost random generators of data. In my early years at Brunel University, George Wright and I, helped by our Eastern undergraduates, devised a series of experiments whose basic finding was that our Chinese, Malaysian and Indonesian participants seemed to think more in ‘black-and-white’ terms than Western participants, who were inclined to consider degrees of uncertainty. Later research showed that the Japanese are more like Westerners, and that the Chinese and others are perfectly capable of assessing accurate probabilities, but unless the task calls for probabilistic thinking, they aren’t inclined to do so. Many Westerners are like this, too. Some important implications here for East-West communication.
Decision Analysis modelling in healthcare
Phillips LD. Best practice for MCDA in healthcare. In: Marsh K, Goetghebeur M, Thokala P, Baltussen R, eds. Multi-Criteria Decision Analysis to Support Healthcare Decisions: Springer International Publishing AG; 2017: 311-29.
This final chapter in this book first describes the decision-theoretic background of MCDA, then proposes 16 best practice principles for creating an MCDA model in a facilitated workshop or decision conference.
This final chapter in this book first describes the decision-theoretic background of MCDA, then proposes 16 best practice principles for creating an MCDA model in a facilitated workshop or decision conference.
Phillips LD, Fasolo B, Zafiropolous N, et al. Modelling the risk-benefit impact of H1N1 influenza vaccines. European Journal of Public Health 2013; 23(4): 674-8.
Shortly after the H1N1 influenza virus reached pandemic status in June 2009, my benefit–risk project team at the European Medicines Agency recognized this presented a research opportunity for testing the usefulness of a decision analysis model to assist in deliberations about approving vaccines soon based on limited data or waiting for more data. I facilitated an afternoon's decision conference in which we constructed a decision tree with two decision points, late September or late October, followed by 24 scenarios that included uncertainty about the severity of the pandemic and the safety and efficacy of the vaccines. The results showed, surprisingly, that uncertainty about safety and efficacy had little effect on the decision, and despite uncertainty about severity, enough information was available to show that fewer expected serious adverse events and deaths would occur for the earlier date. The vaccines were approved by the Committee on Human Medical Products (CHMP) in September.
Shortly after the H1N1 influenza virus reached pandemic status in June 2009, my benefit–risk project team at the European Medicines Agency recognized this presented a research opportunity for testing the usefulness of a decision analysis model to assist in deliberations about approving vaccines soon based on limited data or waiting for more data. I facilitated an afternoon's decision conference in which we constructed a decision tree with two decision points, late September or late October, followed by 24 scenarios that included uncertainty about the severity of the pandemic and the safety and efficacy of the vaccines. The results showed, surprisingly, that uncertainty about safety and efficacy had little effect on the decision, and despite uncertainty about severity, enough information was available to show that fewer expected serious adverse events and deaths would occur for the earlier date. The vaccines were approved by the Committee on Human Medical Products (CHMP) in September.
Benefit-risk balance of prescription drugs
Phillips, L. D., Fasolo, B., Zafiropoulos, N., & Beyer, A. (2011). Is quantitative benefit-risk modelling of drugs desirable or possible? Drug Discovery Today: Technologies, 8(1), e3-e10. doi: 10.1002/pds.3636
The first journal report from the European Medicine Agency's Benefit-Risk Project, which found that regulators in Europe assess the benefit-risk balance of drugs through discussion and voting; no quantitative modelling, other than conventional statistical analysis, is carried out to combine the favourable and unfavourable effects into a single index representing the benefit-risk balance. Field studies are reported to show how that can be accomplished by developing a multi-criteria decision analysis model in a group setting of experts and regulators.
The first journal report from the European Medicine Agency's Benefit-Risk Project, which found that regulators in Europe assess the benefit-risk balance of drugs through discussion and voting; no quantitative modelling, other than conventional statistical analysis, is carried out to combine the favourable and unfavourable effects into a single index representing the benefit-risk balance. Field studies are reported to show how that can be accomplished by developing a multi-criteria decision analysis model in a group setting of experts and regulators.
Phillips, L. D. (2014). Benefit-risk modeling of medicinal products: Methods and applications. Benefit-Risk Assessment in Pharmaceutical Research and Development A. Sashegyi, J. Felli and R. Noel. Boca Raton, FL, CRC Press: 59-96.
This is a detailed report comparing many structured methodologies for modelling the benefit-risk balance of drugs, and explaining how this can be done by applying multi-criteria decision analysis.
This is a detailed report comparing many structured methodologies for modelling the benefit-risk balance of drugs, and explaining how this can be done by applying multi-criteria decision analysis.
Nuclear Waste Disposal
Phillips, L. D., Egan, M., & Airoldi, M. (2006). MCDA Decision Conference (pp. 100). Winchester, UK: Catalyze Ltd.
The final three-day decision conference that incorporated judgements from experts and the public for managing the UK's high- and low-level nuclear waste. Concerns and issues about dealing with nuclear waste were obtained from a large public consultation, criteria were defined by experts who also scored the options on the criteria, and another public consultation engaged various constituents in weighting the criteria. Deep disposal won out over storage, which the government accepted. A new CoRWM is overseeing the Nuclear Decommissioning Agency in its search for a disposal site.
http://www.catalyze.co.uk/resources/docs/pdf/CoRWM%201716%20-%20Final%20MCDA%20report%20Apr%2006%20ver1.5.pdf
The final three-day decision conference that incorporated judgements from experts and the public for managing the UK's high- and low-level nuclear waste. Concerns and issues about dealing with nuclear waste were obtained from a large public consultation, criteria were defined by experts who also scored the options on the criteria, and another public consultation engaged various constituents in weighting the criteria. Deep disposal won out over storage, which the government accepted. A new CoRWM is overseeing the Nuclear Decommissioning Agency in its search for a disposal site.
http://www.catalyze.co.uk/resources/docs/pdf/CoRWM%201716%20-%20Final%20MCDA%20report%20Apr%2006%20ver1.5.pdf
Morton, A., Airoldi, M., & Phillips, L. (2009). Nuclear risk management on stage: The UK's Committee on Radioactive Waste Management. Risk Analysis, 29(5), 764-779.
Based on interviews with members of CoRWM, this research identifies the reasons for the success of the Committee's work: a combination of social processes and technical modelling.
Based on interviews with members of CoRWM, this research identifies the reasons for the success of the Committee's work: a combination of social processes and technical modelling.
Risk Analysis
Phillips, L. D., & Wisbey, S. J. (1993). The Elicitation of Judgmental Probability Distributions from Groups of Experts: A Description of the Methodology and Records of Seven Formal Elicitation Sessions Held in 1991 and 1992 (NSS/R282). Didcot, Oxfordshire: United Kingdom Nirex Limited.
In workshops I facilitated, groups of experts followed a structure process to generate probability distributions for 17 uncertain quantities describing factors that influence the transport of radionuclides from an underground repository via the geosphere into the biosphere. The uncertain quantities fell into seven categories: radionuclide inventory, geology and hydrogeology, near-field chemistry, far-field chemistry, biosphere, gas and human intrusion.
In workshops I facilitated, groups of experts followed a structure process to generate probability distributions for 17 uncertain quantities describing factors that influence the transport of radionuclides from an underground repository via the geosphere into the biosphere. The uncertain quantities fell into seven categories: radionuclide inventory, geology and hydrogeology, near-field chemistry, far-field chemistry, biosphere, gas and human intrusion.
Phillips, L. D. (1999). Group elicitation of probability distributions: Are many heads better than one? In J. Shanteau & B. Mellors & D. Schum (Eds.), Decision Science and Technology: Reflections on the Contributions of Ward Edwards (pp. 313-330). Norwell, MA: Kluwer Academic Publishers.
A study of group elicitation of probability distributions, showing the change in distributions as the group discusses the issues, and comparing the results of two separate groups on the same real-world task. The study shows that behavioural aggregation to consensus results in a different distribution from statistical aggregation.
A study of group elicitation of probability distributions, showing the change in distributions as the group discusses the issues, and comparing the results of two separate groups on the same real-world task. The study shows that behavioural aggregation to consensus results in a different distribution from statistical aggregation.
Probability Assessment
Beach, L. R., & Phillips, L. D. (1967). Subjective probabilities inferred from estimates and bets. Journal of Experimental Psychology, 75, 354-359.
Lee Roy Beach and I found little difference between probabilities that were directly assessed or inferred from betting behaviour.
Lee Roy Beach and I found little difference between probabilities that were directly assessed or inferred from betting behaviour.