RAMAS Elicitor

Introduction

RAMAS Elicitor provides several techniques for choosing an input distribution from sparse or imprecise knowledge. It can be used to construct probability distributions, probability boxes (p-boxes), or Dempster-Shafer structures (DSSs). Using p-boxes and DSSs, the uncertainty inherent in the data can be rigorously represented and propagated through calculations. Quantitatively incorporating uncertainty in this way provides an automatic and exhaustive sensitivity analysis of the importance of uncertainty in the inputs.

Elicitor enhances transparency and reproducibility which increases the credibility of an analysis by encouraging the clear and complete documentation of all sources of information used and the choices and judgements made in constructing each input. Elicitor provides a self-documenting interface which prompts the user for documentation and description of source material and for explanation and justification of each decision made in the derivation of an input. Units are checked and unit conversions are performed automatically to pre-empt errors. Color coding highlights potential problems and directs the user to incompletely documented data sources or unjustified decisions.

Elicitor provides eight ways to use the information you have to specify an uncertain number. To specify your number you may enter:

Elicitor collates the information you enter to specify a probability distribution, a probability box, or a random sets (Dempster-Shafer) structure. Click on the type of information you wish to use in the list above or scroll down this page for more details about how to tell Elicitor what you know and what Elicitor does with that knowledge.

Clicking the data tab opens a page where data may be entered to constrain an input distribution. Both intervals and precise points may be entered. Intervals are preferable in cases where the precise value of the data points are uncertain due to, for example, measurement error. For interval data, enter the minimum in the column labeled "Left" and the maximum in the column labeled "Right". For point observations, enter the same number in the left and right columns. Check the specify weights checkbox to enter weights. The minimum and maximum may be the extremes of the data entered or may be extremes derived from other considerations. The radio boxes present four methods for using the data to specify the uncertain number: stochastic mixture, sample rule, Saw-Yang-Mo, and Kolmogorov-Smirnov confidence limits.

In the figure below, the data consists of three precise points: 1, 5, and 13, and two intervals: [3,4] and [9,12]. The minimum and maximum have been specified as 0 and 14, respectively, and the "Sample rule" method has been chosen to construct bounds around the input distribution given the data.

Selecting the shape tab opens a page where distribution shape information may be entered to constrain the input distribution. You may not know the exact distribution shape, but you may know, for example, that it is surely symmetric. In this case, Elicitor will constrain the set of possible input distributions to those where the mean equals the median. Similarly, if you know that the distribution is surely positive, only distributions with minimums greater than or equal to zero will be considered. Nine other qualitative parameters, including unimodality, concavity, convexity, increasing or decreasing hazard, discreteness, continuousness, and integer or constant valued, may be specifed to constrain the input distribution.

In some cases, you may know that the input is distributed according to a named distribution family. Forty such families are implemented in Elicitor and may be selected from the dropdown menu labeled "Distribution shape". When you select a distribution family, boxes will appear in which you can enter values of the parameters necessary to specify the named distribution. If uncertain numbers, e.g. intervals, are used to parameterize a named distribution, the result will be a p-box or DSS which bounds all of the distributions that could result from any possible realization of the parameters.

Clicking on the parameters tab opens a page where moments may be entered to specify constraints on an input distribution. Moments may be entered as precisely known points or as intervals. Elicitor uses combinations of known moments, whether certain or uncertain in nature, to constrain the limits of the input distribution.

Clicking on the percentiles tab opens a page where order statistics may be entered to specify constraints on an input distribution. Order statistics may be entered as precisely known points or as intervals. Elicitor uses known percentiles, whether certain or uncertain in nature, to constrain the limits of the input distribution. Precise knowledge of an order statistic provides a powerful constraint on the input distribution because the universe of possible inputs is reduced to only those which pass through that single point.

When you select the coverages tab, a page appears which allows you to specify intervals which "cover" the input distribution with known probability. Elicitor uses these intervals and their corresponding probabilities to constrain the limits of the input distribution. An interval is entered by typing its minimum in the "Left bound" box and its maximum in the "Right bound" box, selecting either "exactly", "no less than", or "no greater than" from the "covers" drop down menu, and entering a probability in the "Probability" box. Probabilities entered must be between zero and one in each probability box, however the probabilities across boxes need not add to one. The figure below shows the results of entering two intervals, [1,4] and [2,3]. The first interval covers no less than 95% of the range of the input distribution. The second interval covers no less than 75%. This knowledge constrains the input distribution to be between the yellow and green bounds shown in the display box at the top of the screen.

When you select the density tab, a page where you may specify limits on a probability density function (pdf) is displayed. Double-clicking anywhere in the probability density graph will bring up a little spreadsheet where you can enter x-values and corresponding densities for both the "bubble" and the "cap". The cap is the upper bound on the density at any particular point on the x-axis. The bubble is the lower bound on the density. Between the bubble and the cap is the area from which the uncertain pdf may be drawn. Double-clicking anywhere on the spreadsheet reverts to the pdf graph. Any point on the graph may be manipulated by clicking and dragging it to the desired location. You can zoom in on any part of the image by clicking and dragging a box from upper right to lower left around an area of interest. To un-zoom, click and drag from lower right to upper left.

Selecting the graph tab brings up a page where you can manipulate the graphs produced on other pages. The figure below shows the graph tab after the data was entered in the data tab. You may use your mouse to drag the left and right bounds to adjust the input distribution as necessary. Note that you can zoom in on any part of the image by clicking and dragging a box from upper right to lower left around an area of interest. To un-zoom, click and drag from lower right to upper left.

Please see the FAQ below for answers to some common questions. Questions may also be e-mailed to troy@ramas.com

Frequently Asked Questions:

Unknown value of a parameter or other input.

Entering a Dempster Shafer structure.

Summarizing sampling data.

Getting a picture of a result.

Using a small sample to characterize a distribution.

Sample values were entered but the Show button doesn’t produce a graph.

Yellow colored input fields.

Red colored input fields.

Contradicting inputs.

Avoiding a knot of red color coding and error messages.

Turning Exploratory mode off.

"Test" buttons.

Question: What if I don’t know the value of a parameter or other input?

Answer: If you don’t know the value of a parameter or other input, leave it blank to indicate total ignorance. Alternatively, you can enter a (possibly wide) interval to represent what little you are sure about. To indicate an upper or lower bound only, use the > or < characters. For instance, entering “>0” means that you know the number is positive.

Question: How do I enter a Dempster Shafer structure?

Answer: To enter a Dempster-Shafer structure, use the Data page, entering the focal elements as intervals and the masses as weights that add up to unity. Select “Stochastic mixture” to form the Dempster-Shafer structure. If the focal elements are not closed intervals, then Elicitor can only handle your Dempster-Shafer if you transform them into closed intervals, perhaps by taking their convex hulls. If the frame of discernment W is not the real line, then you would need to map W into the reals.

Question: Can I use Elicitor to summarize sampling data from calculations?

Answer: To use Elicitor to summarize sampling data from calculations, just enter the sample values on the Data page. If you have scalar values, be sure that the lower and upper values of the interval are the same.

Question: How do I get a picture of the result?

Answer: If you double-click on the graph on the upper part of the display (or right-click on it and select “Copy graph to clipboard”), a copy of it will be placed on the clipboard in metagraphics format. You can then paste the graph into other Windows applications. You may want to modify the background and line colors via the Input/Options menu choice.

Question: I only have a very small number of sample values? What can I say about the distribution from which they are drawn?

Answer: When you only have a very small number of sample values, the Kolmogorov-Smirnov confidence limits (available on the Data page) are probably the best representation of your uncertainty unless the underlying population is also very small.

Question: I entered sample values on the Data page but the Show button doesn’t produce a graph? What gives?

Answer: If you entered sample values on the Data page but the Show button doesn't produce a graph, you probably forgot to enter the weights. If you don’t want to use weights, uncheck the “Specify weights” box.

Question: Why did the input field turn yellow after I typed in it?

Answer: After you type in it, an input field may turn yellow. Inputs are color coded to indicate their status. Yellow means that you’ve specified the numerical value but you haven’t yet given a verbal justification to support the entry. To do so, right click on the input field and type in the box labeled “Justification” on the dialog that pops up. You can also specify other information on this dialog. You can change the color used by selecting Input/Options from the main menu and clicking on the little yellow panel labeled “Unjustified input”. You can also turn off the color coding altogether by unchecking the box above the yellow panel, but this is not recommended.

Question: Why did an input field turn red?

Answer: An input field may turn red at any time. The red color coding is telling you that this input contradicts another input you made. You may want to click on the Reset button (in the lower, left-hand corner of the display) and then click on the Yes button to clear all the entries you’ve made on the page. You can change the color used to indicate contradictions by selecting Input/Options from the main menu and clicking on the red panel labeled “Contradiction”.

Question: My inputs have over-determined the uncertain number and created contradictions. What should I do?

Answer: Conflicting inputs tell you that you don’t know as much as you thought you did about the uncertain number. When inputs contradict, you need to relax one or more of them. You may want to click on the Reset button (in the lower, left-hand corner of the display) to clear entries you’ve made. If you want to change some of your inputs but the program keeps resetting them, it may help to press the Relax button, or select Bounding/Relaxed from the main menu. This turns off the automatic cascading that Elicitor uses to propagate your inputs. It can also be useful to select Bounding/Current page to have Elicitor build bounds on the uncertain number based only on the information you’ve specified on the current tabbed page of inputs. Alternatively, you may select Bounding/Envelope if you want to have Elicitor use information from all the pages.

Question: How do I keep from getting tangled up in a knot of red color coding and error messages about bounds crossing or left bounds being greater than right bounds?

Answer: The red color coding and these error messages are telling you that your inputs contradict each other. If you’re just browsing and want to see what the program can do, select Bounding/Exploratory from the main menu. This will turn off both the color coding and the automatic cascading of inputs and allow you to change inputs as you like without having to make them all agree with each other.

Question: How do I turn off Exploratory mode?

Answer: You can’t turn off Exploratory mode. You need to close the program (by selecting File/Exit from the main menu) and invoke Elicitor again fresh. The reason for this is that, when you are exploring, the program is still accumulating histories (those hints that appear when the mouse lingers over an input field) and keeping track of information that you don’t want to be part of any actual session record. Restarting the program from scratch gets rid of this extraneous information. Although you can’t turn off the exploratory mode, you can turn cascading back on (by selecting Bounding/Constrained from the main menu) and restore color coding (by checking the box labeled “Color code input fields” on the options dialog which you invoke by selecting Input/Options from the main menu).

Question: What do the buttons that say "Test" do?

Answer: Buttons labeled “Test” appear in the lower, right-hand corners of several of the pages if you choose the "Exploratory" option from the "Bounding" menu. Clicking on a Test button will put some example inputs in the current screen for you to help you understand how to use the software.

Some related documents of interest are listed below. Many may be acquired by clicking on them.

Abbas, A.E. (2003). Entropy methods for univariate distributions in decision analysis. Bayesian Inference and Maximum Entropy Methods in Science and Engineering: 22nd International Workshop, edited by C.J. Williams, American Institute of Physics. (Available through on-line booksellers.)

Barlow, R.E. and A.W. Marshall (1964). Bounds for Distributions with Monotone Hazard Rate, II. The Annals of Mathematical Statistics 35(3):1258-1274. (JSTOR access required.)

Caselton, W. F. and W. Luo (1992). Decision making with imprecise probabilities: Dempster-Shafer theory and application. Water Resources Research 28(12): 3071–3083. (Available by subscription at http://www.agu.org/journals/wr/)

Dempster, A.P. (1967). Upper and lower probabilities induced by a multi-valued mapping. Annals of Mathematical Statistics 38: 325-339.

Ferson, S. (2002). RAMAS Risk Calc 4.0 Software: Risk Assessment with Uncertain Numbers. Lewis Publishers, Boca Raton, Florida.

Ferson, S. and J.Hajagos (2004). Rigorous and (often) best possible <<>>. Reliabilty and Engineering System Safety

Ferson, S. and W.T. Tucker (2003). Reliability of Risk Analyses for PAH-contaminated Groundwater. In Groundwater Quality Modeling and Management Under Uncertainty, S. Mishra (Ed.). Proceedings of the Probabilistic Approaches & Groundwater Modeling Symposium held during the World Water and Environmental Resources Congress in Philadelphia, Pennsylvania, June 24-26, 2003. Reston, VA: ASCE Publications.

Ferson, S., V. Kreinovich, L. Ginzburg, D.S. Myers and K. Sentz (2003). Constructing probability boxes and Dempster-Shafer structures. SAND2002-4015. Sandia National Laboratory, Albuqueruqe, NM. (compressed pdf file in "http://www.ramas.com/unabridged.zip")

Ferson, S., R.B. Nelsen, J. Hajagos, D.J. Berleant, J. Zhang, W.T. Tucker, L.R. Ginzburg and W.L. Oberkampf (2004). Dependence in probabilistic modeling, Dempster-Shafer theory, and probability bounds analysis. SAND2004-3072. Sandia National Laboratory, Albuqueruqe, NM. (pdf, postscript and Microsoft Word files compressed together with readme.txt in "http://www.ramas.com/depend.zip")

Fetz, T. and M. Oberguggenberger (2004).

Halpern, J.Y. (2003). Reasoning about Uncertainty. MIT Press, Cambridge, Massachusetts.

Hammond, K.R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. Ocford University Press, New York.

Helton, J. and W.T. Oberkampf (2004). Reliabilty and Engineering System Safety (special issue)

Jaynes, E.T. (edited by G. Larry Bretthorst) (2003). Probability Theory: The Logic of Science. Cambridge University Press.

Johnson, P.E., F. Hassebrock, A.S. Duran and J.H. Moller (1982). Multimethod study of clinical judgment. Organizaional Behavior and Human Performance 30: 226-().

Klir, G. and B. Yuan (1995). Fuzzy Sets and Fuzzy Logic: Theory and Applications. Prentice Hall, Upper Saddle River, New Jersey.

Kozine, I.O. and L.V. Utkin (2004). An approach to combining unreliable pieces of evidence and their propagation in a system response analysis. Reliability Engineering and System Safety 85: 103-112.

Kriegler, E. and () Held ();

Kreinovich, V. et al. (2004). Approximating Uncertainty about Output from Black-box Functions Whose Inputs Are Characterized by Uncertain Numbers. Applied Biomathematics Technical Report, Setauket, New York.

Lee, R.C. and W.E. Wright (1994). Development of human exposure-factor distributions using maximum-entropy inference. Journal of Exposure Analysis and Environmental Epidemiology 4:329-341.

Luo, W. B. and B. Caselton (1997). Using Dempster-Shafer theory to represent climate change uncertainties. Journal of Environmental Management 49: 73–93.

Mayo, D. 1996. Error and the Growth of Experimental Knowledge. The University of Chicago Press, Chicago.

Regan, H.M., B.E. Sample and S. Ferson (2002a). Comparison of deterministic and probabilistic calculation of ecological soil screening levels. Environmental Toxicology and Chemistry 21: 882-890.

Regan, H.M., B.K. Hope and S. Ferson (2002b). An analysis of uncertainty in a food web exposure model. Human and Ecological Risk Assessment ().

Saw, J.G., M.C.K. Yang and T.C. Mo (1984). Chebyshev inequality with estimated mean and variance. The American Statistician 38: 130-132.

Saw, J.G., M.C.K. Yang and T.C. Mo (1988). Corrections. The American Statistician 42: 166.

Sentz, K. and S. Ferson (2002). Combination of Evidence in Dempster-Shafer Theory. SAND2002-0835 Technical Report, Sandia National Laboratories, Albuquerque, NM.

Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press, Princeton, New Jersey.

Solana, V. and N.C. Lind (1990). Two principles for data based on probabilistic system analysis. Proceedings of ICOSSAR '89, 5th International Conferences on Structural Safety and Reliability. American Society of Civil Engineers, New York.

Tonon, F., A. Bernardini, and A. Mammino. (2000). Reliability analysis of rock mass response by means of Random Set Theory. Reliability Engineering and System Safety 70(3):263-282.

Tonon, F. (2004). On the use of Random Set Theory to bracket the results of Monte Carlo simulations. Reliable Computing 10:107-137.

Tucker, W.T. and S. Ferson (2003). Probability bounds analysis in environmental risk assessments. Manuscript in pdf form available for download at http://www.ramas.com/pbawhite.pdf.

Yager, R.R. 1986. Arithmetic and other operations on Dempster-Shafer structures. International Journal of Man-machine Studies 25: 357-366.

	RAMAS Elicitor
Introduction Purpose Downloads Technical support Related documents Other links Contact us	Welcome to the RAMAS Elicitor home page. Use the links to the left to find information, downloads, and support for the Elicitor software.

Zoom in: drag from upper left to lower right.	Zoom out: drag from lower right to upper left.