Trust and Machine Learning Interpretation Literature

by Fumeng (fumeng.yang@pnnl.gov, fy@brown.edu)

Fumeng: There are at least twice as many papers as what I list here...but I think these are enough unless we are missing important ones.

I. Trust

MUST AT LEAST CONSIDER TO CITE: Sheridan 1988, Sheridan 1980, 83, 84, Lee and See 2004, Lee and Moray 1994, Rempel et al 1985, Parasuraman 1993, Muir and Moray 1996, Singh 1993, S. Zuboff (1988), Miller 2002 2004, Hwang and Buergers 1997, Couch and Jones (1997), Rotter 1967, Dijkstra et al. 1998, Dijkstra 1999; Muir 1989

Most important papers

• (Survey) Trust in automation - Lee & See 2004

• Measuring Human-Computer Trust - Madsen & Gregor 2000

• (Very Important) Trust Calibration within a Human-Robot Team: Comparing Automatically Generated Explanations [cb 13, Wang & Pynadath, 2013]

  • The study is a between-subject design. Each participant interacted with one of the six simulated robots.
  • AMT
  • low, high x confidence-level explanation, observation explanation and no explanation
  • predisposition to trust [41], propensity to trust [42]
  • (feedback) If there is danger present inside the building, the human teammate will be fatally injured without the protective gear...have to restart
  • the impact of individual differences on trust is not the focus of this paper, such analyses and results are not included here.
  • While the scenario is military reconnaissance, it is simple enough that it does not require prior experience to complete the mission in the study,e.g., the task does not need knowledge of clearing procedures for searching buildings.
  • So the human teammate is incentivized to consider the robot’s findings before deciding how to enter the building.
  • The 2x3 ANOVA tests also show significant main effects of explanation on trust,

• (Very Important) Trust, control strategies and allocation of function in human-machine system (trust per trial) [1992, cb 1040]

  • 19 undergrads
  • 3 x 2h sessions, 3 days, 2h a day; each 10 trials x 3 or 6m, 10 x 6m
  • 10 training + 10, 5 + 1 error + rest; 20 all errors
  • orange juice pasteurization plant..? challenging task
  • auto or manual mode, or combine
  • receive 0.1$/trial if acc >90%; Muir 1989
  • a fault of 15%, 20%, 30%, & 35% err magnitude
  • NOT AT ALL, COMPLETELY
  • a description of trust and how it applies to inanimate objects (do we need this?)
  • questions they asked might be useful
  • casual model, recovery of trust
  • the direction of the casual relation between an occurrence of a fault and the decline in the level of trust seems unambiguous.
  • more faults, more relied on auto mode
  • After training trials operators could use any combination of automatic or manual control that they wished
  • Using naive operators facilitated an analysis of how trust and control strategies develop as operators are trained with new equipment.
  • while the operators were initially naive, the stable control strategies they developed by the 2nd hour indicate an understanding and patterns of controlling the system that might be comparable to trained operators. The model which we develop takes account of the learning curve, so that training to steady state as done by Muir (doc dis) is neither necessary nor apporitate in this experiment.

• (Very Important) Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err [cb 92, 2015]

  • Fumeng: The performance of the models they used is very poor. The estimating grades one is equal or slightly better or even worse than the participants. iIt is not a surprise for me that participants trust more on their own because they think they can improve while the models can't.

    In their experiments, when using a different model that absolutely outform human performance or the prediction is too difficult to make, people tend to slightly trust the model more.

  • evidence-based algorithms more accurately

  • when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster

  • algorithm aversion

  • Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them & Transparency

  • the admissions office had created a statistical model that was designed to forecast student performance. participants received detailed descriptions of the eight variables that they would receive about each applicant

  • Study 1

    • the 1st stage
      • control: skipped
      • model-and-human condition (n = 90): for each of the 15 applicants, make their own forecast, get feedback showing their own prediction, the model’s prediction, and the applicant’s true percentile.
      • human condition: make a forecast and then get feedback showing their own prediction and the applicant’s true percentile.
      • model condition: get feedback showing the model’s prediction and the applicant’s true percentile.
      • They were not incentivized for accurately making these forecasts
    • the 2nd stage
      • they would make 10 “official” incentivized estimates
      • Would you like your estimates or the model’s estimates to determine your bonuses for ALL 10 rounds?
      • Use only the statistical model’s estimates to determine my bonuses for all 10 rounds
      • Use only my estimates to determine my bonuses for all 10 rounds.
      • They received no feedback about their own or the model’s performance while completing these forecasts.
      • After the study, Belief and Confidence Measures (model and themselves)

  • Study 2

    • model and human
    • different bonus strategies

  • Study 3

    • predicting airlines passengers
    • 10 unincentivized forecasts
    • all participants completed 1 incentivized forecast instead of 10. Thus, their decision about whether to bet on the model’s forecast or their own pertained to the judgment of a single state.
    • a perfect forecast. This bonus decreased by $0.15 for each additional unit of error associated with their estimate
    • learned this payment rule before starting the first stage

  • Study 4

    • chose between a past participant’s forecasts and the model’s instead of between their own forecasts and the model’s.

Papers that are important because they had experiments

• Trust between humans and machines, and the design of decision aids [cb 591, Muir 1987]

  • Fumeng: this is not the thing... Muir Doc thesis in 1989 is important but I cam't find it...
  • Early in a relationship, a person bases his trust upon the predictability of another person's behaviours;
  • Later in a relationship, trust in another person is based upon the attribution of a dependable disposition; the opportunity must exist for the person to be undependable; the human requires to make a generalization about a machine's dependability.
  • The final stage in the growth of trust between humans is the development of faith

• The persuasive power of data visualization [cb 45, Pandey ... Bertini 2014]

• Affective Processes in Human–Automation Interactions [cb 52, Merritt 2011]

  • affect, mood was manipulated twice—once prior to training and again prior to the task. Thus, a 2 (training mood) × 2 (task mood) × 2 (machine accuracy) between-subjects design was used.
  • A total of 130 students participated for extra credit.
  • in an X-Ray Screening Task (cf. Merritt & Ilgen, 2008)
  • Included in the training program was information (verbal and visual) concerning how to identify guns and knives in the X-Ray Task. Participants then completed 10 practice trials and received feedback. They next were presented with information on how to use the AWD and watched the AWD perform 10 trials so that they could observe its reliability.
  • The actual reliability of the AWD was set to either 80% or 90% to ensure adequate variance
  • 60 X-ray trials divided into four task blocks.
  • before / after blocks
  • Liking Scale Items
  • trust per block
  • dynamic
  • Manipulation
    • Participants then completed 10 practice trials and received feedback. They next were presented with information on how to use the AWD and watched the AWD perform 10 trials so that they could observe its reliability.
    • recorded their own initial opinion regarding whether or not the image contained a weapon.
    • Next, they received advice from the AWD and recorded their final decision. Thus, it was possible to ascertain whether participants changed their opinion after viewing the AWD’s recommendation.

• I Trust It, but I Don’t Know Why: Effects of Implicit Attitudes Toward Automation on Trust in an Automated System [cb 63, Merritt 2013]

  • completed the explicit propensity to trust measure
  • implicit attitude toward automation measure
  • a 30-trial X-ray screening task similar to that used in past research (three blocks of 10)
  • cognitive load
  • six-item self-report scale employed by Merritt (2011). The item responses were on a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree).
  • This study was an online, three-condition, within-subjects design in which each participant encountered three levels of automation performance (clearly good, ambiguous, and clearly poor). The sample was 69 college students
  • No practice?
  • in an ambiguous / poor situation, implicit MIGHT be a better predictor of trust
  • On each slide, participants first provided their initial, unaided decision, next received the ABI’s advice, and then made a final decision. No performance feedback was provided. The 30 slides were divided into three blocks of 10.

• Can computer personalities be human personalities? [Nass et al, 1995, cb 665]

  • would respond to similar human personalities
  • 2x2 between , balanced, N = 48, dominant and submissive subjects were randomly matched with either a dominant or submissive computer.
  • desert survival items
  • rating 10-Likert scale

• Brains or Beauty: How to Engender Trust in User-Agent Interactions [Yuksel et al, cb 4, 2017]

  • we investigate two factors that can engender user trust in agents: reliability and attractiveness of agents
  • Participants took part in a within-subject 2 (high vs. low reliability)× 2 (high vs. neutral attractiveness) agency design. Participants answered four sets of 15 questions.
  • We stressed to participants that the agent presented at each trial was independent of agents from any previous trials in order to avoid attribution of perceived trustworthiness from previous agents.
  • In the high-reliability conditions, five out of the six suggestions offered by the agent were correct. In the low-reliability conditions, three out of the six suggestions offered by the agent were correct. The percentages for high and low reliability (83% and 50%, respectively) were taken from user trust literature. The order of the correct and incorrect answers were randomized in each trial.
  • Twenty participants
  • Metrics
  • Whether to hear about an answer
  • Self-Reported Measures of Trust in Agent,
  • Self-Reported Measures of Perceived Accuracy of Agent
  • Number of Correct Suggestions Accepted,
  • Number of Incorrect Suggestions Accepted
  • Length of Time Spent on Questions with Suggestions
  • Number of Timeouts with Suggestions
  • Number of Correct Answers
  • Number of Incorrect Answers

• Trust, self-confidence and authority in human-machine systems [Inagaki et al, 1998, cb 44]

  • simulated central heating system to maintain the temperature of the apartment complex shown at the bottom right of Fig. 1.
  • The experiment has a 2 (mode) x 2 (fault) x 3 (reliability, 100%, 90 %, and 70 %) x 2 (order...?) factorial design
  • participants? scale?

• (might be important) Foundations for an Empirically Determined Scale of Trust in Automated Systems [Jian et al 2000, cited by 493]

  • Based on how people feel about words, using factor analysis, cluster analysis, they contributed a questionnaire with 12-item 7 point

• Trust between man and machine in a teleoperation system [Dassonville et al, 1996, cb 24]

  • (1) self-confidence, (2) trust in others, (3) trust in machines, self-confidence, trust in others, and trust in machines in general
  • qualify the a priori trust of
  • interface...?

• The impact of cognitive feedback on judgment performance and trust with decision aids [Seong & Bisantz 2008, cb 61]

  • identify aircraft moving
  • interface (Jian... et al), left side
  • three scenarios per day for 2 consecutive days (3 x 50 x 2)
  • eight participants per condition
  • a brief training exercise session designed to familiarize participants with the identification task was performed with the aid of the experimenter.

• A model for predicting human trust in automated systems [Khasawneh et al, 2003, cb 16]

  • During the experiment participants were asked to rate their trust in the system for every board on a 0-100 scale, and then they were asked to rate their overall trust in the system at the end of each experimental session.
  • PCB boards containing 1, 2, 3, or no defects were inspected by the computer
  • 200 randomly ordered PCB boards

• Measurement of Trust Over Time in Hybrid Inspection Systems [Master et al, 2005, cb 17]

  • 6 grads and undergrads
  • The study used a single factor (response criterion at three system levels) within-subject design. Each repeated 3 times. 13-day time period
  • the subjects inspected 24/48 PCB boards of a perfect hybrid inspection system, where the computer did not make any errors.
  • interface, slider, left side

• The effects of errors on system trust, self-confidence, and the allocation of control in route planning [de Vries, 2003, cb 164]

  • they compared information reliability levels of 100%, 71%, and 43% in familiar versus unfamiliar environments.
  • participants had to complete a total of 26 route-planning trials...in the first 20 trials participants familiarized themselves with both the manual and the automatic mode.
  • The practice trials... complete 10 trials manually, and 10 automatically, ensuring that they would become equally experienced in both manual and automatic mode before they started completing the free trials.
  • 96 undergrads, a 2 (AER: low versus high) x 2 (MER: low versus high) between participants design.

• Trust in and Adoption of Online Recommendation Agents [Benbasat & Wang, cited by 557]

  • nature of trust in technological artifacts should not be fundamentally different from interpersonal trust
  • Measurement Items (Competence, Benevolence, Integrity)
  • Interface (9-point Likert Scale)
  • We randomly assigned participants to the eight experimental conditions described in the previous section.
  • two tasks, first choosing a digital camera for a good friend and then selecting another camera for a close family member.

• Trust in Adaptive Automation: The Role of Etiquette in Tuning Trust via Analogic and Affective Methods [Miller, 2004, cb 43]

  • tested 16 participants, a flight simulation task
  • Two levels of reliability were chosen: low, in which automation provided correct advice 60% of the time and high (80%).
  • 16 participants, the first two tasks

• The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle [Waytz et al, 2014, cb 155]

  • 100 participants, randomly assigned them to condition: Normal, Agentic, or Anthropomorphic
  • practice course -> two courses (6x2m) -> questionnaire

• The perceived utility of human and automated aids in a visual detection task [Dzindolet et al, 2002, cited by 165]

  • good references sources, ANOVA

  • Experiment 1

    • examine differences in the expectations people held of automated and human group members.
    • 2 (type of partner: human or automated) × 2 (expectation of aid’s performance: altered or control) between subjects design was utilized.
    • 68 students
    • 200 slides, believe the soldier was in the slide and the confidence they had in their decision on a 5-point Likert scale.
    • 4 practice trials (...told their aid usually made half as many errors as most students did. Participants performed four easy practice trials for which the aid supplied the correct decisions.)
    • Their expected performance relative to that of their aid on a 9-point Likert-format scale anchored with I will make many more errors and I will make far fewer errors.

  • Experiment 2

    • a simple task, examine the effect of perceived utility, self-serving biases, and bias toward automation on reliance on automated and human aids
    • 2 (type of partner: automated or human) × 2 (utility of the aid: equal or superior) × 2 (feedback: provided or not) between-subjects design; The Contrast Detector” (or “The Prior Participant”) “made 20 errors; you made 40.
    • 128 students
    • present the picture, a Likert scale 1-5 how confident .. decision
    • 4 practice trials
    • 20 (90%), 20 x 9 (~previous, halfen err rate), (feedback), 10 trials randomly chosen from the past 200 trials.
    • half of the sliders contain a solider
    • 10 trials: automated aid creates less bias
    • The strong bias toward self-reliance is also amazing...

  • Experiment 3

    • 71 students
    • All participants were told that their aid was imperfect (5%)
    • 3 (framing of past performance: positive ~, negative 10/200, none) × 2 (type of partner: human or automated)
    • similar procedure, 200 + 10
    • negative 10/200 + feedback will choose to follow the machine in the 10 trials

  • in every study participants did rely on automated and human aids differently

  • people expect machine makes less error than human partners

• (Important for data analysis & whether we should give feedback) Are Well-Calibrated Users Effective Users? Associations Between Calibration of Trust and Performance on an Automation-Aided Task [cb 8, Merritt et al, 2015]

  • Due to the relative simplicity of the task, no formal training was provided
  • participants were given no information about aid reliability prior to this task; therefore their calibration was not affected by any prior information.
  • Participants were simply instructed to visually search for guns and knives in the luggage images and that an automated aid would provide advice,which they could either choose to accept or reject when making a final decision.
  • first provided their initial opinion
  • then received the advice of a fictitious automated screener.
  • the participant made a final decision and received feedback on whether a weapon was in fact present in the image.
  • each 20-slide block x 4, after each block rate trust
  • In the increasing condition, the aid was correct 80%, 85%, 90%, and 95% of the time, respectively. In the decreasing condition the aid’s reliability decreased from 95% to 80% over the course of the blocks
  • A second potential explanation may be that although users are able to identify a general or overall trend in reliability across a sample of observations (particularly when given performance feedback), the ability to determine the accuracy of any individual decision is more error prone.
  • Although users may adjust their trust levels as reliability changes, they may not adjust the “correct” amount.

• Not All Trust Is Created Equal: Dispositional and History- Based Trust in Human-Automation Interactions [cb 189, Merrit & Ilgen 2008]

• Tuning trust using cognitive cues for better human-machine collaboration [Cai & Lin, 2010, cb 13]

  • 20 participants, within, Without/Visual/Auditory Assistance x Self-confidence (Fixed/Variable)

• Trust in New Decision Aid Systems [Atoyan et al, 2006, cb 56]

  • incorporates a seven point rating scale in the range from "not at all" to "extremely".
  • Reveal the rules and algorithms used by the automation, and if it is possible, keep the algorithms simple

• Affect- and cognition-based trust as foundations for interpersonal cooperation in organizations [McAllister, 1995, cb 7544]

  • human-human
  • Trust is cognition-based in that "we choose whom we will trust in which respects and under what circumstances, and we base the choice on what we take to be 'good reasons,' constituting evidence of worthiness.
  • Affective foundations for trust also exist, consisting of the emotional bonds between individuals
  • Reliably measured two dimensions of trust they labeled "reliableness" and "emotional trust"
  • Questionnaire, 25 items, 194 managers and professionals

Other Important Surveys

• (famous survey paper) An Integrative Model of Organizational Trust

• Similarities and differences between human–human and human–automation trust: an integrative review

Others

• Trust in decision aids A Model and its training implications

  • a key feature of trust is that it evolves not only as the user gains experience with an aid, but also as the user moves through the various phases of a particular mission or task.
  • nice summary of Muir, Barber, and Zuboff

• On Deep Learning for Trust-Aware Recommendations in Social Networks

  • I think they are just using strength of the social network

• A survey of trust in computer science and the Semantic Web

• Trustworthiness of command and control systems [Sheridan 1989, cb 104]

• Humans and Automation: Use, Misuse, Disuse, Abuse (Parasuraman & Riley, 1997, cb 2407)

  • theoretical, empirical, and analytical studies

• Trust in Close Relationships [Rempel et al, 2973, 1985]

  • questionaire based

• Measurement of Trust in Hybrid Inspection Systems: Review and Evaluation of Current Methodologies and Future Approach

  • ?

• Trust Models for Community Aware Identity Management

• Trust in Electronic Commerce: Definition and Theoretical Considerations

• On-line trust: concepts, evolving themes, a model

  • have references for human-human
  • trust between software agents

• Trust metrics in information fusion

• Towards a cognitive approach to human+machine cooperation in dynamic situations

• Measuring Levels of Trust [Couch & Jones, 1997, cb 253]

• Modeling Trust Negotiation for Web Services

• The Dyadic Trust Scale: Toward Understanding Interpersonal Trust in Close Relationship

  • Dyadic trust, human-human trust, benevolence, honesty
  • Participants 195, survey

• A Machine Learning Based Trust Evaluation Framework for Online Social Networks

  • Just the strength of social network

• How do We Learn to Trust? A Confirmatory Tetrad Analysis of the Sources of Generalized Trust [Glanville & Paxton, 2007, cb 218]

Summary from Jean

• Madsen & Gregor 2000 article and actual measure (word doc) – This is a state measure that captures the antecedents of trust (personal attachment, faith, reliability, technical competence, understandability). It is specifically designed for human-computer trust. I adapted it for one of my human-human trust studies and found it to be pretty good. I think all of the constructs predicted self-report trust in my study.

• Mayer adapted for human-machine trust – this is the self-report measure of trust that I have had the most success with. However, it was designed to measure human-human trust and I have never used this adapted version for human-machine trust (I adapted it for some upcoming human-machine trust research). You should be able to just substitute UAV for your tool. I think it’s pretty good, but if you don’t want to use this measure I believe Merritt has specifically designed a measure for human-machine trust I can get my hands on.

• Merritt et al 2015 - Table three in this article shows the items for the perfect automation schema measure. I don’t know a lot about this measure. I would consider it a trait or individual difference measure and the construct sounds really interesting to me. I have never used it, but I think I know some who have and can reach out to them if you want me to.

• Singh et al 1993 – I can’t for the life of me find this measure, but the reference is attached. I haven’t used this one either. As you can tell from the reference it has been around for a while and I have heard the questions are a bit dated. This is another trait/individual difference measure.

• For Madison & Gregor and Mayer I can give you a lot more info if you want it. For the others I know less. Overall, I thought these measures would probably be best based on our conversation. However, if none of these fit the bill or you are looking for more, I have a few more that I can pull together including some implicit measures of trust.

• (Muir) Automation that is predictable, dependable, and inspires faith that it will behave as expected in unknown situations will be seen as more trustworthy.

• (Lee and See) When trust in the automation was higher than self-confidence in manual operation, the participant tended to allocate the task to the automated system

• Toward Establishing Trust in Adaptive Agents

Trust Calibration

• A Design Methodology for Trust Cue Calibration in Cognitive Agents

  1. Select a scenario that involves trust in a cognitive agent
  2. Conduct a task analysis to identify critical trust related tasks
  3. Identify key pieces of information for operator decision making
  4. Verify pieces of information against the trust cue taxonomy
  5. Construct a visual display representing this information

• The Calibration of Trust in an Automated System: A Sensemaking Process

• A Design Methodology for Trust Cue Calibration in Cognitive Agents

  • case study

II. Machine Learning Interpretation via Visualization

Instance-based

• Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations (AD)
• Principles of explanatory debugging to personalize interactive machine learning

Feature- and Subset- based

• Familiarity Vs Trust: A Comparative Study of Domain Scientists’ Trust in Visual Analytics and Conventional Analysis Methods (AD)
• all papers from VIS 2017 Session: ML1
• “Why Should I Trust You?” Explaining the Predictions of Any Classifier
• A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations (AD)
• item The mythos of model interpretability
• Visualizing Dataflow Graphs of Deep Learning Models in TensorFlow
• Understanding black-box predictions via influence functions
• Grad-cam: Visual explanations from deep networks via gradient-based localization
• (!) The Role of Uncertainty, Awareness, and Trust in Visual Analytics (and its references)
• (references) Interacting with predictions: visual inspection of black-box machine learning models, ``By providing interactive partial dependence diagnostics, data scientists can understand how features affect the prediction overall.''
• Dis-function
• (references) ACTIVIS: Visual Exploration of Industry-Scale Deep Neural Network Models
• A Visual Interaction Framework for Dimensionality Reduction Based Data Exploration(AD)
• Opening black box Data Mining models using Sensitivity Analysis
• Opening the Black Box - Data Driven Visualization of Neural Networks.
• BaobabView: Interactive construction and analysis of decision trees.
• Seeking Inside the Black Box: Visualizing Statistical Learning With Plots of Individual Conditional Expectation
• Illuminating the “black box”: a randomization approach for understanding variable contributions in artificial neural networks
• Visualizing high-dimensional predictive model quality.
• Discovering Structure in High-Dimensional Data Through Correlation Explanation
• EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers
• Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization

Some Self-citations

• Empirical Analysis of the Subjective Impressions and Objective Measures of Domain Scientists’ Visual Analytic Judgments
• Beyond usability: Evaluation aspects of visual analytic environments.