publications

2023

Improving Evaluation Using Visualization Decision-Making Models: A Practical Guide

Melanie Bancilhon, Lace Padilla, and Alvitta Ottley

In Visualization Psychology, 2023

Abs HTML PDF

In visualization research, evaluation is a crucial step to assess the impact of visualization on decision-making. Existing work often gauges how good a visualization is by measuring its ability to induce accurate and fast judgment. While those measures provide some insight into the efficacy of a graph, underlying cognitive processes responsible for reasoning and judgment are often overlooked when they can have significant implications for visualization recommendation. Cognitive processes do not need to be a black box. There exist multiple models that explain decision processes, such as theories from behavioral economics and dual process theories. In this chapter, we compare and contrast different models and advocate for integrating a dual-process approach to visualization evaluation in the context of decision-making. Our goal is to guide visualization researchers to adopt a more granular approach to evaluation at the intersection of visualization and cognitive science.
What Do We Mean When We Say “Insight”? A Formal Synthesis of Existing Theory

Leilani Battle, and Alvitta Ottley

IEEE Transactions on Visualization and Computer Graphics, 2023

Abs HTML PDF

Researchers have derived many theoretical models for specifying users’ insights as they interact with a visualization system. These representations are essential for understanding the insight discovery process, such as when inferring user interaction patterns that lead to insight or assessing the rigor of reported insights. However, theoretical models can be difficult to apply to existing tools and user studies, often due to discrepancies in how insight and its constituent parts are defined. This paper calls attention to the consistent structures that recur across the visualization literature and describes how they connect multiple theoretical representations of insight. We synthesize a unified formalism for insights using these structures, enabling a wider audience of researchers and developers to adopt the corresponding models. Through a series of theoretical case studies, we use our formalism to compare and contrast existing theories, revealing interesting research challenges in reasoning about a user’s domain knowledge and leveraging synergistic approaches in data mining and data management research.
What Exactly is an Insight? A Literature Review

Leilani Battle, and Alvitta Ottley

2023 IEEE Visualization and Visual Analytics (VIS), 2023

Abs PDF

Insights are often considered the ideal outcome of visual analysis sessions. However, there is no single definition of what an insight is. Some scholars define insights as correlations, while others define them as hypotheses or aha moments. This lack of a clear definition can make it difficult to build visualization tools that effectively support insight discovery. In this paper, we contribute a comprehensive literature review that maps the landscape of existing insight definitions. We summarize key themes regarding how insight is defined, with the goal of helping readers identify which definitions of insight align closely with their research and tool development goals. Based on our review, we also suggest interesting research directions, such as synthesizing a unified formalism for insight and connecting theories of insight to other critical concepts in visualization research.
Do You Trust What You See? Toward A Multidimensional Measure of Trust in Visualization

Saugat Pandey, Oen G McKinley, R Jordan Crouser, and 1 more author

In 2023 IEEE Visualization and Visual Analytics (VIS), 2023

Abs PDF

Few concepts are as ubiquitous in computational fields as trust. However, in the case of information visualization, there are several unique and complex challenges, chief among them: defining and measuring trust. In this paper, we investigate the factors that influence trust in visualizations. We draw on the literature to identify five factors likely to affect trust: credibility, clarity, reliability, familiarity, and confidence. We then conduct two studies investigating these factors’ relationship with visualization design features. In the first study, participants’ credibility, understanding, and reliability ratings depended on the visualization design and its source. In the second study, we find these factors also align with subjective trust rankings. Our findings suggest that these five factors are important considerations for the design of trustworthy visualizations.
Mini-VLAT: A Short and Effective Measure of Visualization Literacy

Saugat Pandey, and Alvitta Ottley

Computer Graphics Forum, In Proceedings of the 25th EG Conference on Visualization (EuroVis), 2023

BEST PAPER AWARD

Abs HTML PDF

The visualization community regards visualization literacy as a necessary skill. Yet, despite the recent increase in research into visualization literacy by the education and visualization communities, we lack practical and time-effective instruments for the widespread measurements of people’s comprehension and interpretation of visual designs. We present Mini-VLAT, a brief but practical visualization literacy test. The Mini-VLAT is a 12-item short form of the 53-item Visualization Literacy Assessment Test (VLAT). The Mini-VLAT is reliable (coefficient omega = 0.72) and strongly correlates with the VLAT. Five visualization experts validated the Mini-VLAT items, yielding an average content validity ratio (CVR) of 0.6. We further validate Mini-VLAT by demonstrating a strong positive correlation between study participants’ Mini-VLAT scores and their aptitude for learning an unfamiliar visualization using a Parallel Coordinate Plot test. Overall, the Mini-VLAT items showed a similar pattern of validity and reliability as the 53-item VLAT. The results show that Mini-VLAT is a psychometrically sound and practical short measure of visualization literacy.
Human-Computer Collaboration for Visual Analytics: an Agent-based Framework

Shayan Monadjemi, Mengtian Guo, David Gotz, and 2 more authors

Computer Graphics Forum, In Proceedings of the 25th EG Conference on Visualization (EuroVis), 2023

Abs PDF

The visual analytics community has long aimed to understand users better and assist them in their analytic endeavors. As a result, numerous conceptual models of visual analytics aim to formalize common workflows, techniques, and goals leveraged by analysts. While many of the existing approaches are rich in detail, they each are specific to a particular aspect of the visual analytic process. Furthermore, with an ever-expanding array of novel artificial intelligence techniques and advances in visual analytic settings, existing conceptual models may not provide enough expressivity to bridge the two fields. In this work, we propose an agent-based conceptual model for the visual analytic process by drawing parallels from the artificial intelligence literature. We present three examples from the visual analytics literature as case studies and examine them in detail using our framework. Our simple yet robust framework unifies the visual analytic pipeline to enable researchers and practitioners to reason about scenarios that are becoming increasingly prominent in the field, namely mixed-initiative, guided, and collaborative analysis. Furthermore, it will allow us to characterize analysts, visual analytic settings, and guidance from the lenses of human agents, environments, and artificial agents, respectively.
Why Combining Text and Visualization Could Improve Bayesian Reasoning: A Cognitive Load Perspective

Melanie Bancilhon, Amanda Wright, Sunwoo Ha, and 2 more authors

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023

Abs PDF

Investigations into using visualization to improve Bayesian reasoning and advance risk communication have produced mixed results, suggesting that cognitive ability might affect how users perform with different presentation formats. Our work examines the cognitive load elicited when solving Bayesian problems using icon arrays, text, and a juxtaposition of text and icon arrays. We used a three-pronged approach to capture a nuanced picture of cognitive demand and measure differences in working memory capacity, performance under divided attention using a dual-task paradigm, and subjective ratings of self-reported effort. We found that individuals with low working memory capacity made fewer errors and experienced less subjective workload when the problem contained an icon array compared to text alone, showing that visualization improves accuracy while exerting less cognitive demand. We believe these findings can considerably impact accessible risk communication, especially for individuals with low working memory capacity.

2022

Inferential Tasks as an Evaluation Technique for Visualization

Ashley Suh, Ab Mosca, Shannon Robinson, and 4 more authors

In Proceedings of the 24th EG Conference on Visualization, 2022

BEST PAPER AWARD

Abs PDF

Designing suitable tasks for visualization evaluation remains challenging. Traditional evaluation techniques commonly rely on ‘low-level’ or ‘open-ended’ tasks to assess the efficacy of a proposed visualization, however, nontrivial trade-offs exist between the two. Low-level tasks allow for robust quantitative evaluations, but are not indicative of the complex usage of a visualization. Open-ended tasks, while excellent for insight-based evaluations, are typically unstructured and require time-consuming interviews. Bridging this gap, we propose inferential tasks: a complementary task category based on inferential learning in psychology. Inferential tasks produce quantitative evaluation data in which users are prompted to form and validate their own findings with a visualization. We demonstrate the use of inferential tasks through a validation experiment on two well-known visualization tools.
Guided Data Discovery in Interactive Visualizations via Active Search

Shayan Monadjemi, Sunwoo Ha, Quan Nguyen, and 3 more authors

In Proceedings of IEEE Visualization and Visual Analytics (VIS), 2022

Abs PDF

Recent advances in visual analytics have enabled us to learn from user interactions and uncover analytic goals. These innovations set the foundation for actively guiding users during data exploration. Providing such guidance will become more critical as datasets grow in size and complexity, precluding exhaustive investigation. Meanwhile, the machine learning community also struggles with datasets growing in size and complexity, precluding exhaustive labeling. Active learning is a broad family of algorithms developed for actively guiding models during training. We will consider the intersection of these analogous research thrusts. First, we discuss the nuances of matching the choice of an active learning algorithm to the task at hand. This is critical for performance, a fact we demonstrate in a simulation study. We then present the results of a user study for the particular task of data discovery guided by an active learning algorithm specifically designed for this task.
A Unified Comparison of User Modeling Techniques for Predicting Data Interaction and Detecting Exploration Bias

Sunwoo Ha, Shayan Monadjemi, Roman Garnett, and 1 more author

IEEE Transactions on Visualization and Computer Graphics, In Proceedings of IEEE Visualization and Visual Analytics (VIS), 2022

Abs PDF

The visual analytics community has proposed several user modeling algorithms to capture and analyze users’ interaction behavior in order to assist users in data exploration and insight generation. For example, some can detect exploration biases while others can predict data points the user will interact with before that interaction occurs. Researchers believe this collection of algorithms can help create more intelligent visual analytics tools. However, the community lacks a rigorous evaluation and comparison of these existing techniques. As a result, there is limited guidance on which method to use and when. Our paper seeks to fill this gap by comparing and ranking eight user modeling algorithms based on their performance on a diverse set of four user study datasets. We analyze exploration bias detection, data interaction prediction, and algorithmic complexity, among other measures. Based on our findings, we highlight open challenges and new directions for analyzing user interactions and visualization provenance.
Vizxp: A visualization framework for conveying explanations to users in model reconciliation problems

Ashwin Kumar, Stylianos Loukas Vasileiou, Melanie Bancilhon, and 2 more authors

In Proceedings of the International Conference on Automated Planning and Scheduling, 2022

Abs PDF

Advancements in explanation generation for automated planning algorithms have moved us a step closer towards realizing the full potential of human-AI collaboration in real-world planning applications. Within this context, a framework called \emphmodel reconciliation has gained a lot of traction, mostly due to its deep connection with a popular theory in human psychology, known as the \emphtheory of mind. Existing literature in this setting, however, has mostly been constrained to algorithmic contributions for generating explanations. To the best of our knowledge, there has been very little work on how to effectively convey such explanations to human users, a critical component in human-AI collaboration systems. In this paper, we set out to explore to what extent visualizations are an effective candidate for conveying explanations in a way that can be easily understood. Particularly, by drawing inspiration from work done in visualization systems for classical planning, we propose a visualization framework for visualizing explanations generated from model reconciliation algorithms. We demonstrate the efficacy of our proposed system in a comprehensive user study, where we compare our framework against a text-based baseline for two types of explanations – domain-based and problem-based explanations. Results from the user study show that users, on average, understood explanations better when they are conveyed via our visualization system compared to when they are conveyed via a text-based baseline.
A Grammar-Based Approach for Applying Visualization Taxonomies to Interaction Logs

Sneha Gathani, Shayan Monadjemi, Alvitta Ottley, and 1 more author

In Computer Graphics Forum, In Proceedings of the 24th EG Conference on Visualization (EuroVis), 2022

Abs PDF

Researchers collect large amounts of user interaction data with the goal of mapping user’s workflows and behaviors to their high-level motivations, intuitions, and goals. Although the visual analytics community has proposed numerous taxonomies to facilitate this mapping process, no formal methods exist for systematically applying these existing theories to user interaction logs. This paper seeks to bridge the gap between visualization task taxonomies and interaction log data by making the taxonomies more actionable for interaction log analysis. To achieve this, we leverage structural parallels between how people express themselves through interactions and language by reformulating existing theories as regular grammars. We represent interactions as terminals within a regular grammar, similar to the role of individual words in a language, and patterns of interactions or non-terminals as regular expressions over these terminals to capture common language patterns. To demonstrate our approach, we generate regular grammars for seven existing visualization taxonomies and develop code to apply them to three public interaction log datasets. In analyzing these regular grammars, we find that the taxonomies at the low-level (i.e., terminals) show mixed results in expressing multiple interaction log datasets, and taxonomies at the high-level (i.e., regular expressions) have limited expressiveness, due to primarily two challenges: inconsistencies in interaction log dataset granularity and structure, and under-expressiveness of certain terminals. Based on our findings, we suggest new research directions for the visualization community to augment existing taxonomies, develop new ones, and build better interaction log recording processes to facilitate the data-driven development of user behavior taxonomies.
Testing theories of task in visual analytics

Leilani Battle, and Alvitta Ottley

ACM Interactions, 2022

Abs PDF

Theory research plays a critical role in many areas of computer science, including artificial intelligence, programming languages, and human-computer interaction. However, striving to encapsulate the human experience within a math equation or a concise model can seem reductive at best, especially when the goal is to design software that actively cooperates with people to achieve complex goals, such as in visualization research. That being said, naive reductions can also help researchers test their understanding of the complexities of human behavior. This article explores the role of theory in visualization research and our vision for the future of visualization theory work.

2021

FairVizARD: A Visualization System for Assessing Fairness of Ride-Sharing Matching Algorithms

Ashwin Kumar, Sanket Shah, Meghna Lowalekar, and 3 more authors

In Proceedings of the International Conference on Automated Planning and Scheduling (ICAPS), 2021

Abs PDF

There is growing interest in algorithms that match passengers with drivers in ride-sharing problems and their fairness for the different parties involved (passengers, drivers, and ride-sharing companies). Researchers have proposed various fairness metrics for matching algorithms, but it is often unclear how one should balance the various parties’ fairness, given that they are often in conflict. We present FairVizARD, a visualization-based system that aids users in evaluating the fairness of ride-sharing matching algorithms. FairVizARD presents the algorithms’ results by visualizing relevant spatio-temporal information using animation and aggregated information in charts. FairVizARD also employs efficient techniques for visualizing a large amount of information in a user friendly manner, which makes it suitable for real-world settings. We conduct our experiments on a real-world large-scale taxi dataset and, through user studies and an expert interview, we show how users can use FairVizARD not only to evaluate the fairness of matching algorithms but also to expand on their notions of fairness.
Does interaction improve bayesian reasoning with visualization?

Ab Mosca, Alvitta Ottley, and Remco Chang

In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021

Abs PDF

Interaction enables users to navigate large amounts of data effectively, supports cognitive processing, and increases data representation methods. However, there have been few attempts to empirically demonstrate whether adding interaction to a static visualization improves its function beyond popular beliefs. In this paper, we address this gap. We use a classic Bayesian reasoning task as a testbed for evaluating whether allowing users to interact with a static visualization can improve their reasoning. Through two crowdsourced studies, we show that adding interaction to a static Bayesian reasoning visualization does not improve participants’ accuracy on a Bayesian reasoning task. In some cases, it can significantly detract from it. Moreover, we demonstrate that underlying visualization design modulates performance and that people with high versus low spatial ability respond differently to different interaction techniques and underlying base visualizations. Our work suggests that interaction is not as unambiguously good as we often believe; a well-designed static visualization can be as, if not more, effective than an interactive one.

2020

DRAGON-V: Detection and Recognition of Airplane Goals with Navigational Visualization.

Christabel Wayllace, Sunwoo Ha, Yuchen Han, and 4 more authors

In Proceedings of Association for the Advancement of Artificial Intelligence (AAAI), 2020

Abs PDF

We introduce Detection and Recognition of Airplane GOals with Navigational Visualization (DRAGON-V), a visualization system that uses probabilistic goal recognition to infer and display the most probable airport runway that a pilot is approaching. DRAGON-V is especially useful in cases of miscommunication, low visibility, or lack of airport familiarity which may result in a pilot deviating from the assigned taxiing route. The visualization system conveys relevant information, and updates according to the airplane’s current geolocation. DRAGON-V aims to assist air traffic controllers in reducing incidents of runway incursions at airports.
Investigating the role of locus of control in moderating complex analytic workflows

R Jordan Crouser, Alvitta Ottley, Kendra Swanson, and 1 more author

2020

HONARABLE MENTION AWARD

Abs PDF

Throughout the last decade, researchers have shown that the effectiveness of a visualization tool depends on the experience, personality, and cognitive abilities of the user. This work has also demonstrated that these individual traits can have significant implications for tools that support reasoning and decision-making with data. However, most studies in this area to date have involved only short-duration tasks performed by lay users. This short paper presents a preliminary analysis of a series of exercises with 22 trained intelligence analysts that seeks to deepen our understanding of how individual differences modulate expert behavior in complex analysis tasks.
Competing Models: Inferring Exploration Patterns and Information Relevance via Bayesian Model Selection

Shayan Monadjemi, Roman Garnett, and Alvitta Ottley

IEEE Transactions on Visualization and Computer Graphics, 2020

Abs PDF

Analyzing interaction data provides an opportunity to learn about users, uncover their underlying goals, and create intelligent visualization systems. The first step for intelligent response in visualizations is to enable computers to infer user goals and strategies through observing their interactions with a system. Researchers have proposed multiple techniques to model users, however, their frameworks often depend on the visualization design, interaction space, and dataset. Due to these dependencies, many techniques do not provide a general algorithmic solution to user exploration modeling. In this paper, we construct a series of models based on the dataset and pose user exploration modeling as a Bayesian model selection problem where we maintain a belief over numerous competing models that could explain user interactions. Each of these competing models represent an exploration strategy the user could adopt during a session. The goal of our technique is to make high-level and in-depth inferences about the user by observing their low-level interactions. Although our proposed idea is applicable to various probabilistic model spaces, we demonstrate a specific instance of encoding exploration patterns as competing models to infer information relevance. We validate our technique’s ability to infer exploration bias, predict future interactions, and summarize an analytic session using user study datasets. Our results indicate that depending on the application, our method outperforms established baselines for bias detection and future interaction prediction. Finally, we discuss future research directions based on our proposed modeling paradigm and suggest how practitioners can use this method to build intelligent visualization systems that understand users’ goals and adapt to improve the exploration process.
Survey on the analysis of user interactions and visualization provenance

Kai Xu, Alvitta Ottley, Conny Walchshofer, and 3 more authors

Computer Graphics Forum, 2020

Abs PDF

There is fast-growing literature on provenance-related research, covering aspects such as its theoretical framework, use cases, and techniques for capturing, visualizing, and analyzing provenance data. As a result, there is an increasing need to identify and taxonomize the existing scholarship. Such an organization of the research landscape will provide a complete picture of the current state of inquiry and identify knowledge gaps or possible avenues for further investigation. In this STAR, we aim to produce a comprehensive survey of work in the data visualization and visual analytics field that focus on the analysis of user interaction and provenance data. We structure our survey around three primary questions: (1) WHY analyze provenance data, (2) WHAT provenance data to encode and how to encode it, and (3) HOW to analyze provenance data. A concluding discussion provides evidence-based guidelines and highlights concrete opportunities for future development in this emerging area. The survey and papers discussed can be explored online interactively at https://provenance-survey.caleydo.org.
Survey on individual differences in visualization

Zhengliang Liu, R Jordan Crouser, and Alvitta Ottley

In Computer Graphics Forum, 2020

Abs PDF

Developments in data visualization research have enabled visualization systems to achieve great general usability and application across a variety of domains. These advancements have improved not only people’s understanding of data, but also the general understanding of people themselves, and how they interact with visualization systems. In particular, researchers have gradually come to recognize the deficiency of having one-size-fits-all visualization interfaces, as well as the significance of individual differences in the use of data visualization systems. Unfortunately, the absence of comprehensive surveys of the existing literature impedes the development of this research. In this paper, we review the research perspectives, as well as the personality traits and cognitive abilities, visualizations, tasks, and measures investigated in the existing literature. We aim to provide a detailed summary of existing scholarship, produce evidence-based reviews, and spur future inquiry.
Adaptive and Personalized Visualization

Alvitta Ottley

Synthesis Lectures on Visualization, 2020

Abs

There is ample evidence in the visualization community that individual differences matter. These prior works highlight various personality traits and cognitive abilities that can modulate the use of the visualization systems and demonstrate a measurable influence on speed, accuracy, process, and attention. Perhaps the most important implication of this body of work is that we can use individual differences as a mechanism for estimating when a design is effective or to identify when people may struggle with visualization designs. These effects can have a critical impact on consequential decision-making processes. One study that appears in this book investigated the impact of visualization on medical decision-making showed that visual aides tended to be most beneficial for people with high spatial ability, a metric that measures a person’s ability to represent and manipulate two- or three-dimensional representations of objects mentally. The results showed that participants with low spatial ability had difficulty interpreting and analyzing the underlying medical data when they use visual aids. Overall, approximately 50% of the studied population were unsupported by the visualization tools when making a potentially life-critical decision. As data fluency continues to become an essential skill for our everyday lives, we must embrace the growing need to understand the factors that may render our tools ineffective and identify concrete steps for improvement. This book presents my current understanding of how individual differences in personality interact with visualization use and draws from recent research in the Visualization, Human-Computer Interaction, and Psychology communities. We focus on the specific designs and tasks for which there is concrete evidence of performance divergence due to personality. Additionally, we highlight an exciting research agenda that is centered around creating tailored visualization systems that are aligned with people’s abilities. The purpose of this book is to underscore the need to consider individual differences when designing and evaluating visualization systems and to call attention to this critical research direction.
Let’s Gamble: How a Poor Visualization Can Elicit Risky Behavior

Melanie Bancilhon, Zhengliang Liu, and Alvitta Ottley

In IEEE Visualization Conference (VIS), 2020

Abs PDF

Data visualizations are standard tools for assessing and communicating risks. However, it is not always clear which designs are optimal or how encoding choices might influence risk perception and decision-making. In this paper, we report the findings of a large-scale gambling game that immersed participants in an environment where their actions impacted their bonuses. Participants chose to either enter a lottery or receive guaranteed monetary gains based on five common visualization designs. By measuring risk perception and observing decision-making, we showed that icon arrays tended to elicit economically sound behavior. We also found that people were more likely to gamble when presented area proportioned triangle and circle designs. Using our results, we model risk perception and discuss how our findings can improve visualization selection.
Did You Get The Gist Of It? Understanding How Visualization Impacts Decision-Making

Melanie Bancilhon, and Alvitta Ottley

In VisPych Workshop, IEEE VIS, 2020

Abs PDF

As visualization researchers evaluate the impact of visualization design on decision-making, they often hold a one-dimensional perspective on the cognitive processes behind making a decision. Several psychological and economic researchers have shown that to make decisions, people rely on quantitative reasoning as well as gist-based intuition – two systems that operate in parallel. In this position paper, we discuss decision theories and provide suggestions to bridge the gap between the evaluation of decision-making in visualization and psychology research. The goal is to question the limits of our knowledge and to advocate for a more nuanced understanding of decision-making with visualization.

2019

Follow the clicks: Learning and anticipating mouse interactions during exploratory data analysis

Alvitta Ottley, Roman Garnett, and Ran Wan

Computer Graphics Forum, 2019

BEST PAPER AWARD FINALIST

Abs PDF

The goal of visual analytics is to create a symbiosis between human and computer by leveraging their unique strengths. While this model has demonstrated immense success, we are yet to realize the full potential of such a human-computer partnership. In a perfect collaborative mixed-initiative system, the computer must possess skills for learning and anticipating the users’ needs. Addressing this gap, we propose a framework for inferring attention from passive observations of the user’s click, thereby allowing accurate predictions of future events. We demonstrate this technique with a crime map and found that users’ clicks can appear in our prediction set 92% – 97% of the time. Further analysis shows that we can achieve high prediction accuracy typically after three clicks. Altogether, we show that passive observations of interaction data can reveal valuable information that will allow the system to learn and anticipate future events.