Exploiting Causal Structure for Transportability in Online, Multi-Agent Environments. [Published at AAMAS-2022]

Autonomous agents may encounter the transportability problem when they suffer performance deficits from training in an environment that differs in key respects from that in which they are deployed. Although a causal treatment of transportability has been studied in the data sciences, the present work expands its utility into online, multi-agent, reinforcement learning systems in which agents are capable of both experimenting within their own environments and observing the choices of agents in separate, potentially different ones. In order to accelerate learning, agents in theseMulti-agent Transport (MAT) problems face the unique challenge of determining which agents are acting in similar environments,and if so, how to incorporate these observations into their policy.We propose and compare several agent policies that exploit local similarities between environments using causal selection diagrams,demonstrating that optimal policies are learned more quickly than in baseline agents that do not. Simulation results support the efficacy of these new agents in a novel variant of the Multi-ArmedBandit problem with MAT environments.

Schedulion: NET Prediction and Scheduling for LMU Athletics

Scheduling application built for the LMU athletics department that uses machine learning from KenPom data to predict team NET rankings, matchups with other teams, and a variety of scheduling mechanics to achieve a high strength of schedule.

Beaker: Interdisciplinary Research Connections Made Easier

Web application used to make interdisciplinary research at the university level simpler across department lines.

Source-specific Biases and Predictors in Online Political News

Examines the propensities of online news readers from different political affiliations to both seek and rate articles as biased as a function of the reported source.

Simulated Game Balancing in Settlers of K'tah

Balancing strategic video games can be challenging, especially when there are multiple interworking mechanics. A common way that developers ensure that games are well-balanced is through playtesting, where the game is released to small groups of players who provide feedback on the gameplay. However, playtesting can be resource-intensive, since it often requires many test players and the results are hard to judge objectively. This research sought to determine the value of programmatic simulations, as opposed to playtesting, for determining optimal game balance. The examined game, Settlers of K'tah, is a strategy game wherein players compete to collect resources, build armies, and fight off a zombie horde. To ensure that all victory paths in the game are equally difficult to achieve, Python simulations were written to tweak various in-game parameters such as building costs and battle outcome probabilities. Then, simulated agents competed against each other and the results of each game were collected, including how many points each player earned, the victory path taken, and the duration of the game. The simulation results were evaluated and different parameter settings were assessed based on how evenly-matched the games were. This led to a set of parameterized values that provided the optimal game balance for Settlers of K'tah. The results showed that simulations are a working method for determining game balance, but playtesting is still necessary for determining overall user satisfaction. In the future, agents that employ different gameplay strategies could be incorporated, as opposed to just the greedy approach.

Recommender Systems in Confounded Decision-Making Scenarios

The day-to-day decisions of people are often affected by Unobserved Confounders (UCs) which result in uncontrolled variations to human decision making and its outcomes. Although seemingly innocuous, there exists ample evidence that important decisions are subject to the influence of these confounding factors, like two physicians who would choose to treat the same patient in different ways. This may result in poor decision making that escapes detection from any dataset. The recommender system literature lacks research into the formalization of such confounded decision-making scenarios which the present study attempts to provide and assess. In this study, we suggest that the use of artificial recommendation systems will make progress in mitigating confounded decision making. Using word-association quizzes, we construct a reinforcement learning task in which the intuitive answer is always incorrect, and then observe the effect of a recommender system in correcting these intuitions. Performance across a number of recommender policies is discussed alongside implications for decision-assistance software.

Measuring Human Perceptiveness in Identifying Deep-Faked Images

Generative Adversarial Networks, or GANs, build upon the foundations of deep learning by introducing an "adversarial" / "discriminator" network that contrasts sample data with generated data, pushing the generative network to yield realistic results. With NVIDIA's addition of open source "style transfer" models, programs that generate realistic images are accessible to anyone. This technology creates real-world consequences, such as the abuse of generated facial images online through fake social media accounts. Yet, there is little research that focuses on the human ability to determine whether an image is real or generated and which types of fake images are more convincing. In this work, we document this ability through a survey that identifies how accurately participants can distinguish between generated and real images and which categories of images they are more likely to identify. In addition, the survey asks "cognitive reflection" questions to determine whether each participant relies more heavily on instinct or thoughtful reflection. Finally, participants are prompted to provide relevant demographic information. Our data should provide new insight on human accuracy to spot fakes within specific image categories and any correlations between one's reliance on instinct versus reflection and one's ability to identify fakes.

Challenging Human Instinct: Dynamic Difficulty Adjustment in Video Games

Dynamic Difficulty Adjustment (DDA) is the process of automatically adjusting the parameters of a video game based on the performance of a player to adapt the game's difficulty level to the player's changing abilities, and with the intention of keeping players engaged. Past studies have focused on understanding the design requirements for an effective DDA system in the context of keeping game players engaged and motivated. This study observes if changing the elements of a game's environment after a game player has already acclimated to the game creates a more challenging experience and potentially leads to cycles of adaptation throughout the playthrough. This research examines data collected from players of a 2-dimensional side-scroller game using responses to a post-play survey. The objective of the game is to jump onto moving platforms to (1) stay alive and (2) collect the coins placed on each platform. The game system consists of a metapolicy which chooses between various subpolicies. A subpolicy defines the pattern of the moving platforms presented in the game; more specifically, what the lengths and heights of the platforms will be. The game agent modifies the environment through off-policy reinforcement learning and causal inference in order to challenge game players' natural instincts with unpredictability. The agent evaluates the player's acclimation to a subpolicy based on the player's scores before switching to another subpolicy, and then measuring how the player performed after the subpolicy switch. The results and implications for interactive environments are discussed.

Briefcase: Intelligent Case Management

Briefcase is an interdisciplinary effort with the LMU Law School's Project for the Innocent. The Project provides free legal counsel to those who believe they have been wrongfully convicted, at which point the team reviews their case to attempt exoneration. The problem: the case briefs are anything but, sometimes 2000+ pages in length and place demands on the small team sorting through them. The Project has over 1000 cases in their queue, and are realistically able to process 30 a year; some convicts may therefore expire before their case has been reviewed, and so the team needs tools to help process these massive trial transcripts. Our elite team of Lucille Njoo and Manny Barreto are developing an app (Briefcase) that will provide Assisted Intelligence solutions to parse, label, annotate, collaborate on, and otherwise help streamline consumption and collaboration on input trial transcripts by the Project. The result may help increase throughput and lead to more exonerations of those wrongfully imprisoned.

Washington Abstract

This capstone-turned-startup app Washington Abstract is aimed at providing legislative transparency and advocacy on recent bills and active politicians at the federal level. My particular research interests are in application of what I call the "Legislative Genome," a feature of Washington Abstract aimed at forming a graphical causal model linking lobbies, politicians, and the bills they support (+ the language they add to those authored). The endeavor is an ambitious one that may take some time to reach fruition, but seems particularly important in the current political climate with growing distrust of representatives and the money behind them. I am working with students / co-founders Patrick Utz and Mohammed Hayat on a growing team that is also supported by Dr. David Choi in the School of Business.

Dynamic Difficulty Adjustment in Adversarial Games

Designing challenging but replayable video games gives players added value in their purchase and experience, but can be a difficult balance to strike as players become acclimated to the in-game obstacles and adversaries they encounter. Adversarial environments in typical video games tend to either be fixed or utilize randomized policies that can be exploited by experienced players. Even machine learning techniques that are overfit to challenge the average player do not necessarily account for players of a particular type (such as those who have quick reaction times), or players who employ particularly exploitative strategies. To combat these limitations, this research aims to build an adversary that understands players' instinctive reactions to game states and uses that understanding to dynamically challenge the player as the game is played. This project attempts to characterize different player types and discover adversarial strategies that directly challenge and adapt to their play style. This is accomplished through a 2D bullet-hell, platformer style video game that employs techniques from reinforcement learning (a variant of SARSA) and causal inference (empirical counterfactuals). This project is being developed by stellar students Veda Ashok and Kira Toal and represents another interdisciplinary pursuit combining animation and computer science.

Harmony: A Self-Harm Prevention App

Harmony represents an interdisciplinary effort between Computer Science and Psychology to provide users with relief and distraction from habits of self-harm. It provides a number of gamified, but research-supported, behavioral-cognitive approaches to allow users to consult resources that will allow the urge to pass.

Cognitive Agents in the Iterated Prisoner's Dilemma

The Prisoner's Dilemma (PD) is a simple game that serves as the basis for research on social dilemmas in a variety of fields. It is a two player, general sum game often parameterized by a payoff tuple (R, P, S, T). Each player is given the option of one of two actions: cooperation (C) or defection (D). The payoffs of the actions are in the figure below. The game is modeled such that T > R > P > S and 2R > T + S. From a game theoretic perspective, the best option for each player is always to defect, which results in the socially deficient Nash Equilibrium. A traditional PD is a one round game, but an Iterated Prisoner's Dilemma (IPD) is a sequence of PDs between the same two players. It is often studied to understand the effects of previous actions and the emergence of mutual cooperation. This project examines deployment of new Causal, Counterfactually regret minimizing agents in the IPD to examine their efficacy in an ecology against repeated opponents.

Applications of Superposition in POMDPs

Markov Decision Processes (MDPs) are discrete, mathematical formu- lations used to simulate generalized sequential decision making. An agent acting within an MDP must optimize expected future reward by making a decision on which action to take given the current state. Partially Ob- servable MDPs (POMDPs) are MDPs in which the state is not known by the agent, and instead the agent must act upon observations given by the environment at each time step. This work presents a novel solution that is more sample-efficient than traditional methods for fully-specified POMDPs, viz. when transition probabilities between states as well as observations are given to the agent. Currently, traditional approaches for solving POMDPs require iterative learning processes that converge slowly when not exploiting opportunities for linear parallelism. Consequently, we present a closed-form solution that is derived algebraically from tradi- tional iterative update rules. Solving this closed-form solution yields an accelerated learning rate that enables a jump start unemployed by tradi- tional iterative methods. Simulation results support the efficacy of this method on traditional POMDPs. Additionally, applied and theoretical impliciations of this method are discussed.

Bias Clustering for Online Political Articles

This project examines trends of biased language in articles on similar topics across a variety of traditionally polarized news outlets, providing a metric of valence.

DunGen: Causal Inference in Procedural Dungeon Generation

DunGen provides a Dungeons and Dragons room-layout generator based on a number of preset room types and features, that can then later be tweaked by users with tools that span associational, causal, and counterfactual relations between features.

Fusions of Deep and Causal Reasoning for Autonomous Poker Players

This project examines the fusion of deep learning networks in detection of facial cue recognition as a component of larger structural causal models, for the purpose of reading players' tells to update belief states in an autonomous Texas Hold'em Poker player.