The Need

In high-stakes industries, the integration of humans and advanced automation systems demands evaluation methods that reliably predict performance under varying challenges. Current testing methods often focus on individual components, failing to assess how human-machine teams operate as a unit, particularly under conditions of increased difficulty. This gap hinders the ability to anticipate system brittleness—when performance collapses in critical scenarios—leaving industries vulnerable to costly and catastrophic failures.

The Technology

Joint Activity Testing (JAT) is an innovative testing and evaluation (T&E) methodology designed to assess the performance of human-machine teams (HMTs) across a spectrum of challenges. By analyzing joint performance through structured experimental scenarios, JAT identifies vulnerabilities and quantifies system brittleness. The method uses performance and challenge metrics to create Joint Performance Graphs (JPGs) that illuminate how systems balance efficiency and accuracy as difficulty increases. This repeatable approach provides a nuanced, multidimensional view of system resilience, making it uniquely suited for evaluating complex, adaptive technologies such as AI-driven automation.

Commercial Applications

• Healthcare: Evaluating human-machine collaborations in patient monitoring and diagnostic systems.
• Defense: Enhancing decision-making in AI-supported tactical operations.
• Aviation: Assessing pilot-automation performance in advanced cockpit systems.
• Manufacturing: Optimizing robotic systems in high-stakes assembly environments.
• Autonomous Systems: Testing human oversight in self-driving vehicles and drones.

Benefits/Advantages

• System-Level Insights: Focuses on joint performance, providing mission-relevant evaluations rather than isolated component analysis.
• Multidimensional Analysis: Examines efficiency, accuracy, and trade-offs, offering a comprehensive understanding of system behavior.
• Extrapolation Capabilities: Enables predictions beyond test conditions, identifying brittleness in rare but critical scenarios.
• Generalizable Methodology: Applicable across diverse industries, especially those deploying AI and advanced automation.
• Enhanced Resilience: Identifies areas for improvement, helping design systems that gracefully adapt to increasing challenges.