This thesis describes the evaluation of an Intelligent Team Tutoring System (ITTS) designed to teach team and task skills to improve team and individual performance. Previous work has revealed how team communication, shared situational awareness and mental models, and collective efficacy contribute to the success of a team and how these phenomena are molded by the team members’ interactions. However, less research has explored the impacts of an ITTS on these dimensions of teamwork. The present study was conducted on 37 teams of three who took on one of two roles – spotter (two people) or sniper – in a military-style task. The teams completed three trials in their original roles, then one spotter and the sniper switched roles in the fourth trial.

Additionally, individuals either received public or private automated feedback from the ITTS on their performance in the task. Results were mixed. Role experience contributed to the mental model or shared situational awareness of that role as it was defined in training, but not to increased similarity of mental models among teammates. Public feedback positively influenced, although only marginally, the percentage of accurately timed communications and was significantly related to lower overall missed communication actions. Individuals’ performance was also influenced by the frequency of video game play and the amount of team experience, but only for certain actions.

Collective efficacy was impacted by an interaction between experience with cooperative gameplay and frequency of video gaming, where individuals with low gaming frequency but high cooperative gameplay experience had significantly lower collective efficacy than low gamers with no or low co-op experience. Lastly, performance errors were related to individuals’ self-reported use of the feedback, in that ignoring the feedback negatively impacted performance, but selectively following the feedback improved performance. Given previous literature on team dynamics and ITSs, these results are largely unexpected but suggest the feedback style had less impact than was predicted. One team dynamic, collective efficacy, was also shown to be impacted by video game team experience in unanticipated ways, indicating that video game experience and team game experience are indirectly influential to team performance. This research enables the designers of future ITTSs to consider the effects of feedback on coordination and communication tasks more carefully and highlights the importance of the design principle of ensuring a transparent mapping between the feedback and the behavioral triggers that led to it.