With DARPA Award of $5.9 Million, Corso and Team’s Artificial Intelligence Program Will Learn to Guide Humans Through Complex Tasks
This major multi-PI, multi-institution grant through Stevens Institute for Artificial Intelligence will allow the team to develop “MILLY,” an AI program that continuously learns while helping human users
Suppose a mechanic who is trained on traditional combustion engines is asked to repair a Tesla. Obviously, some of their experience will be relevant to this new task. However, as the whole power train is different in an electric vehicle as compared to a gasoline car, the mechanic will need guidance in this new project domain.
Now imagine what the mechanic could do with a “thinking cap” in the form of an augmented reality device — which would tell them if they did something wrong (“You put that bolt in the wrong place”), or allow the mechanic to ask questions (“Where do I connect this power cord?”).
Principal investigator Jason Corso and co-investigator Enrique Dunn in the Department of Computer Science, along with a team of co-investigators — Jeffrey Siskind (Purdue University), Chenliang Xu (University of Rochester), and Joyce Chai (University of Michigan) — are creating a program nicknamed “MILLY,” which can guide a person through tasks such as that which faced the mechanic in our example.
Corso, Dunn, and team have been awarded a DARPA grant of $5,913,977 to pursue what Corso calls “a culmination of more than a decade of work from my own research group and that of the co-investigators.” Their project “MILLY—Multi-Directional Loosely-Linked Archetype Models for Perceptually-Enabled Task Guidance” is a multi-PI, multi-institution grant through Stevens Institute for Artificial Intelligence (SIAI).
Through MILLY, the team seeks to develop foundational artificial intelligence (AI) capabilities that can learn procedures from visual and lingual examples and use the learned models to interactively guide nonexpert humans through complex tasks. MILLY will be able to continuously learn through a variety of sources, be they YouTube videos, instruction manuals, checklists, and so forth, and practice “active learning” in order to adapt on the fly. Then, for any given task, MILLY must be able to model the task, understand the task, map the task, determine what language will be relevant, communicate with the human user, and then guide the human user.
Of this process, Dunn said, “Task guidance is a complex multi-modal problem. The goal of automating the entirety of this process is an ambitious challenge that expands the frontiers of current research in AI and augmented reality, which our entire interdisciplinary team is excited to undertake.”
In short, Corso said, MILLY will be able to “understand the task, understand the human, and understand what’s missing to fill in that gap.”
Learn more about Stevens Institute for Artificial Intelligence.
Learn more about computer science at Stevens: