When the Machine Brings the Human Into the Loop
Technological achievements have led to a myriad of advancements in cancer treatment, but access to those therapies remains a particularly human issue. Deciding which patients meet the criteria for groundbreaking clinical trials can be a painstaking process requiring cross-checking data across dozens of categories.
Kenneth Kern, MD, MPH, MS is in the early stages of inserting those same technological leaps into the selection process.
Dr. Kern is a former oncology surgeon and cancer drug development professional with a large pharmaceutical company, with more than 40 years of experience in treating patients and advancing cancer treatments. He currently works as independent oncology drug development consultant who helps biotechnology companies and venture capital firms identify the inherent risks of drug development in its early stages and reduce these risks, which will improve development decisions and outcomes.
“My role now involves a lot of careful scientific analysis of several areas in cancer drug development, one of which we call translational science, which is going from the laboratory to the bedside,” he explained. “It’s analyzing the data to see if it's going to support the idea of using a drug in humans and then be able to give that evidence to the FDA that the drug is going to be safe and potentially effective in the first human trials used to test it. The whole point of clinical trials in humans is to see if a drug effectively controls cancer, but you're not going to be able to put a drug into a human being in a Phase 1, first-in-human study unless it's proven to be reasonably safe. In addition, there has to be some evidence that the drug is going to control or stop the growth of the cancer based on the pathways that the experimental drug is proposed to inhibit."
It’s no surprise the technology Dr. Kern chose to utilize is artificial intelligence, also known as machine learning, which has become a part of nearly everyone’s day-to-day life in some way, even if they don’t know it. Calling on knowledge he learned in more than 30 courses, which have earned him three advanced certificates from some of the top institutions in the country, including the Stevens School of Business Management of AI program, he developed a semi-automated, agentic AI patient enrollment tool he named "TrialTriage".
The agentic AI platform addresses a chokepoint in clinical trials, which is determining if patients are eligible for specific trials of cancer treatments. The current process involves careful checking by the research staff eligibility criteria present in the medical record, such as laboratory values, x-rays and clinical examination results. Currently this time-consuming, costly process is done manually and can result in ambiguous cases if information is incomplete or missing. Usually, these specific cases are put on a list to be clarified days or weeks later. Depending on their diagnosis, some patients cannot wait to be cleared and may remove their name to move on to another trial. In some cases, they are just dropped as a potential match.
Dr. Kern's proof-of-concept, semi-automated workflow introduces an iterative, agentic AI-to-doctor feedback loop. First the agentic AI classifies patients entered into the workflow by an email listing as to whether they are classified as eligible, ineligible or are ambiguous and need more information to classify. When the agentic AI classifies the patient as an ambiguous case, it immediately emails the physician or research staff, notifying them of what specific data is missing and requesting results be sent as a response to the email. The agent then incorporates the data into the workflow and repeats the classification procedure. If there is no response, TrialTriage follows up after 24 hours with a second request for more information. The agent suggests by email that eligibility completion be done by manual review after three unanswered attempts.
“I think the greatest contribution of the project is the repeated feedback loop that is sent immediately from the agentic AI, asking the doctor or research for more information,” Dr. Kern said. “This was a proof-of-concept study, to test the concept using ‘synthetic’ or artificial information, not real patient data. However, the data was modeled after patients and did have all the characteristics of real cases. This process of eligibility screening was automated, with me in the loop supervising it and checking for errors. This is called a semi-autonomous agent with a human-in-the-loop. I sent an email to an agentic workflow, with a listing of patient data, which automatically put the data into in a language model.”
“The agent gave the output of whether a patient was eligible or not. Within a matter of seconds, the agentic AI classified them as eligible, not eligible or 'I'm not sure,’ and put their classification as ambiguous,” Dr. Kern continued. “As soon as it said, ‘I'm not sure,’ it sent an email to the doctor automatically without having to go through a human being. The email listed the missing information and asked the physician to supply it. All the doctor had to do was reply to the email with the missing information, and TrialTriage would then re-classify the patient."
Applying seven criteria against 90 synthetic patient cases (artificially created cases modeled after real cases) which were generated independently using three different AI models, TrialTriage's classifications were correct 100% of the time, processing the cases in an average time of 2.3 minutes. Five independent human reviewers, all expert oncologists, evaluated a 30-case subset of that data, achieving a mean accuracy of 96.7% and taking a mean time of 9.8 minutes to classify the cases. The individual reviewers were occasionally tripped up calculating the time of previous drug exposures and other human biases or assumptions the agentic AI didn't share. The results of his experiment were recently published in Journal of Medical Information Research (JMIR) Formative Research under the title “TrialTriage, a Semiautonomous Prescreening Workflow for Resolving Ambiguity in Phase I Oncology Trial Eligibility: Development and Proof-of-Concept Study Using Synthetic Cases.”*
“When you can get a quick answer to clarify ambiguous cases of eligibility, you're going to get patients into clinical trials more quickly,” he explained. “You're going to identify more patients. You're not going to leave so many patients behind, and it's less expensive to the people that do the trials. Sometimes potential patients get pushed forward into formal enrollment even though the research staff doesn't have a complete answer. These patients just get turned back at the final step of screening just after they sign a consent to do more detailed evaluations for the stage and more scientific details about the molecular pathways stimulating their cancer to grow. The company and the cancer research organization spend a lot of money trying to see if the patient is safe to enter a cancer trial with a new drug, but then they turn out not to be. The eligibility question would be answered far more quickly using rapid, iterative communication with the research staff. The patient could go somewhere else for experimental therapy instead of waiting and waiting for a decision and not passing the testing at the final step.”
“This human-to-AI communication loop is not present in a lot of big screening tools, looking at hundreds or thousands of patients,” Dr. Kern explained. “In fact, that's what makes this iterative communication loop so valuable. When these big screening tools run into an ambiguity, they just give it to a human to resolve, which increases the burden on manual methods to screen patients, delays screening, raises cost and put the patient in a longer time before they enter a potentially effective clinical trial. The big screening workflows do not have the immediate, iterative communication loop with the research staff that TrialTriage does."
The paper was completed after he earned his Stevens certificate, but the idea was hatched while working on the program’s capstone project. Dr. Kern credits the hands-on teaching approach in spurring his creativity and requiring him to think through the work from multiple angles.
“During the classes leading up to the final capstone project is where I first developed the concept of a pilot project for this pre-screening workflow,” he said. “Then the capstone project really helped me understand how to create an agentic AI that was semi-autonomous, which means it had human oversight, but it could review on its own the medical records of patients who are going to see if they are eligible to enter a cancer trial. I had to go a little bit beyond the Stevens’ training to complete TrialTriage, but the point is, without Stevens Institute training in AI I would have never even conceived of this project. What really helped was how certain professors designed their classes to help students build a working AI model. In the first class, the assignment was coming up with concept of an idea. The next class required coming up with a little more sophisticated concept, and then finally, in the capstone, they wanted a ‘minimally viable’ working AI model of the project. After completing the training, I built it out in more detail on my own, but the kernel of the idea, and its earliest model, came from being asked to come up with a project.”
That framework of building an idea gradually, class by class is what Dr. Kern points to when explaining why Stevens stood out among the many courses he's taken. He'd already studied AI extensively at institutions like MIT, Stanford and Harvard Business School before enrolling, but says Stevens' hands-on structure and practical learning through actually building an AI workflow added a critical dimension to his understanding of both generative and agentic AI that was truly eye opening.
"All the classes I’ve taken at phenomenal schools have been outstanding, providing incredible background and theory. But, if it hadn't been for the hands-on approach Stevens takes, then I wouldn't know as much as I know now about AI," he said.
Dr. Kern was particularly impressed with the Stevens professors’ ability to work through problems in real time. Rather than sitting at a podium and lecturing without wanting to be questioned, his professors were willing to admit when they didn’t have all the answers, and then, even more importantly, work through the answer in front of the whole class, using a variety of large language models.
"Watching Professor (Alkis) Vazacopoulos, when he was asked a question that he wanted more information about, ask the question of one or more large language models in real time, right in front of us, was one of the most important things I learned," he said. "These foundational models can serve as a tutor, mentor and personal educator when it comes to highly technical questions. We saw that happen in real-time and learned a great lesson from it. Of course, you should never trust everything a language model says without verifying it, but I think everyone in the classes I took were impressed with the fact that the professors at Stevens were humble enough to use large language models to add detail to their answers to tough questions, instead of hemming and hawing and answering with in an ambiguous way."
Dr. Kern is carrying the lesson of Stevens' professors in using large language models to improve the answers to technical and complex questions into his work with TrialTriage. Publishing the initial workflow and its machine vs. human findings was an important first step, but he stresses his work to date is a proof of concept and not a finished product.
"You actually have to scale it up to a far larger population of real patients with real medical records, and then it's going to face some issues,” he said. “What happens if the description of the patient is a page long? What happens if the patient's chart is 200 pages long, stretching over many years? How is the agent and its foundational model going to accurately get through all that, and are you going to do it all while maintaining HIPAA compliance for patient confidentiality? A human must stay in the loop, but where will they be placed in the workflow? How will guardrails be placed around the autonomous aspects of the agentic workflow so that the integrity of the process remains intact? All of this will take money and a concerted effort. I know this is being worked on now by several companies, and I also know that I've never seen an eligibility screening model that had an iterative communication loop with the research staff by immediate email. While they construct these newer versions of AI for cancer trial screening, I hope they will use TrialTriage as a model because that would be a big step forward for cancer patients hoping to get into clinical trials.”
Just like his work, Dr. Kern’s time as a Stevens AI student may not be finished.
"I took four classes, and at the end of that fourth class, everybody asked the same question, ‘Well, what's next at Stevens Institute for us to take related to AI?” he said. “Professor Vazacopoulos told us, 'It’s very likely there will be more to come in the future, keep checking back.' I was glad to hear that. I would have signed up right then for another class if there was one, and when the next AI class is offered in the future, I'm definitely going to sign up for it."
*JMIR Form Res 2026;10:e100779; doi: 10.2196/100779.
