An interdisciplinary University of Wisconsin-Madison team placed among the top performers in an international competition testing how machine learning models can analyze endoscopic surgical videos.
The team consists of Ruffin Bryant and Noah Kalthoff, both seniors studying biomedical engineering, and Aadya Ganjigunta, a junior studying computer science and data science. The team competed in the MICCAI EndoVis SurgVU Challenge, hosted by the Medical Image Computing and Computer Assisted Intervention Society. The challenge focused on developing AI systems that could analyze short endoscopic surgery videos, including identifying surgical tools and what was happening in each clip.
The EndoVis competition drew 78 entries from 12 countries. The UW-Madison team placed third in its category with a new algorithm developed using previous experience developing systems such as OpScribe-AI to respond to questions about what was happening in video clips. OpScribe is a platform Bryant and Kalthoff began developing in fall 2025 and turned into a business venture a few months later. The student team also won recognition for the best report during the competition, and has been invited to co-author a paper reporting their findings with the challenge organizers.
In this Q&A, team members discuss preparations for the competition, what they learned in the process, and what they hope to do next with OpScribe-AI.
Q: What prompted you to enter the EndoVis MICCAI SurgVU Challenge?
A: We are co-founders of OpScribe-AI, an agentic AI platform for automated surgical documentation, planning and decision support. We entered this competition to benchmark our technology against other surgical AI groups around the world and to showcase what we’ve built so far. The challenge was also sponsored by Intuitive, the leader in surgical robotics, so it was great to get our work in front of their team.
In the EndoVis challenge, our model was fed 30-second clips of endoscopic surgery and then asked questions about what it saw. That’s very close to what our main system does, so it was a natural fit and a way to test our approach on the international stage. Because the competition required the model to run fast with limited computing resources, we ended up developing a new model built specifically to answer questions about short clips.
Q: How did you prepare? What goes into developing a model for this sort of competition?
A: The main objective in this challenge was to answer a question about a short surgical video. We built a custom model for this based on OpScribe, but it isn’t the same system. We had to shrink our model, because our allotted computing power was a lot smaller than what we typically use for OpScribe. We also experimented more with tool use, meaning the model can call on other programs to help it answer questions.
Because of those limits on model size and runtime, we chose a hybrid approach. We combined traditional natural language processing and machine learning techniques for understanding the question with more modern computer vision models and vision-language models (VLMs), which are AI models that can read both images and text. We used a decision tree for intent parsing, which is essentially figuring out what a question is asking, and then followed different strategies to answer it. In some cases, we were able to choose the answer from a pre-populated answer bank. In others, we used information from vision models trained to identify instruments and surgical tasks. And in other cases, we used a fine-tuned VLM to answer the question. This hybrid of older and newer technologies helped us succeed.
The challenge had a one-month preliminary phase where we could test new versions of our model up to 10 times. That let us take a piece-by-piece approach to building this smaller system. We slowly tweaked parts of the model and figured out what made it better and what made it worse. OpScribe won that preliminary phase with 93% accuracy.
Q: What was your biggest lesson from working on this project?
A: From a technical standpoint, our main lesson was how to build a workflow out of decision trees, vision models and language models to perform a complex task. Debugging a system like that isn’t easy, so we developed our own internal benchmark to test our method as we went.
From an experiential standpoint, the biggest lesson came from going to the conference itself. Being able to speak with researchers from the Netherlands, Japan, Taiwan, Mexico, China and the United States—all leaders in this exact space—was fascinating. We got to see what they were working on and how we could bring their ideas into our own system.
It also gave us a lot of confidence. We got to see how our model stacks up against state-of-the-art work from large research institutions all over the world. Knowing we’re competitive with other surgical AI groups, and that we were able to win parts of this challenge, tells us we’re on the right track. We came away with a lot of new tools, technologies and ideas we can add to our system and test further.
Q: When it came to the competition, what set your model apart from the others?
A: We let the question guide the answer. There were two inputs to the challenge. The first was the question. For example: “What type of forceps are being used here?” or “What part of the surgery is this?” or “Where is this tissue being dissected?” The second was the 30-second clip that went with that question.
A typical AI model will look at the video and give you an answer. We almost inverted that process. First, as we mentioned, our decision tree determined what the question was actually asking. Then the system used tools to pick the right strategy for that type of question. Sometimes we pulled the answer from our answer bank. Sometimes we used our vision models to identify the instruments and tasks in the clip. For harder questions, we fed everything into our fine-tuned VLM, so it already had most of the evidence it needed to answer.
That’s what made us successful. Because every answer followed a clear path, we could see exactly where things went wrong and fix them using our internal benchmark. We were around 66% accurate across 102 videos in the final challenge, placing third out of 78 entries from 12 countries. We also won Best Methodology Report, which reflected how clearly every answer could be traced back to what the model actually saw in the video.
Q: What are your next steps, and do you plan to continue developing the model?
A: OpScribe is being developed in both a research and a commercial setting. On the research side, we’re taking what we learned at MICCAI and building it into OpScribe v2.0, our new system. We are currently working on a manuscript where we systematically evaluate OpScribe on both endoscopic procedures and open surgeries. We’re also working on integrating electronic health records and additional tool-use capabilities in our platform. Moving forward, we’ll bring the technologies that worked well in this challenge into version 3.0.
On the commercial side, the Greater Madison Chamber of Commerce recently selected us as one of the top five high growth potential companies in Wisconsin, and we have been invited to pitch to more than 35 investors in San Francisco. We hope to work on OpScribe-AI full-time after graduating next year, and we’re actively raising seed funding.
All in all, this was a fantastic experience for us. It was really a privilege to go to France, compete on a world stage and do extremely well.
Featured image caption: Ruffin Bryant, Noah Kalthoff and Aadya Ganjigunta earned third place in an international competition that tested how AI models fare when analyzing endoscopic surgical videos.