An artificial intelligence system is a highly complex operation with numerous interdependent constituents where one straggler can slow down the entire system.
If the straggler is just a touch behind its counterparts, things usually run fairly smoothly. But one drastically slow worker can cause an entire operation to come grinding to a halt.
It turns out that concepts from information theory and coding can help solve the straggler problem for large-scale machine learning systems, where the computation takes place across multiple processors.
Kangwook Lee, a new assistant professor of electrical and computer engineering, dove into that notion during his PhD research at the University of California, Berkeley. In doing so, he bridged the gap between two traditionally separate communities: information theory and coding researchers and machine learning scientists.
“The existing solutions weren’t good enough,” says Lee. “We realized information theory and coding can develop a much more efficient solution for the straggler problem.”
A paper describing those solutions, which Lee co-authored with UW-Madison ECE assistant professor Dimitris Papailiopoulos (who was, at the time, a postdoctoral scholar at UC Berkeley) went on to become one of the top-five most-accessed publications in IEEE Transactions on Information Theory, the most prestigious information theory journal.
Yet, even though Lee’s work sparked vibrant discussion about applying information theory and coding to machine learning, he’s been frustrated at the slow pace of progress.
“There’s a huge gap between what we want to do and what’s been accomplished,” says Lee. “To be honest, I think it’s time to forget about the small problems and start working on a more general class of machine learning applications.”
That’s one of Lee’s goals for his research at UW-Madison: to extend information theory and coding beyond simple machine learning problems like linear regressions and to apply the concepts to trickier problems like deep learning.
Deep learning algorithms are advanced programs that use multiple layers to transform input data into increasingly abstract representations. Deep learning excels at complicated nonlinear tasks, such as image recognition or other open-ended applications, but underneath the hood, the algorithms are complicated and messy. That’s why information theory and coding have been slow to infiltrate deep learning.
“Information theory and coding are beautiful areas,” says Lee. “But because they place high priority on clean analysis and deep understanding of the theory behind it, the fields didn’t go past simple machine learning problems with linear code.”
Lee plans to change that, but he’ll need an entirely new set of tools that can tackle nonlinear problems. And he won’t need to look very far for the solution.
“It might be possible to use deep learning to design new solutions,” says Lee.
During his postdoctoral work at the Korea Advanced Institute of Science and Technology (KAIST), Lee made sure to immerse himself in deep learning. His time at KAIST also satisfied the mandatory military service requirement for all adult Korean men.
Even though much of Lee’s time in the Korean military was spent in the lab, he still completed the four-week boot camp required for all service members. In fact, Lee left for boot camp almost immediately after receiving his offer letter from UW-Madison.
After three years in Korea, Lee is eager to return to the United States and reunite with friends and colleagues. In addition to working (and hanging out) with Papailiopoulos at UC Berkely, Lee was in the same cohort as Assistant Professor Varun Jog.
“Varun was one of my best friends since the beginning of graduate school,” says Lee. “We started together, studied together and had so much fun together.”
Madison also brings Lee closer to his girlfriend, who’s at the University of Michigan in Ann Arbor. But it is more than just the presence of friendly faces that motivated Lee to join the UW-Madison faculty.
“While I was in Korea, I saw my friends being successful at UW-Madison,” says Lee. “I could see that the College of Engineering is a good research environment and a good place for assistant professors to learn and grow.”