Allen School Ph.D. student Cheng-Yu Hsieh is interested in tackling one of the biggest challenges in today’s large-scale machine learning environment — how to make artificial intelligence development more accessible. Large foundation models trained on massive datasets have revolutionized AI, however, these scaling efforts are often out of reach for many except for well-resourced companies, Hsieh explained. His research focuses on making both data and model scaling more efficient and affordable to help democratize AI development. 

“I develop effective data curation techniques for training large foundation models, as well as efficient methods to deploy and adapt these models to various downstream applications. My research spans key stages of today’s artificial intelligence development pipeline,” said Hsieh, who works with professor Ranjay Krishna in the UW RAIVN Lab and affiliate professor Alex Ratner in the Data Science Lab. 

For his contributions, Hsieh was awarded a 2024 Google Ph.D. Fellowship in machine intelligence. The fellowship recognizes outstanding graduate students from around the world representing the next generation of leaders with the potential to influence the future of technology through their research in computer science and related fields. 

“This fellowship will support my research on making large-scale AI systems more efficient, accessible and adaptable. I’m excited to continue exploring how we can make AI technology more sustainable and inclusive,” Hsieh said.

Hsieh designs methods to help mitigate the high costs and other complexities associated with large-scale AI model development. For example, one of the major bottlenecks in today’s machine learning pipeline is manually labeling or curating large datasets, which can be labor intensive. Hsieh and his collaborators introduced Nemo, an end-to-end interactive system that guides users through creating informative datasets using weak supervision techniques in order to lower the barrier for building capable AI models in low-resource settings. Nemo was able to improve overall workflow efficiency by 20% on average compared to other weak supervision approaches, Hsieh found.

Some of his research projects have been put into practice and have already made a real-world impact. As part of a collaboration between UW and Google, Hsieh helped develop the distilling step-by-step method that enables users to train smaller task-specific models using less training data compared to other standard fine-tuning or distillation approaches. With this method, a smaller 770M parameter T5 model trained with only 80% of the data on a benchmark can outperform a much larger 540B PaLM model. The team launched the project on Google Vertex AI, the company’s generative AI development platform, and Google highlighted the research at the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023). Hsieh’s research into model adaptation was also integrated into the Vertex platform, allowing users to adapt models to new applications without needing explicit training data.

“Cheng-Yu is a self-sufficient, diligent, effective and productive researcher,” said Krishna. “His recent papers propose solutions to a wide range of pertinent problems in natural language processing, efficient machine learning and retrieval augmented generation, and I have no doubt that he will continue to produce impactful research.”

As part of his goal to make data and model scaling more efficient and affordable, in future research, Hsieh is interested in developing new approaches for querying powerful, but oftentimes expensive, generative AI models to help create informative and controllable datasets for model training and alignment. 

“This fellowship is both a recognition of the work I’ve done and an incredible encouragement to continue pushing my research direction in AI. I am very thankful to my advisors, mentors and collaborators who have supported me along the way,” Hsieh said. “I am excited to continue pursuing research with real-world impact in this fast-paced era of AI development.“

Read more about the 2024 Google Ph.D. Fellowship.