【海韵讲座】2024年第43期- Accelerating Large Mixture-of-Experts Models via Pipelining and Scheduling
报告题目：Accelerating Large Mixture-of-Experts Models via Pipelining and Scheduling
主讲人： 褚晓文教授，香港科技大学（广州），数据科学与分析学域主任，国家海外高层次人才
报告时间：2024年09月13日（星期五）15:00-16:30
报告地点：厦门大学翔安校区bwin必赢1号楼108会议室
报告摘要：
In recent years, large-scale deep neural network models have been able to scale to trillions of parameters with the sparsely activated mixture-of-experts (MoE) approach. This significantly enhances model quality while necessitating only a sub-linear increase in training costs. However, the dynamic nature of data routing and the high communication costs associated with training MoE models can lead to low scaling efficiency within GPU cluster training systems. This talk will introduce two of our latest advancements aimed at enhancing the training efficiency of MoE-based large language models (LLMs): PipeMoE and ScheMoE. PipeMoE adaptively pipelines communications and computations within MoE layers, aiming to minimize communication time by leveraging tensor partitioning with an optimal pipeline degree. On the other hand, ScheMoE offers a versatile scheduling framework that enables the optimal scheduling of communication and computation tasks.
报告人简介： 
Prof. Xiaowen Chu received the Bachelor degree in Computer Science from Tsinghua University, China, in 1999, and the Ph.D. degree in Computer Science from The Hong Kong University of Science and Technology in 2003. Currently, he is a Full Professor and Head of Data Science and Analytics Thrust at The Hong Kong University of Science and Technology (Guangzhou). His research interests include GPU Computing, Distributed Machine Learning, Cloud Computing, and Wireless Networks. He has won six Best Paper Awards at different international conferences, including IEEE INFOCOM 2021. He has published over 250 research articles at international journals and conference proceedings. He has served as an associate editor or guest editor of IEEE Transactions on Cloud Computing, IEEE Transactions on Network Science and Engineering, IEEE Transactions on Big Data, IEEE IoT Journal, IEEE Network, IEEE Transactions on Industrial Informatics, etc.
邀请人：计算机科学与技术系 向乔教授