Fan Wenshu @ LAMDA, NJU-CS

Email: fanws [at] lamda.nju.edu.cn
      

• Wen-Shu Fan, Su Lu, Shangyu Xing, Xin-Chun Li, De-Chuan Zhan. Maximizing the Effectiveness of Larger BERT Models for Compression. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL'25), Vienna, Austria, 2025.

We find that student BERT models taught by larger BERT teachers exhibit limited linear differences among their intermediate layers, leading to suboptimal distillation performance. Building on this insight, we enhance the distillation method, thereby improving the effectiveness of BERT compression.

• Wen-Shu Fan, Su Lu, Xin-Chun Li, De-Chuan Zhan, Le Gan. Revisit the Essence of Distilling Knowledge through Calibration. In: Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vienna, Austria, 2024. [Paper]

A counter-intuitive phenomenon known as capacity mismatch in knowledge distillation has been identified, wherein KD performance may not be good when a better teacher instructs the student. In this paper, we propose a unifying analytical framework to pinpoint the core of capacity mismatch based on calibration.

• Xin-Chun Li , Wen-Shu Fan, Shaoming Song, Yinchuan Li, Bingshuai Li, Yunfeng Shao, De-Chuan Zhan. Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again. In: Advances in Neural Information Processing Systems 35 (NeurIPS'22), New Orleans, Louisiana, USA, 2022. [Paper]

We find traditional temperature scaling limits the efficacy of class discriminability and hinders large teachers from teaching well. We propose a method which enriches class discriminability by separately applying a higher/lower temperature to the correct/wrong class.

• Wen-Shu Fan, Xin-Chun Li, De-Chuan Zhan. Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch. Frontiers of Computer Science (FCS). [Paper]

We find that variance of non-target classes correlates with distillation performance, and that different teacher models exhibit relative class affinity. We propose a simple method that more effectively amplifies wrong-class variance, improving the distillation performance of large teachers.