Hai-Long Sun @ LAMDA, NJU-AI

M.Sc. Student, LAMDA Group
School of Artificial Intelligence
National Key Laboratory for Novel Software Technology
Nanjing University, Nanjing 210023, China

Supervisor: Han-Jia Ye(叶翰嘉)

Laboratory: Yifu Building, Xianlin Campus of Nanjing University

Email: sunhl@lamda.nju.edu.cn

• Hai-Long Sun, Da-Wei Zhou, Yang Li, Shiyin Lu, Chao Yi, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye. Parrot: Multilingual Visual Instruction Tuning. arXiv:2406.02539. [Paper] [Code] [Project Page] [Huggingface]

We introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Parrot makes the visual tokens condition on diverse language inputs and uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens. Moreover, considering the current lack of benchmarks for evaluating multilingual capabilities within the field, we collect and make available a Massive Multilingual Multimodal Benchmark which includes 6 languages, 15 categories, and 12,000 questions, named as MMMB.

• Hai-Long Sun, Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan. PILOT: A Pre-Trained Model-Based Continual Learning Toolbox. arXiv:2309.07117, 2023. [Paper] [Code] [中文解读]

We introduce PILOT, a pre-trained model-based continual learning toolbox. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT also fits typical class-incremental learning algorithms (e.g., DER, FOSTER, and MEMO) within the context of pre-trained models to evaluate their effectiveness.

• Hai-Long Sun, Da-Wei Zhou, Hanbin Zhao, Le Gan, De-Chuan Zhan, Han-Jia Ye. MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning. The 39th Annual AAAI Conference on Artificial Intelligence (AAAI'25). [Paper] [Code]

We propose MOdel Surgery (MOS) to rescue the model from forgetting previous knowledge. To mitigate parameter-level forgetting, we present an adapter merging approach to learn task-specific adapters, which aims to bridge the gap between different components while reserve task-specific information. Besides, to address retrieval-level forgetting, we introduce a training-free self-refined adapter retrieval mechanism during inference, which leverages the model’s inherent ability for better adapter retrieval.

• Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, De-Chuan Zhan. Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR'24). [Paper] [Code]

We propose Expandable Subspace Ensemble (EASE) for PTM-based CIL. To enable model updating without conflict, we train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. These adapters span a high-dimensional feature space, enabling joint decision-making across multiple subspaces. As data evolves, the expanding subspaces render the old class classifiers incompatible with new-stage spaces. Correspondingly, we design a semantic-guided prototype complement strategy that synthesizes old classes’ new features without using any old class instance.

• Da-Wei Zhou, Hai-Long Sun, Han-Jia Ye, De-Chuan Zhan. Continual Learning with Pre-Trained Models: A Survey. The 33rd International Joint Conference on Artificial Intelligence (IJCAI'24). [Paper] [Code] [中文解读] [Media]

We present a comprehensive survey of the latest advancements in PTM-based CL. We categorize existing methodologies into three distinct groups, providing a comparative analysis of their similarities, differences, and respective advantages and disadvantages. Additionally, we offer an empirical study contrasting various state-of-the-art methods to highlight concerns regarding fairness in comparisons.