Model Reuse: Concepts, Algorithms, and Applications

AAAI 2024 Tutorial, Feb. 20, 2024, 2:00 pm - 6:00 pm (PST), Vancouver, Canada

Reusing Pre-Trained Models (PTMs) instead of training from scratch has become a new paradigm in machine learning. Despite its widespread use in AI applications, model reuse is currently explored from various specific aspects, such as parameter-efficient fine-tuning. Our tutorial offers a concise overview of model reuse, including core concepts and recent advancements these model-centric techniques. In detail, we categorize model reuse methods from data-level, model-level, and algorithm-level, and discuss how to reuse heterogeneous PTMs as well as multiple PTMs. In addition, a review of one crucial step in model reuse — the selection of PTMs from a model zoo containing heterogeneous PTMs—is also covered in this tutorial, where the fitness-based, vectorization-based, and specification-based approaches are covered.

Keywords: Machine Learning, Deep Learning, Pre-Trained Model Reuse, Hypothesis Transfer, Data-Free, Pre-Trained Model Selection, Transferability Estimation.

Organizer: Han-Jia Ye (yehj@lamda.nju.edu.cn) , Nanjing University

Yao-Xiang Ding (dingyx.gm@gmail.com) , Zhejiang University

Outline:

² Introduction

² Types of Model Reuse

Ø Why should we care about model reuse?

Ø The taxonomy.

² Reuse Pre-Trained Models

Ø Reuse PTMs from data-level, model-level, and algorithm level.

Ø Reuse heterogeneous models.

Ø Reuse multiple PTMs.

Ø Other types of model reuse.

² Selecting PTMs

Ø PTM selection and transferability measure

Ø Fitness-based, Vectorization-based, and specification-based approaches.

² Applications

² Conclusion

Slides: [Part1] [Part2]

If you find any compelling work in the realm of model reuse that we might have overlooked, please do not hesitate to inform us.