{"id":25279,"date":"2025-11-26T10:48:34","date_gmt":"2025-11-26T10:48:34","guid":{"rendered":"https:\/\/pokecon.jp\/job\/?p=25279"},"modified":"2025-11-26T10:48:34","modified_gmt":"2025-11-26T10:48:34","slug":"investigating-fine-tuning-limitations-for-vlms-with-three-case-studies","status":"publish","type":"post","link":"https:\/\/pokecon.jp\/job\/25279\/","title":{"rendered":"Investigating fine-tuning limitations for VLMs with three case studies"},"content":{"rendered":"\n<\/p>\n
\n

Hello, this is Aur\u00e9lie, working as an Artificial Intelligence Engineer at Ridge-i. Today I would like to share some insights about fine-tuning vision<\/a> language models (VLMs)!\u00a0<\/p>\n

Introduction<\/h2>\n

As interest in vision<\/a> language models (VLMs) grows, fine-tuning is increasingly seen as a promising way to adapt models for specific applications. For teams exploring this path, it\u2019s important to take some time to assess whether fine-tuning is the most<\/a> suitable solution and whether it\u2019s likely to lead to meaningful improvement, given the high computational cost and large amounts of data requirements.<\/p>\n

In this blog post, we share some examples of internal experiments where fine-tuning failed to improve<\/a> our model\u2019s capabilities and highlight some key challenges. We consider full supervised fine-tuning (SFT) and Low-Rank Adaptation method (LoRA) which adds small trainable matrices to some specific weights of a frozen model, making fine-tuning efficient with minimal changes.<\/p>\n

Note: All experiments used InternVL2.0 models [1]<\/a> and were conducted in October 2024. Since then, InternVL3.0 has been released. All experiments were done using a single NVIDIA<\/a> A100 GPU<\/a> with 80GB of memory.<\/p>\n

To fine-tune or not to fine-tune<\/h2>\n
\n

Prior to the rise of LLMs, fine-tuning was commonly used for smaller-scale models (100M \u2013 300M parameters). However, with the advent of larger models (> 1B parameters), the question of fine-tuning has become more nuanced.<\/em><\/p>\n<\/blockquote>\n

\n Quote from a Meta blog post: “To fine-tune or not to fine-tune”<\/em> [2]<\/a>
\n <\/figcaption><\/figure>\n<\/p>\n

State-of-the-art models are often released in multiple sizes, with larger models offering the best performance but also requiring significantly more resources and large amounts of high-quality data to fine-tune.<\/p>\n

As a result, the first obstacle encountered by organizations looking to fine-tune a large model is the requirements in terms of computing infrastructure and volume of high-quality data.<\/p>\n

Even if computing resources and access<\/a> to data are not an issue, fine-tuning may not be suitable for all types of tasks.<\/strong> While it is useful for adapting the model\u2019s output style, vocabulary, tone, it is generally not recommended to inject external knowledge<\/strong> [2]<\/a>. This is because LLMs and VLMs are not designed to memorize<\/a> and reliably retrieve highly specific facts (e.g. temperature in a specific place on a specific day) and it is difficult to control or verify that the knowledge gets accurately embedded into the weights during fine-tuning.<\/p>\n

Therefore, it\u2019s important to begin by identifying the source of the performance limitations and to assess whether fine-tuning is the most<\/a> appropriate solution.<\/p>\n

Finally, fine-tuning a large pretrained model with high capabilities always comes with some risk. Some issues like catastrophic forgetting,<\/strong> where the model loses previously acquired general knowledge, can lead to degraded performance and a reduced ability to follow instructions.<\/strong><\/p>\n

In this blog post, we share three quick fine-tuning experiments on different datasets that failed to improve<\/a> performance, and we explore the underlying reasons.<\/p>\n

Which modules to keep frozen in a VLM when fine-tuning<\/h2>\n

A VLM is usually composed of 3 main modules:<\/p>\n