Understanding Alignment in Multimodal LLMs: A Comprehensive Study

來源原標題

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

原始連結

https://machinelearning.apple.com/research/alignment-multimodal-llms

摘要原文

research area Computer Vision , research area Speech and Natural Language Processing content type paper published August 2026 Understanding Alignment in Multimodal LLMs: A Comprehensive Study Authors Elmira Amirloo*, Jean-Philippe Fauconnier*, Christoph Roesmann*, Christian Kerl†, Rinu Boney†, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch View publication Copy Bibtex Preference alignment has become a crucial component in enhancing the performance of Large Language Models (LLMs), yet its impact in Multimodal Large Language Models (MLLMs) remains comparatively underexplored. Similar to language models, MLLMs for image understanding tasks encounter challenges like hallucination. In MLLMs, hallucination can occur not only by stating incorrect facts but also by producing responses that are inconsistent with the image content. A primary objective of alignment for MLLMs is to encourage these models to align responses more closely with image information. Recently, multiple works have introduced preference datasets for MLLMs and examined different alignment methods, including Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO). However, due to variations in datasets, base model types, and alignment methods, it remains unclear which specific elements contribute most significantly to the reported improvements in these works. In this paper, we independently analyze each aspect of preference alignment in MLLMs. We start by categorizing the alignment algorithms into two groups, offline (such as DPO), and online (such as online-DPO), and show that combining offline and online methods can improve the performance of the model in certain scenarios. We review a variety of published multimodal preference datasets and discuss how the details of their construction impact model performance. Based on these insights, we introduce a novel way of creating multimodal preference data called Bias-Driven Hallucination Sampling (BDHS) that needs neither additional annotation nor external models, and show that it can achieve competitive performance to previously published alignment work for multimodal models across a range of benchmarks. * Authors contributed equally as first authors. † Authors contributed equally. Related readings and updates. Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment September 22, 2025 research area Computer Vision conference NeurIPS Despite Contrastive Language-Image Pretraining (CLIP)‘s remarkable capability to retrieve content across modalities, a substantial modality gap persists in its feature space. Intriguingly, we discover that off-the-shelf MLLMs (Multimodal Large Language Models) demonstrate powerful inherent modality alignment properties. While recent MLLM-based retrievers with unified architectures partially mitigate this gap, their reliance on coarse modality… Read more Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison October 23, 2024 research area Data Science and Annotation , research area Methods and Algorithms conference NeurIPS The goal of aligning language models to human preferences requires data that reveal these preferences. Ideally, time and money can be spent carefully collecting and tailoring bespoke preference data to each downstream application. However, in practice, a select few publicly available preference datasets are often used to train reward models for reinforcement learning from human feedback (RLHF). While new preference datasets are being introduced… Read more Discover opportunities in Machine Learning. Our research in machine learning breaks new ground every day. Work with us