在人工智慧應用於醫學領域的時代，乳房X光影像分析需要一種能夠同時理解視覺資訊和臨床背景的模型，以提高乳癌診斷的準確性。然而，傳統的純視覺方法往往難以捕捉影像與臨床描述之間的語義關係，導致分類表現不佳。本研究旨在開發一種提示引導的多視角視覺語言模型，透過整合來自不同圖像視角的資訊和描述性文字引導，來改進多模態表示。本研究使用的資料集為VinDr-Mammo，包含5000例檢查，涵蓋四個主要視角（LCC、LMLO、RCC、RMLO）以及腫塊、鈣化和BI-RADS等臨床資訊。研究方法包括一個預訓練階段，該階段使用對比學習方法來對齊圖像和文字嵌入；以及一個微調階段，該階段使用多視角提示引導方案。文字提示由大型語言模型（LLM）構建，用於產生結構化且與臨床相關的視覺描述。本研究在零樣本和線性探測情境下，採用曲線下面積（AUC）、平衡準確率（B-ACC）和計算時間指標對模型進行了評估。結果表明，與基線模型相比，所提出的模型能夠持續提升分類效能。在零樣本場景下，模型在鈣化類別中實現了高達0.85的AUC和0.78的B-ACC。在線性偵測場景下，效能進一步提升至0.91的AUC和0.85的B-ACC。此外，該模型保持了較高的效率，計算時間與基準模型基本相當。總體而言，提示資訊和多視圖的整合有效提升了模型的判別能力，但對於具有複雜視覺特徵（例如腫塊）的病例，仍存在一些挑戰。本研究強調了多模態方法對於提升人工智慧診斷系統在實際應用中的準確性和可靠性的重要性。;In the era of artificial intelligence development in the medical field， mammogram image analysis requires a model capable of simultaneously understanding visual information and clinical context to improve the accuracy of breast cancer diagnosis. However， conventional visual-only approaches are often limited in capturing semantic relationships between images and clinical descriptions， resulting in suboptimal classification performance. This study aims to develop a Prompt-Guided Multi-View Vision Language Model to improve multimodal representation by integrating information from various image viewpoints and descriptive text guidance. The dataset used is VinDr-Mammo， which consists of 5，000 examinations with four main views (LCC， LMLO， RCC， RMLO) and clinical informations such as masses， calcifications， and BI-RADS. The research method includes a pre-training stage using the contrastive learning approach to align image and text embeddings， and a fine-tuning stage using a multi-view prompt-guided scheme. Textual prompts are built using a Large Language Model (LLM) to generate structured and clinically relevant visual descriptions. Evaluation was conducted using Area Under the Curve (AUC)， Balanced Accuracy (B-ACC)， and computation time metrics in zero-shot and linear probe scenarios. The results showed that the proposed model was able to consistently improve the classification performance compared to the baseline. In the zero-shot scenario， the model achieved an AUC of up to 0.85 and a B-ACC of 0.78 in the calcification category. In the linear probe scenario， the performance increased to an AUC of 0.91 and a B-ACC of 0.85. Furthermore， the model remained efficient with a computation time relatively equivalent to the baseline. Overall， the integration of prompts and multi-views proved effective in improving the model′s discrimination ability， although challenges remain in cases with complex visual characteristics such as masses. This study emphasizes the importance of a multimodal approach to improve the accuracy and reliability of AI-based diagnostic systems in the real world.
