如何在不取得原始碼、亦不依賴私有 API 的前提下擴充既有 Windows 桌面應用程式，是本實驗室近年投入的研究課題；我們將此類非侵入式擴充技術統稱為應用程式擴充（Application Augmentation）。實務上大量Legacy System、商業軟體與第三方應用缺乏官方擴充介面，開發者難以透過傳統方式進行功能整合與自動化控制。為解決此問題，本實驗室先前提出 Venom Programming 技術，以非侵入式方式附著於既有應用程式之上，透過透明覆蓋視窗與狀態追蹤機制實現操作控制與介面擴充。然而，現有 Venom 架構高度依賴 Microsoft UI Automation（UIA）所提供之結構化介面資訊；當宿主程式採用自定義控制項、非標準輔助功能實作時，UIA 往往無法取得有效元件資訊。此外，在遠端桌面、虛擬桌面與視訊串流式應用情境中，介面僅以畫面形式呈現，UIA 更完全失去辨識能力，顯著限制既有方法之適用範圍。 為突破上述限制，本研究提出一套基於電腦視覺之 Venom 應用程式擴充框架，以視覺資訊作為 UIA 結構化資訊之補充識別路徑。研究中建構 CV-Server 與 CV-Inspector 兩項核心模組：CV-Server 整合 UI-DETR 物件偵測模型、OCR 文字辨識與模板比對技術，對螢幕影像進行 UI 元件定位、語意關聯分析與辨識信心評估；CV-Inspector 則提供視覺化註冊與驗證介面，協助開發者建立宿主程式與目標元素之 UI Descriptor，降低電腦視覺導入之開發門檻。本研究進一步於 Venom SDK 中設計雙軌 UI 識別機制，使開發者可依應用場景彈性選擇 UIA 結構式辨識或 CV 影像式辨識路徑，兼顧標準控制項環境之效率與複雜介面場景之適應能力。 為驗證所提框架之可行性，本研究以三項代表性案例研究（Case Study）進行驗證，分別對應宿主自繪控制項、遠端視訊串流（遠端桌面）與虛擬化隔離（虛擬桌面 Terminal）三類 UIA 失效成因。案例研究結果顯示，於上述缺乏結構化介面資訊之情境下，所提方法仍能有效辨識按鈕等關鍵元件，並於重複執行測試中維持穩定之自動化操作能力。此外，CV-Inspector 所建立之 UI Descriptor 可重複套用，提升整體開發效率。 本研究驗證了以電腦視覺技術補強 UI Automation 限制之可行性，並透過雙軌 UI 識別架構擴展 Venom Programming 於遠端桌面與虛擬化環境之適用性，為非侵入式應用程式擴充技術提供更具彈性與實務價值之解決方案。;Enhancing existing Windows desktop applications without access to source code or proprietary APIs has been a focus of our laboratory′s recent research; we refer to this class of non-intrusive extension techniques as application augmentation. In practice， many legacy systems， commercial software products， and third-party applications lack official extensibility interfaces， making conventional integration and automation approaches difficult to apply. To address this issue， our laboratory previously proposed Venom Programming， a non-intrusive application augmentation approach that attaches to existing applications through transparent overlay windows and state-tracking mechanisms to enable operational control and interface enhancement. However， the current Venom framework relies heavily on structured accessibility information provided by Microsoft UI Automation (UIA). When target applications employ custom controls or non-standard accessibility implementations， UIA often fails to retrieve meaningful component information. Moreover， in remote desktop， virtual desktop， and video-streamed application scenarios， interfaces are presented solely as visual content， rendering UIA completely ineffective and significantly limiting the applicability of existing methods. To overcome these limitations， this study proposes a computer vision-based Venom application augmentation framework， using visual information as a complementary identification pathway alongside UIA-based structured information. The proposed framework introduces two core components: CV-Server and CV-Inspector. CV-Server integrates the UI-DETR object detection model， Optical Character Recognition (OCR)， and template matching techniques to perform UI element localization， semantic association analysis， and recognition confidence evaluation from screen images. CV-Inspector provides a visual registration and validation interface that enables developers to define UI descriptors for host applications and target elements， thereby reducing the development overhead associated with integrating computer vision techniques. Furthermore， a dual-path UI identification mechanism is designed within the Venom SDK， allowing developers to flexibly select between UIA-based structural identification and CV-based visual identification according to application scenarios， balancing efficiency in standard control environments with adaptability in complex interface conditions. To evaluate the feasibility of the proposed framework， this study validates the framework through three representative case studies， corresponding to three categories of UIA failure: host self-drawn controls， remote video streaming (remote desktop)， and virtualization isolation (virtual desktop terminal). The case study results demonstrate that， even in these scenarios lacking structured interface information， the proposed method can still effectively identify key UI components such as buttons while maintaining stable automation across repeated executions. In addition， the UI descriptors created by CV-Inspector can be reused， improving overall development efficiency. The results validate the feasibility of overcoming the limitations of UI Automation using computer vision techniques. By introducing a dual-path UI identification architecture， this study extends the applicability of Venom Programming to remote desktop and virtualized application environments， providing a more flexible and practical solution for non-intrusive application augmentation.
