Computer-Using Agent（CUA）讓使用者可以用自然語言要求AI 觀 察畫面並操作圖形使用者介面（Graphical User Interface， GUI）；這類代理 人通常仰賴生成式AI 與大型語言模型進行畫面理解、工具選擇與動作生 成。若每個步驟都由模型在執行時即時判斷，路徑與動作就可能受到模 型輸出的不確定性影響。對製造、醫療與受管制企業後台等高風險場域 而言，重點不只是能否完成任務，而是能否先知道哪些操作路徑已被實 際錄製、可被接受，且缺少知識時必須停止。這需要一份能描述畫面、 操作意圖、前置條件與可走路徑的GUI 操作知識。 本研究提出GUI Model Learning，作為自動化執行之前的GUI 操作 知識形成方法。系統以操作手冊或SOP 建立GUI Model 初版，並透過錄 製保存熟練使用者的實際操作。錄製資料中的螢幕畫面、OCR 與UI 元 件定位結果、鍵盤滑鼠事件，會被整理成候選GUI Model。多次錄製可 逐步補上穩定路徑、不同使用者的可接受做法，以及應停止的受阻情境； 相較於傳統RPA 需要為多條完整路徑分別錄製或維護腳本，本研究將錄 製結果整理為可組合的模型單位。這些資料可先用於知識傳承、教育訓 練與流程觀察，當資料要成為正式模型或交給後續自動化使用時，再由 審核工程師確認、退回或接受。 GUI Model 不只保存成功路徑，也保存「何時不應繼續」的判斷。例 如前置條件不足、權限限制或資料驗證錯誤，都可能代表後續自動化應 停止，而不是嘗試其他未知操作。這使GUI Model Learning 的重點從單 次腳本重放，轉為建立一份可供人與系統共同使用的GUI 操作知識。 受控工業檢測流程個案顯示，日常錄製能形成候選畫面情境、操作 意圖、流程路徑與操作步驟；後續自動化元件只讀取已接受GUI Model 產生 RPA 腳本，並在缺少起點、路徑或步驟時停止。比較分析指出， RPA 適合固定執行，CUA 適合依畫面推理，但兩者若要用於高風險GUI 流程，都需要一層由人類示範與實際錄製形成的本地操作知識作為邊界。 本研究補上的，正是從本地使用者操作經驗建立GUI Model 的方法，使 以提示詞驅動的不確定操作目標能轉為沿已知路徑產生腳本的受控自動 化流程。;Computer-Using Agents (CUAs) allow users to request graphical user interface operations in natural language. These agents often rely on generative AI and large language models for screen interpretation， tool selection， and action generation; if every step is decided at runtime， the resulting operation path can remain uncertain. In manufacturing， medical， and regulated enterprise settings， the central issue is not only whether an agent can complete a task， but whether the system knows which operation paths have been demonstrated， can be accepted， and must stop when knowledge is missing. This requires GUI operation knowledge that describes screens， operation intents， preconditions， and executable paths. This thesis proposes GUI Model Learning as a method for forming GUI operation knowledge before automation is executed. The system builds an initial GUI Model from a manual or SOP and records skilled users’ actual operations. Screen content， OCR and UI-element localization results， keyboard and mouse events are organized into a candidate GUI Model. Repeated demonstrations gradually add stable paths， acceptable variants across users， and blocked situations where execution should stop. Unlike traditional RPA， which often requires separate recording or maintenance for multiple complete paths， this work organizes demonstrations into composable model units. Candidate updates are reviewed by a human reviewer only when they are promoted into a formal model or used by downstream automation. A GUI Model stores not only successful paths， but also decisions about when execution should not continue. Missing prerequisites， permission limits， and validation errors may indicate that downstream automation should stop instead of trying unknown actions. GUI Model Learning therefore shifts the focus from replaying a single script to building GUI operation knowledge that both humans and automation systems can use. A controlled industrial inspection case study shows that daily recordings can form candidate screen contexts， operation intents， process paths， and action steps. Downstream automation components generate RPA scripts only from the accepted GUI Model; when the model lacks a start state， route， or action steps， the system stops instead of guessing. The comparison indicates that RPA is suitable for fixed execution and CUAs are suitable for screen-based task reasoning， but high-risk GUI workflows need a local operation-knowledge layer before either approach can be used responsibly. This thesis contributes that layer: a method for building a reviewable GUI Model from local demonstrations， so that prompt-driven uncertain operation goals can be transformed into controlled automation scripts along accepted paths.
