為突破此限制，本研究提出一項基於階層式強化學習（Hierarchical Reinforcement Learning， HRL）的派工框架，將複雜的排程任務拆解為兩個層級。其中，高階代理人（Manager）負責巨觀的配方目標指派，低階代理人（Worker）則執行局部的工單派工，兩個代理人皆使用近端策略優化演算法。實驗結果證實，此雙層架構不僅能有效免除計算全局特徵的龐大成本，在較多工單及較少機台的場景下仍具備卓越的擴展性與穩定的收斂能力，進而顯著降低總延遲時間。更重要的是，透過集中指派配方的機制，本模型自然湧現出大幅減少換線次數的行為，即使在訓練過程中並未明確給予減少換線的獎勵訊號。因此，相較於傳統排程演算法以及採用 Linear Thompson Sampling 的 Manager，該方法可將換線次數降低超過 50 %。;The Parallel-Machine Scheduling Problem (PMSP) focuses on assigning jobs to multiple machines to maximize production efficiency. In modern factories， such as semiconductor manufacturing， this is very difficult because of sequence-dependent setup times and strict due dates. Traditional methods， like heuristic rules or mathematical programming， struggle to balance real-time speed and scheduling quality. They can no longer handle highly dynamic production needs. Although Deep Reinforcement Learning (DRL) is a promising solution， traditional single-level DRL models face the "curse of dimensionality" and huge computing costs in large-scale environments.
To solve this problem， this study proposes a Hierarchical Reinforcement Learning (HRL) framework that breaks the complex scheduling task into two levels. A high-level agent (Manager) assigns macro-level recipe goals， while a low-level agent (Worker) handles local job dispatching. Both agents utilize the Proximal Policy Optimization (PPO) algorithm. Experimental results show that this two-level design avoids the heavy cost of processing global features. The proposed method demonstrates good scalability and stable training behavior， particularly in scenarios involving many jobs and few machines， where it substantially reduces total tardiness. Furthermore， centralized recipe assignment enables the model to implicitly learn to minimize setup operations， even without an explicit training objective for setup reduction. Consequently， the method reduces setup frequency by more than 50% compared with both traditional scheduling algorithms and a Manager based on Linear Thompson Sampling.
