| 摘要: | 金融科技是人工智慧應用的主要領域之一，其中包括股票漲跌幅預測、資
產配置等項目。然而，僅依賴股票價格預測並無法保證投資回報的最大化，投資
者還必須兼顧資產配置策略，以達到最大化收益或最小化損失的目的。在這種
需要與環境互動以獲得報酬的情境下，強化學習（Reinforcement Learning，RL）成為一種理想的解決方法。因此，在本研究中，我們提出了一種使用RL的ActorCritic技術進行股票投資的策略。為了提升投資決策的效果，我們運用自動編碼器（AutoEncoder，AE）學習股票中的多項技術指標特徵，並用其進行股票持有配
置的決策和回報估計。
然而，投資組合管理在優化配置策略和精確預測回報方面仍面臨挑戰，特別
是在面對市場波動時。傳統策略通常偏向專注於短期或長期投資，這造成市場上
缺乏一個能靈活應對各種情境的模型。為解決這個問題，我們的研究提出了一種
新的結合強化學習和自動編碼器的方法，希望透過這種方式填補市場上的這個空缺。
我們透過消融實驗來探討AutoEncoder的編碼維度與歷史資料長度對狀態編
碼的影響。結果顯示，使用過去30日的歷史資料並將其壓縮至5個維度，能夠得到
最佳的狀態編碼效果。我們也發現加入AutoEncoder Predictor的預測結果能提高累積收益。此外，我們更進一步探討了三種不同的投資策略：RL+AE Predictor，RL Only，以及AE Predictor。透過效能分析、與大盤相關係數的探討，以及誤判率分析，我們評估這三種策略在不同市場環境下的表現。
實驗結果顯示，作為約束的投資策略，RL+AE Predictor在資產最大化上表
現最佳，且學習過程穩定。尤其在市場劇變時，該策略展現了良好的抗風險能力
並能維持穩定的投資回報。此外，該策略與大盤相關係數較低，顯示出其與市場
指數波動的獨立性。在誤判率分析中，RL+AE Predictor模型的FPR(False Positive Rate，誤判率)為6.46%相較於AE Predictor 38.38%及RL Only 36.09%來的低，顯示其在預測股票資產配置的表現最佳，誤判率最低。
我們將此方法驗證於台灣的股票市場環境，以2019年至2021年的台股資料進
行實驗，並與TW50指數、傳統投資組合理論(Mean-variance optimization，MVO)、以及使用強化學習Policy Gradient技術的Jiang’s研究進行比較。實驗結果顯示，本研究的贏率在短期投資3個月至中長期6-9個月的投資週期以及長期投
資(1年-2年)下優於比較的基準TW50、Jiang’s及MVO，且在長期12個月及24個月
的長期投資週期下達到最高總收益。即使在2019年多頭牛市及2020年熊市兩個不同投資起始點進行的2年固定投資時間的長期投資比較中，本論文所提出的方法仍
能贏過TW50指數、MVO以及Jiang’s。
總結來說，這項研究提供了強化學習和自動編碼器在投資組合管理中，無論
在累積回報率還是夏普比率上，都優於傳統的MVO、TW50指數以及Jiang的混合
型深度學習方法的實證證據，並強調了AI在複雜的金融決策中的潛力，並指出了
需要一個更靈活，通用的模型來填補短期和長期投資策略之間的差距。這些研究
成果為投資策略的發展和改進提供了重要的參考價值。;Financial technology (FinTech) has emerged as one of the key areas for the application of artificial intelligence (AI), including but not limited to, the prediction of stock
market movements and asset allocation. However, relying solely on stock price forecasting does not guarantee the maximization of investment returns. An investor also needs to
consider asset allocation strategies to either maximize the returns or minimize the losses.
In such a scenario that requires interactions with the environment to reap rewards, reinforcement learning (RL) emerges as an ideal solution. Consequently, in this study, we
propose a strategy for stock investment that employs the Actor-Critic techniques of RL.
To enhance the effectiveness of investment decisions, we employed an AutoEncoder (AE) to learn the features of various technical indicators in stocks, which then aids in making
decisions on stock allocation and return estimation.
However, portfolio management still faces challenges in optimizing allocation strategies and accurately forecasting returns, especially during market volatility. Traditional
strategies often focus on either short-term or long-term investments, leading to a lack of a model in the market that can adapt flexibly to various situations. To address this
problem, we have introduced a novel method that combines reinforcement learning and autoencoders, hoping to fill this gap in the market.
We employed ablation experiments to explore the effects of the dimension of AutoEncoder encoding and the length of historical data on state encoding. The results indicate that by compressing the past 30 days of historical data into five dimensions, the optimal state encoding effect can be achieved. We also discovered that incorporating the
prediction results of the AutoEncoder Predictor can enhance cumulative earnings. Furthermore, we investigated three different investment strategies: RL+AE Predictor, RL
Only, and AE Predictor. Through performance analysis, correlation with the broader market, and error rate analysis, we evaluated the performance of these three strategies in
various market environments.
Experimental results reveal that as a constrained investment strategy, RL+AE Predictor performs the best in maximizing assets and exhibits a stable learning process. Especially during significant market changes, this strategy showcases superior risk resistance and can maintain stable investment returns. Moreover, this strategy has a lower correlation coefficient with the broader market, indicating its independence from market index volatility. In the error rate analysis, the RL+AE Predictor model has a False Positive
Rate (FPR) of 6.46%, which is lower compared to AE Predictor at 38.38% and RL Only at 36.09%. This shows that it outperforms in predicting stock asset allocation, with the lowest error rate.
We validated this method in Taiwan’s stock market environment, conducting experiments with Taiwan stock data from 2019 to 2021 and compared it with the TW50
Index, traditional portfolio theory (Mean-variance optimization, MVO), and Jiang’s
research that uses reinforcement learning Policy Gradient techniques. The experimental results show that the win rate of this study in short-term (3 months), mid-to-long term
(6-9 months), and long-term (1-2 years) investment periods is superior to the benchmarks
TW50, Jiang’s, and MVO, reaching the highest total return in the 12 and 24 month lognterm investment periods. Even in the two-year fixed investment time comparison, starting
at two different investment points, the bull market of 2019 and the bear market of 2020,
the method proposed in this thesis still outperforms the TW50 Index, MVO, and Jiang’s.
In summary, this research provides empirical evidence that a combination of reinforcement learning and autoencoders in portfolio management outperforms the traditional
MVO, TW50 Index, and Jiang’s hybrid deep learning methods in both cumulative return rate and Sharpe ratio. It highlights the potential of AI in complex financial decisions and
points out the need for a more flexible, universal model to bridge the gap between shortterm and long-term investment strategies. These research findings provide significant
reference value for the development and improvement of investment strategies |
