在5G-CV2X 環境下利用Dueling Double DQN 最小化任務卸載時間之研究

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：41

、訪客IP：18.220.1.239

姓名

徐偉銘(Wei-Ming XU) 查詢紙本館藏

畢業系所

通訊工程學系

論文名稱

在5G-CV2X 環境下利用Dueling Double DQN 最小化任務卸載時間之研究
(On Optimizing Low-Latency Task Offloading with Dueling Double DQN in 5G-CV2X Environments)

相關論文

★ 非結構同儕網路上以特徵相似度為基準之搜尋方法	★ 以階層式叢集聲譽為基礎之行動同儕網路拓撲架構
★ 線上RSS新聞資料流中主題性事件監測機制之設計與實作	★ 耐延遲網路下具密度感知的路由方法
★ 整合P2P與UPnP內容分享服務之家用多媒體閘道器：設計與實作	★ 家庭網路下簡易無縫式串流影音播放服務之設計與實作
★ 耐延遲網路下訊息傳遞時間分析與高效能路由演算法設計	★ BitTorrent P2P 檔案系統下載端網路資源之可調式配置方法與效能實測
★ 耐延遲網路中利用訊息編碼重組條件之資料傳播機制	★ 耐延遲網路中基於人類移動模式之路由機制
★ 車載網路中以資料匯集技術改善傳輸效能之封包傳送機制	★ 適用於交叉路口環境之車輛叢集方法
★ 車載網路下結合路側單元輔助之訊息廣播機制	★ 耐延遲網路下以靜態中繼節點（暫存盒）最佳化訊息傳遞效能之研究
★ 耐延遲網路下以動態叢集感知建構之訊息傳遞機制	★ 跨裝置影音匯流平台之設計與實作

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

隨著自動駕駛技術的發展，在車載資通訊環境下車輛裝置產生龐大且持續成倍增加的資料量，通訊網路系統不但需要提高資料傳輸效率，更要維持不同車載應用的服務品質。傳統的通訊網路系統主要在提供車輛裝置上行及下載的數據傳遞服務，當前自動駕駛車輛與ITS系統的迅速發展，車輛及其他鄰近車輛間的訊息交換與計算工作協作，遂發展出一新興的分散式資料傳輸模式，同時也擴大通訊網路系統在資料處理工作的負荷。近年來，車輛間資料傳輸之研究經常採用強化學習技術來解決任務卸載的問題，強化學習能夠在沒有事先知識的情況下學習並與環境互動，從而發展出一優化策略。
最近關於自動駕駛數據處理的研究文獻提出的許多方法在這些技術中，在計算卸載問題上並沒有考慮到車輛安全問題，在計算卸載量大幅增加的情況下，傳輸上恐發生過度延遲，或者，在高速行駛的狀況下強化學習的行為決策無法壓制在延遲門閥值之下。強化學習能夠在沒有先驗知識的情況下學習並與環境互動，從而發現最優策略。深度強化學習方法已經被探索出來，以實現最低的能源消耗或最小的延遲。
因此我們的研究提出了一套適用在車載資通訊環境下並具備低延遲的資料計算卸載機制。此機制建立在以5G-CV2X的行動網路環境下，本機制採納車輛本機端、多接取邊緣架構伺服器以及雲端伺服器的資料處理負荷等因素，此機制功能包括：透過深度強化學習來動態決定車輛資料計算的卸載和合作策略，使車輛能夠隨著車輛和道路情況動態調整卸載策略、透過計算卸載的方式，有效地分散資料處理負荷使車輛穩定性提高、實現車輛的低延遲卸載和高駕駛安全性。

摘要(英)

With the advancement of autonomous driving technology, the mass of in-vehicle devices generate a massive and continuously increasing amount of data in vehicular communication environments. Communication network systems not only need to improve data transmission efficiency, but also maintain the service quality of different vehicular applications.Traditional communication network systems mainly focus on providing data transmissions for in-vehicle devices′ uplink and download services. However, with the rapid development of autonomous driving vehicles and Intelligent Transportation Systems (ITS), a new emerging distributed data transmission model has emerged for facilitating message exchange and collaborative computing among vehicles and neighboring vehicles. This expansion places a heavier workload of data processing task on the communication network.
In the literature, recent research recent research on autonomous driving data processing has proposed various methods for computation offloading. However, many of these methods have not adequately considered vehicle safety. As computation offloading increases significantly, it may cases in excessive delays during data transmissions. In the high-speed driving, reinforcement learning-based decision-making cannot suppress the delays below the threshold.
To address these challenges, our study proposes a low-latency data computation offloading mechanism suitable for vehicular communication environments. This mechanism can operate in a 5G-CV2X mobile network environment and can be incorporated with several factors such as local vehicle capabilities, Multi-access Edge Computing (MEC) servers, and cloud servers for data processing loads. In addition, the functionality of this mechanism includes the dynamic determination of vehicle data computation offloading and the collaboration strategies using deep reinforcement learning. This mechanism enables vehicles to adjust offloading strategies dynamically based on vehicle and road conditions, effectively distributing data processing loads to enhance vehicle stability, achieve low-latency computation offloading, and ensure high driving safety.

關鍵字(中)

★ 計算卸載
★ 強化學習
★ 多接取邊緣架構
★ C-V2X

關鍵字(英)

★ Computing Offloading
★ Reinforcement Learning
★ Multi-access Edge Computing
★ C-V2X

論文目次

1 簡介 1
2 研究背景及文獻探討 3
2.1 計算卸載 (Computing Offloading) 3
2.2 車輛群策感知協作計算 5
2.2.1 多接取邊緣架構 (MEC , Multi-access Edge Computing) 6
2.2.2 雲計算 7
2.2.3 V2V（Vehicle-to-Vehicle） 8
2.3 車聯網 (V2X, Vehicle-to-everything) 9
2.3.1 DSRC (Dedicated Short-Range Communications) 10
2.3.2 C-V2X（Vehicle-to-Everything）11
2.4 強化學習 12
2.4.1 車聯網結合強化學習背景 12
2.4.2 強化學習 13
2.4.3 深度強化學習 16
3 研究方法 20
3.1 系統架構 21
3.2 環境及問題定義 24
3.3 深度強化學習 26
4 實驗結果 35
4.1 模擬環境配置 35
4.2 實驗參數設計 36
4.2.1 車載模擬環境設計 37
4.2.2 網路傳輸環境設計 40
4.2.3 模型參數設計 41
4.3 實驗結果 42
4.3.1 V2X 與 DSRC 延遲比較 42
4.3.2 學習率對於 Dueling-DDQN 的影響 44
4.3.3 衰減率對於 Dueling-DDQN 的影響 46
4.3.4 不同演算法之間獎勵分數比較 48
4.3.5 三重驗證 Dueling-DDQN 分佈型態 51
4.3.6 不同卸載演算法執行車載行為決策影響 59
5 結論與未來研究 63
參考文獻 64

參考文獻

[1] J. Liu, W. Wang, D. Li, S. Wan, and H. Liu, “Role of gifts in decision making: an endowment effect incentive mechanism for offloading in the iov,” IEEE Internet of Things Journal, vol. 6, no. 4, pp. 6933–6951, 2019.
[2] L. Liu, C. Chen, Q. Pei, S. Maharjan, and Y. Zhang, “Vehicular edge computing and networking: A survey,” Mobile networks and applications, vol. 26, pp. 1145–1168, 2021.
[3] X. Cao, F. Wang, J. Xu, R. Zhang, and S. Cui, “Joint computation and communication cooperation for mobile edge computing,” in 2018 16th International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt). IEEE, 2018, pp. 1–6.
[4] S. Balin, “Non-identical parallel machine scheduling using genetic algorithm,” Expert Systems with Applications, vol. 38, no. 6, pp. 6814–6821, 2011.
[5] M. Qin, N. Cheng, Z. Jing, T. Yang, W. Xu, Q. Yang, and R. R. Rao, “Service-oriented energy-latency tradeoff for IoT task partial offloading in mec-enhanced multi-rat networks,” IEEE Internet of Things Journal, vol. 8, no. 3, pp. 1896–1907, 2020.
[6] K. Zhang, S. Leng, X. Peng, L. Pan, S. Maharjan, and Y. Zhang, “Artificial intelligence inspired transmission scheduling in cognitive vehicular communications and networks,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1987–1997, 2018.
[7] Q. Chen, S. Tang, Q. Yang, and S. Fu, “Cooper: Cooperative perception for connected autonomous vehicles based on 3D point clouds,” in 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). IEEE, 2019, pp. 514–524.
[8] Q. Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu, “F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3d point clouds,” in Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, 2019, pp. 88–100.
[9] D. Kuo. (2021, Oct) Road traffic safety protal site on statistics on traffic accidents. [Online]. Available: https://oie.nccuc.tw/knowledge/view/15
[10] M. of Transportation and Communications. Office of innovation and entrepreneurship. [Online]. Available: https://roadsafety.tw/Dashboard/Custom?type=30%E6%97%A5%E6%AD%BB%E4
[11] Z. Xiao, X. Dai, H. Jiang, D. Wang, H. Chen, L. Yang, and F. Zeng, “Vehicular task offloading via heat-aware mec cooperation using game-theoretic method,” IEEE Internet of Things Journal, vol. 7, no. 3, pp. 2038–2052, 2019.
[12] W. C. Tamayo, N. E. Chelbi, D. Gingras, and F. Faulconnier, “Improving object distance estimation in automated driving systems using camera images, lidar point clouds and hierarchical clustering,” in 2021 IEEE Intelligent Vehicles Symposium Workshops (IV Workshops). IEEE, 2021, pp. 299–305.
[13] C.-F. Liu, M. Bennis, M. Debbah, and H. V. Poor, “Dynamic task offloading and resource allocation for ultra-reliable low-latency edge computing,” IEEE Transactions on Communications, vol. 67, no. 6, pp. 4132–4150, 2019.
[14] P. Liu, J. Li, and Z. Sun, “Matching-based task offloading for vehicular edge computing,” IEEE Access, vol. 7, pp. 27 628–27 640, 2019.
[15] S. Nath and J. Wu, “Deep reinforcement learning for dynamic computation offloading and resource allocation in cache-assisted mobile edge computing systems,” Intelligent and Converged Networks, vol. 1, no. 2, pp. 181–198, 2020.
[16] S. Jošilo and G. Dán, “Decentralized algorithm for randomized task allocation in fog computing systems,” IEEE/ACM Transactions on Networking, vol. 27, no. 1, pp. 85–97, 2018.
[17] L. Tang and H. Hu, “Computation offloading and resource allocation for the Internet of Things in energy-constrained MEC-enabled hetnets,” IEEE Access, vol. 8, pp. 47 509–47 521, 2020.
[18] B. Yu, X. Zhang, I. You, and U. S. Khan, “Efficient computation offloading in edge computing enabled smart home,” IEEE Access, vol. 9, pp. 48 631–48 639, 2021.
[19] Q. Luo, C. Li, T. H. Luan, and W. Shi, “Collaborative data scheduling for vehicular edge computing via deep reinforcement learning,” IEEE Internet of Things Journal, vol. 7, no. 10, pp. 9637–9650, 2020.
[20] X. Wang, Z. Ning, and S. Guo, “Multi-agent imitation learning for pervasive edge computing: A decentralized computation offloading algorithm,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 2, pp. 411–425, 2020.
[21] Z. HussainR, “Autonomous cars: research results, issues, and future challenges,” IEEE Communications Surveys & Tutorials, vol. 21, no. 2, p. 1275, 2019.
[22] I. Yaqoob, L. U. Khan, S. A. Kazmi, M. Imran, N. Guizani, and C. S. Hong, “Autonomous driving cars in smart cities: Recent advances, requirements, and challenges,” IEEE Network, vol. 34, no. 1, pp. 174–181, 2019.
[23] H. Bagheri, M. Noor-A-Rahim, Z. Liu, H. Lee, D. Pesch, K. Moessner, and P. Xiao, “5G NR-V2X: Toward connected and cooperative autonomous driving,” IEEE Communications Standards Magazine, vol. 5, no. 1, pp. 48–54, 2021.
[24] U. S. D. of Transportation. (2017, Dec) Nhtsa-iihs announcement on aeb. [Online]. Available: https://www.nhtsa.gov/press-releases/nhtsa-iihs-announcement-aeb
[25] T. Germain. (2017, Sep) Where automakers stand on automatic emergency braking pledge. [Online]. Available: https://www.consumerreports.org/car-safety/where-automakers-stand-on-automatic-emergency-braking-pledge/
[26] Y. Zhang, C. Li, T. H. Luan, C. Yuen, and Y. Fu, “Collaborative driving: Learning-aided joint topology formulation and beamforming,” IEEE Vehicular Technology Magazine, vol. 17, no. 2, pp. 103–111, 2022.
[27] E. I. S. Group. (2016, Mar) ETSI GS MEC 002 (2016) Mobile edge computing (MEC) technical requirements v1.1.1. [Online]. Available: https://www.etsi.org/deliver/etsi_gs/MEC/001_099/002/01.01.01_60/gs_MEC002v010101p.pdf
[28] N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, “Mobile edge computing: A survey,” IEEE Internet of Things Journal, vol. 5, no. 1, pp. 450–465, 2017.
[29] N. Alliance, “5G white paper,” Next generation mobile networks, white paper, vol. 1, no. 2015, 2015.
[30] P. Mach and Z. Becvar, “Mobile edge computing: A survey on architecture and computation offloading,” IEEE communications surveys & tutorials, vol. 19, no. 3, pp. 1628–1656, 2017.
[31] What is cloud computing? [Online]. Available: https://cloud.google.com/learn/what-is-cloud-computing
[32] P.-J. Maenhaut, H. Moens, B. Volckaert, V. Ongenae, and F. De Turck, “Resource allocation in the cloud: From simulation to experimental validation,” in 2017 IEEE 10th International Conference on Cloud Computing (CLOUD). IEEE, 2017, pp. 701–704.
[33] Z. Yao, Z. Yang, T. Wu, L. Chen, K. Zhu, L. Zhang, and S. Su, “Implementing its applications by LTE-V2X equipment-challenges and opportunities,” in 2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC). IEEE, 2018, pp. 120–124.
[34] R. Yee, E. Chan, B. Cheng, and G. Bansal, “Collaborative perception for automated vehicles leveraging vehicle-to-vehicle communications,” in 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1099–1106.
[35] S.-W. Kim, B. Qin, Z. J. Chong, X. Shen, W. Liu, M. H. Ang, E. Frazzoli, and D. Rus, “Multivehicle cooperative driving using cooperative perception: Design and experimental validation,” IEEE Transactions on Intelligent Transportation
[36] J.-Y. Choi, H.-S. Jo, C. Mun, and J.-G. Yook, “Deep reinforcement learning-based distributed congestion control in cellular v2x networks,” IEEE Wireless Communications Letters, vol. 10, no. 11, pp. 2582–2586, 2021.
[37] S. B. Prathiba, G. Raja, S. Anbalagan, K. Arikumar, S. Gurumoorthy, and K. Dev, “A hybrid deep sensor anomaly detection for autonomous vehicles in 6g-v2x environment,” IEEE Transactions on Network Science and Engineering, vol. 10, no. 3, pp. 1246–1255, 2022.
[38] Y. Sun, M. Peng, and S. Mao, “Deep reinforcement learning-based mode selection and resource management for green fog radio access networks,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 1960–1971, 2018.
[39] W. Liu, G. Qin, Y. He, and F. Jiang, “Distributed cooperative reinforcement learning-based traffic signal control that integrates v2x networks′ dynamic clustering,” IEEE transactions on vehicular technology, vol. 66, no. 10, pp. 8667–8681, 2017.
[40] L. Lei, T. Liu, K. Zheng, and L. Hanzo, “Deep reinforcement learning aided platoon control relying on v2x information,” IEEE Transactions on Vehicular Technology, vol. 71, no. 6, pp. 5811–5826, 2022.
[41] C. Arvind and J. Senthilnath, “Autonomous RL: Autonomous vehicle obstacle avoidance in a dynamic environment using MLP-SARSA reinforcement learning,” in 2019 IEEE 5th International Conference on Mechatronics System and Robots (ICMSR). IEEE, 2019, pp. 120–124.
[42] Q. Zhou, C. Guo, C. Wang, and L. Cui, “Radio resource management for C-V2X using graph matching and actor–critic learning,” IEEE Wireless Communications Letters, vol. 11, no. 12, pp. 2645–2649, 2022.
[43] W. Yang, B. Jeon, C. Mun, and H.-S. Jo, “Cellular-v2x QoS adaptive distributed congestion control: A deep Q network approach,” in 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC). IEEE, 2023, pp. 967–968.
[44] J. Miao, X. Chai, X. Song, and T. Song, “A DDQN-based energy-efficient resource allocation scheme for low-latency V2V communication,” in 2022 IEEE 5th International Electrical and Energy Conference (CIEEC). IEEE, 2022, pp. 53–58.
[45] T.-W. Ban, “An autonomous transmission scheme using dueling DQN for D2D communication networks,” IEEE transactions on vehicular technology, vol. 69, no. 12, pp. 16 348–16 352, 2020.
[46] Y. Li, C. Yang, M. Deng, X. Tang, and W. Li, “A dynamic resource optimization scheme for MEC task offloading based on policy gradient,” in 2022 IEEE 6th Information Technology and Mechatronics Engineering Conference (ITOEC), vol. 6. IEEE, 2022, pp. 342–345.
[47] Z. Mlika and S. Cherkaoui, “Deep deterministic policy gradient to minimize the age of information in cellular V2X communications,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 23 597–23 612, 2022.
[48] F. Jiang, R. Ma, C. Sun, and Z. Gu, “Dueling deep Q-network learning based computing offloading scheme for F-RAN,” in 2020 IEEE 31st Annual International Symposium on Personal, Indoor and Mobile Radio Communications. IEEE, 2020, pp. 1–6.
[49] K. Li, X. Xu, and S. Han, “A multi-agent dueling DQN based route selection scheme for IAB congestion controlling,” in 2022 IEEE Wireless Communications and Networking Conference (WCNC). IEEE, 2022, pp. 512–517.
[50] H. Song, Y. Liu, J. Zhao, J. Liu, and G. Wu, “Prioritized replay dueling DDQN based grid-edge control of community energy storage system,” IEEE Transactions on Smart Grid, vol. 12, no. 6, pp. 4950–4961, 2021.
[51] Y. Sun, W. Zuo, and M. Liu, “See the future: A semantic segmentation network predicting ego-vehicle trajectory with a single monocular camera,” IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3066–3073, 2020.
[52] B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, “A survey of motion planning and control techniques for self-driving urban vehicles,” IEEE Transactions on intelligent vehicles, vol. 1, no. 1, pp. 33–55, 2016.
[53] W. Lim, S. Lee, M. Sunwoo, and K. Jo, “Hierarchical trajectory planning of an autonomous car based on the integration of a sampling and an optimization method,” IEEE Transactions on Intelligent Transportation Systems, vol. 19, no. 2, pp. 613–626, 2018.
[54] Z. Huang, D. Chu, C. Wu, and Y. He, “Path planning and cooperative control for automated vehicle platoon using hybrid automata,” IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 3, pp. 959–974, 2018.
[55] M. H. C. Garcia, A. Molina-Galan, M. Boban, J. Gozalvez, B. Coll-Perales, T. Şahin, and A. Kousaridas, “A tutorial on 5G NR V2X communications,” IEEE Communications Surveys & Tutorials, vol. 23, no. 3, pp. 1972–2026, 2021.
[56] S. Jošilo and G. Dán, “Decentralized algorithm for randomized task allocation in fog computing systems,” IEEE/ACM Transactions on Networking, vol. 27, no. 1, pp. 85–97, 2018.
[57] J. Yan, J. Xiao, and X. Hong, “Dueling-DDQN based virtual machine placement algorithm for cloud computing systems,” in 2021 IEEE/CIC International Conference on Communications in China (ICCC). IEEE, 2021, pp. 294–299.
[58] J. Ren, H. Wang, T. Hou, S. Zheng, and C. Tang, “Collaborative edge computing and caching with deep reinforcement learning decision agents,” IEEE Access, vol. 8, pp. 120 604–120 612, 2020.
[59] Y. Lu, H. Ma, E. Smart, and H. Yu, “Real-time performance-focused localization techniques for autonomous vehicle: A review,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 6082–6100, 2021.
[60] C. Urmson, “Driving beyond stopping distance constraints,” in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2006, pp. 1189–1194.
[61] J. Khoury, K. Amine, R. Abi Saad et al., “An initial investigation of the effects of a fully automated vehicle fleet on geometric design,” Journal of Advanced Transportation, vol. 2019, 2019.
[62] S. Heinrich and L. Motors, “Flash memory in the emerging age of autonomy,” Flash Memory Summit, pp. 1–10, 2017.
[63] A. Ndikumana, K. K. Nguyen, and M. Cheriet, “Age of processing-based data offloading for autonomous vehicles in multirats open RAN,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 21 450–21 464, 2022.
[64] G. Noh, J. Kim, S. Choi, N. Lee, H. Chung, and I. Kim, “Feasibility validation of a 5G-enabled mmWave vehicular communication system on a highway,” IEEE Access, vol. 9, pp. 36 535–36 546, 2021.
[65] 交通部高速公路交通資料庫. [Online]. Available: https://tisvcloud.freeway.gov.tw/
[66] 交通部行高速公路局行車時間預測. [Online]. Available: https://1968.freeway.gov.tw/tp_future

指導教授

胡誌麟(Chih-Lin Hu)

審核日期

2023-8-14

推文