一种改进的双深度Q网络服务功能链部署算法

刘道华; 魏丁二; 宣贺君; 余长鸣; 寇丽博

doi:10.19665/j.issn1001-2400.20230310

您当前的位置：

首页 >

文章列表页 >

一种改进的双深度Q网络服务功能链部署算法

信息与通信工程 | 更新时间：2024-04-03

- 一种改进的双深度Q网络服务功能链部署算法
- Improved double deep Q network algorithm for service function chain deployment
- 西安电子科技大学学报 2024年51卷第1期页码：52-59
- 作者机构：
  
  信阳师范大学计算机与信息技术学院,河南信阳 464000
- 作者简介：
  
  [ "刘道华(1974—),男,教授,博士,E-mail:[email protected];" ]
  [ "魏丁二(1998—),男,信阳师范大学硕士研究生,E-mail:[email protected];" ]
  [ "宣贺君(1988—),男,副教授,博士,E-mail: [email protected];" ]
  [ "余长鸣(1999—),女,信阳师范大学硕士研究生,E-mail:[email protected]; " ]
  [ "寇丽博(1998—),女,信阳师范大学硕士研究生,E-mail: [email protected]." ]
- 基金信息：
  
  国家自然科学基金(61572417);河南省科技攻关项目(222102210265);河南省本科高校研究性教学改革项目(2022SYJXLX061);河南省高等学校重点科研项目(22A520007);河南省研究生教育改革与质量提升工程项目(YJS2024AL104)
- DOI：10.19665/j.issn1001-2400.20230310
  中图分类号： TP393
- 纸质出版日期：2024-1-20，
  
  网络出版日期：2023-8-30，
  
  收稿日期：2022-11-28，
扫描看全文
刘道华, 魏丁二, 宣贺君, 等. 一种改进的双深度Q网络服务功能链部署算法[J]. 西安电子科技大学学报, 2024,51(1):52-59.

Daohua LIU, Dinger WEI, Hejun XUAN, et al. Improved double deep Q network algorithm for service function chain deployment[J]. Journal of Xidian University, 2024,51(1):52-59.
刘道华, 魏丁二, 宣贺君, 等. 一种改进的双深度Q网络服务功能链部署算法[J]. 西安电子科技大学学报, 2024,51(1):52-59. DOI： 10.19665/j.issn1001-2400.20230310.

Daohua LIU, Dinger WEI, Hejun XUAN, et al. Improved double deep Q network algorithm for service function chain deployment[J]. Journal of Xidian University, 2024,51(1):52-59. DOI： 10.19665/j.issn1001-2400.20230310.

摘要

网络功能虚拟化已成为未来通信网络的关键技术

动态服务功能链的高效部署是提高网络性能迫切需要解决的问题之一。为降低通信网络服务器能量消耗以及改善通信网络服务质量

提出一种改进的双深度Q网络的动态服务功能链部署算法。由于网络状态及服务功能链的动态性

首先将服务功能链部署问题建模为马尔可夫决策过程。根据通信网络中资源的状态以及所选择的动作计算奖励函数值

对双深度Q网络进行在线训练

得到最优深度神经网络模型

从而确定最优的在线服务功能链部署策略。为解决传统深度强化学习从经验回放池中采用均匀抽取经验样本而导致神经网络学习效率低的问题

设计一种基于重要性采样的优先级经验回放方法以抽取经验样本

从而有效地避免训练样本之间的高度相关性

进一步提高离线学习神经网络的效率。仿真结果表明

所提出基于改进双深度Q网络的服务功能链部署算法能够提高奖励值

与传统的双深度Q网络算法相比

在能量消耗与阻塞率方面分别降低约19.89%~36.99%与9.52%~16.37%。

Abstract

Network Function Virtualization(NFV) has become the key technology of next generation communication.Virtual Network Function Service Chain(VNF-SC) mapping is the key issue of the NFV.To reduce the energy consumption of the communication network server and improve the quality of service

a Function Chain(SFC) deployment algorithm based on an improved Double Deep Q Network(DDQN) is proposed to reduce the energy consumption of network servers and improve the network quality of service.Due to the dynamic change of the network state

the service function chain deployment problem is modeled as a Markov Decision Process(MDP).Based on the network state and action rewards

the DDQN is trained online to obtain the optimal deployment strategy for the service function chain.To solve the problem that traditional deep reinforcement learning draws experience samples uniformly from the experience replay pool leading to low learning efficiency of the neural network

a prioritized experience replay method based on importance sampling is designed to draw experience samples so as to avoid high correlation between training samples to improve the learning efficiency of the neural network.Experimental results show that the proposed SFC deployment algorithm based on the improved DDQN can increase the reward value

and that compared with the traditional DDQN algorithm

it can reduce the energy consumption and blocking rate by 19.89%~36.99% and 9.52%~16.37%

respectively.

关键词

服务功能链马尔科夫决策过程网络能耗双深度Q网络

Keywords

service function chainMarkov decision processnetwork energy consumptionDDQN

references

乔文欣, 卢昱, 刘益岑, 等. 空天地协同的边缘云服务功能链动态编排方法[J]. 西安电子科技大学学报, 2022, 49(2):79-88.

QIAO Wenxin, LU Yu, LIU Yicen, et al. Dynamic Scheduling Method for Service Function Chains in Space Air Terrestrial Aided Edge Cloud Network[J]. Journal of Xidian University, 2022, 49(2):79-88.

SHAH H A, ZHAO L. Multiagent Deep-Reinforcement-Learning-Based Virtual Resource Allocation Through Network Function Virtualization in Internet of Things[J]. IEEE Internet of Things Journal, 2021, 8(5):3410-3421.

YANG S, LI F, TRAJANOVSI S, et al. Recent Advances of Resource Allocation in Network Function Virtualization[J]. IEEE Transactions on Parallel and Distributed Systems, 2021, 32(2):259-314.

FAN W, CUI Q, LI X, et al. On Credibility-Based Service Function Chain Deployment[J]. IEEE Open Journal of the Computer Society, 2021, 2:152-163.

CHEN J, CHEN X, CHEN J, et al. A Lightweight SFC Embedding Framework in SDN/NFV-Enabled Wireless Network Based on Reinforcement Learning[J]. IEEE Systems Journal, 2022, 16(3):3817-3828.

FARKIANI S, BAKHSHI B, MIRHASSANI S A, et al. Prioritized Deployment of Dynamic Service Function Chains[J]. IEEE/ACM Transactions on Networking, 2021, 29(3):979-993.

KARIMZADEH F M, MANSOURI S V, NIYATO D. A Dynamic Reliability-Aware Service Placement for Network Function Virtualization(NFV)[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2):318-333.

KHEBBACHE S, HADJI M, ZEGHLACHE D. Scalable and Cost-Efficient Algorithms for VNF Chaining and Placement Problem[C]//Proceedings of the 20th Conference on Innovations in Clouds,Internet and Networks(ICIN).Piscataway:IEEE, 2017:92-99.

PEI J, HONG P, XUE K, et al. Resource Aware Routing for Service Function Chains in SDN and NFV-Enabled Network[J]. IEEE Transactions on Services Computing, 2021, 14(4):985-997.

VIZARRETA P, CONDOLUCI M, MACHUCA C M, et al. QoS-Driven Function Placement Reducing Expenditures in NFV Deployments[C]//Proceedings of the 2017 IEEE International Conference on Communications(ICC). Piscataway:IEEE, 2017:1-7.

魏亮, 黄韬, 张娇, 等. 基于强化学习的服务链映射算法[J]. 通信学报, 2018, 39(1):90-100.

WEI Liang, HUANG Tao, ZHANG Jiao, et al. Service Chain Mapping Algorithm Based on Reinforcement Learning[J]. Journal on Communications, 2018, 39(1):90-100.

JIN Q, GE S, ZENG J, et al. ScaRL:Service Function Chain Allocation Based on Reinforcement Learning in Mobile Edge Computing[C]//Proceedings of the 7th International Conference on Advanced Cloud and Big Data(CBD). Piscataway:IEEE, 2019:327-332.

陈卓, 冯钢, 何颖, 等. 运营商网络中基于深度强化学习的服务功能链迁移机制[J]. 电子与信息学报, 2020, 42(9):2173-2179.

CHEN Zhuo, FENG Gang, HE Ying, et al. Service Function Chain Migration Mechanism Based on Deep Reinforcement Learning in Operator networks[J]. Journal of Electronics & Information Technology, 2020, 42(9):2173-2179.

HOU Y, LIU L, WEI Q, et al. A Novel DDPG Method with Prioritized Experience Replay[C]//IEEE International Conference on Systems,Man,and Cybernetics(SMC). Piscataway:IEEE, 2017,316-321.

金明, 李琳琳, 张文瑾, 等. 基于深度强化学习的服务功能链映射算法[J]. 计算机应用研究, 2020, 37(11):3456-3460.

JIN Ming, LI Linlin, ZHANG Wenjin, et al. Mapping Algorithm of Service Function Chain Based on Deep Reinforcement Learning[J]. Computer Application Research, 2020, 37(11):3456-3460.

更多指标>

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据