南京大学学报(自然科学版) ›› 2021, Vol. 57 ›› Issue (5): 847–856.doi: 10.13232/j.cnki.jnju.2021.05.015

• • 上一篇    

信息年龄受限下最小化无人机辅助无线供能网络的能耗：一种基于DQN的方法

刘玲珊1, 熊轲1(), 张煜2, 张锐晨1, 樊平毅3,4   

  1. 1.北京交通大学计算机与信息技术学院，北京，100044
    2.国网能源研究院有限公司，北京，102209
    3.清华大学电子工程系，北京，100084
    4.北京信息科学与技术国家研究中心，北京，100084
  • 收稿日期:2021-06-16 出版日期:2021-09-29 发布日期:2021-09-29
  • 通讯作者: 熊轲 E-mail:kxiong@bjtu.edu.cn
  • 作者简介:E⁃mail：kxiong@bjtu.edu.cn
  • 基金资助:
    国家自然科学基金(62071033);国家重点研发计划(2020YFB1806903);国网能源研究院有限公司研究项目(526700190002)

Energy minimization in UAV⁃assisted wireless powered sensor networks with AoI constraints: A DQN⁃based approach

Lingshan Liu1, Ke Xiong1(), Yu Zhang2, Ruichen Zhang1, Pingyi Fan3,4   

  1. 1.School of Computer and Information Technology，Beijing Jiaotong University，Beijing，100044，China
    2.State Grid Energy Research Institute Co. ，Ltd. ，Beijing，102209，China
    3.Department of Electronic Engineering，Tsinghua University，Beijing，100084，China
    4.Beijing National Research Center for Information Science and Technology，Beijing，100084，China
  • Received:2021-06-16 Online:2021-09-29 Published:2021-09-29
  • Contact: Ke Xiong E-mail:kxiong@bjtu.edu.cn

RichHTML

16523

PDF (PC)

961

摘要：

随着5G/B5G的不断发展，无人机在实时数据采集系统中将有广泛应用.利用无人机先给传感器节点进行无线充电，然后传感器节点利用收集到的能量将感知的信息上传无人机，可有效解决户外物联网节点的供电与数据采集问题.然而，由于无人机本身的电量受限，如何在保证无人机充电辅助物联网系统顺利完成新鲜数据采集任务的前提下最小化无人机的能耗至关重要.为此，在满足信息采集新鲜度的要求下，通过联合优化无人机的飞行时间、加速度、转角和传感器节点信息上传和能量收集调度模式，建立无人机能耗最小化优化问题.由于该问题含有整数变量，大规模情况下求解较为困难.因此，首先将其建模为马尔科夫决策过程，然后提出了一种基于DQN （Deep Q Network）的无人机能耗优化算法框架求解，并设计相对应的状态空间、动作空间和奖励函数.仿真结果验证了所提DQN算法的收敛性，同时表明提出的DQN算法比传统的贪婪算法可降低8%~30%的无人机能耗.当传感器个数超过八个时，传统的贪婪算法很难求解，而所提DQN算法仍然能找到最优解.另外，随着AoI （Age of Information）限制值的缩小或传感器数量的增加，无人机的能量消耗会不断地增加，并且由于考虑了转角约束，所提算法优化得到的无人机飞行轨迹会更平滑.

关键词: 无人机辅助无线网络, 信息年龄, 能量收集, 深度强化学习

Abstract:

With the development of 5G/B5G，UAV (Unmanned Aerial Vehicle) will be widely employed in real?time data collection system. UAV can wirelessly charge ground sensors，and then sensors use the collected energy to upload the perceived information to UAV，which can effectively solve the problem of power supply and data collection in outdoor Internet of Things (IoT) systems. However，due to the limited power of UAV，how to minimize the energy consumption of UAV is very important on the premise of ensuring the freshness of collected data in UAV?assisted wireless powered sensor network. Therefore，this paper investigates the optimization problem of UAV's energy consumption minimization under the Age of Information (AoI) constraint by jointly optimizing UAV's flight time，acceleration，rotation angle and scheduling of information collection and energy harvesting. As the problem is a combinational optimization problem with a set of binary variables，it is difficult to be solved in large?scale network. Thus，it's first modeled as a Markov decision process，and a Deep Q Network (DQN)?based UAV's energy?minimal algorithm framework is proposed to solve it，and the corresponding state spaces，action spaces and reward function are designed. Simulation results demonstrate the convergence of the proposed DQN scheme，and also show that the proposed DQN scheme can reduce the UAV's energy consumption by about 8%~30% compared with the traditional greedy scheme. When the sensors' amount is more than eight，the traditional greedy scheme becomes very difficult to solve the problem，while our presented DQN method can still find an optimal solution. Moreover，with the decrease of AoI or the increment of the number of sensors，the energy consumption of UAV increases and the trajectory of UAV becomes smoother with the rotation angle constraint.

Key words: UAV?assisted wireless network, Age of Information (AoI), energy harvesting, Deep Q Network (DQN)

中图分类号: 

  • TP391

图1

系统模型"

图2

AoI模型"

图3

无人机能耗优化算法框架"

表1

仿真参数"

变量取值变量取值
β0 (dB)-50M (mJ)20
S (bits)800a6400
B (MHz)1b0.003
σ02 (dBm)-100vmin (m·s-1)5
δ (s)0.1vmax (m·s-1)20
Pu (w)0.1omin-60°
c10.002omax60°
c270.698amin (m·s-2)-5
g (m·s-2)9.8amax (m·s-2)5
m (kg)9.65β0.001
ε0.1γ0.8

图4

算法收敛性"

图5

不同场景下无人机飞行轨迹和速度变化过程"

图6

AoI对无人机能耗的影响"

图7

本文算法与传统贪心算法的比较"

1 Ullah Z，Al?Turjman F，Mostarda L. Cognition in UAV?aided 5G and beyond communications：A survey. IEEE Transactions on Cognitive Communications and Networking，2020，6(3)：872-891.
2 Villarruel J E G，Corona B T. Proposal for a remote surgery system based on wireless communications，electromyography and robotics∥Proceedings of Electronics，Robotics and Automotive Mechanics Conference. Cuernavaca，Mexico：IEEE，2008：93-98.
3 Kamenov S S，Todorova V D. Some challenges in creating of tactile sensor network in internet environment∥2018 IEEE 27th International Scientific Conference：Electronics. Sozopol，Bulgaria：IEEE，2018：1-3.
4 Yates R D，Kaul S. Real?time status updating：Multiple sources∥2012 IEEE International Symposium on Information Theory. Cambridge，MA，USA：IEEE，2012：2666-2670.
5 Kosta A，Pappas N，Angelakis V. Age of information：A new concept，metric，and tool. Foundations and Trends? in Networking，2017，12(3)：162-259.
6 Wu X W，Yang J，Wu J X. Optimal status updating to minimize age of information with an energy harvesting source∥2017 IEEE International Conference on Communications. Paris，France：IEEE，2017：1-6.
7 Hu L M，Chen Z C，Dong Y Q，et al. Status update in IoT networks：Age of information violation probability and optimal update rate. IEEE Internet of Things Journal，2021，doi：10.1109/JIOT. 2021.3051722.
8 Farazi S，Klein A G，Brown D R. Average age of information for status update systems with an energy harvesting server∥IEEE Conference on Computer Communications Workshops. Honolulu，HI，USA：IEEE，2018：112-117.
9 Yang L，Zeng Y，Zhang R. Wireless power transfer with hybrid beamforming：How many RF chains do we need?IEEE Transactions on Wireless Communications，2018，17(10)：6972-6984.
10 Zhang R C，Xiong K，Guo W，et al. Q?Learning?based adaptive power control in wireless RF energy harvesting heterogeneous networks. IEEE Systems Journal，2021，15(2)：1861-1872.
11 Di X F，Xiong K，Fan P Y，et al. Simultaneous wireless information and power transfer in cooperative relay networks with rateless codes. IEEE Transactions on Vehicular Technology，2017，66(4)：2981-2996.
12 Dabiri M，Emadi M J. Average age of information minimization in an energy harvesting wireless sensor node∥2018 9th International Symposium on Telecom?munications. Tehran，Iran：IEEE，2018：123-126.
13 Wu X W，Yang J，Wu J X. Optimal status update for age of information minimization with an energy harvesting source. IEEE Transactions on Green Communications and Networking，2018，2(1)：193-204.
14 Arafa A，Yang J，Ulukus S，et al. Age?minimal transmission for energy harvesting sensors with finite batteries：Online policies. IEEE Transactions on Information Theory，2020，66(1)：534-556.
15 Li L H，Cai R T，Jiang H，et al. Rate?energy tradeoff for SWIPT systems with multi?user interference channels under non?linear energy harvesting model∥2019 IEEE 89th Vehicular Technology Conference. Kuala Lumpur，Malaysia：IEEE，2019：1-6.
16 Lu Y，Xiong K，Fan P Y，et al. Global energy efficiency in secure MISO SWIPT systems with non?linear power?splitting EH model. IEEE Journal on Selected Areas in Communications，2019，37(1)：216-232.
17 Boshkovska E，Ng D W K，Zlatanov N，et al. Robust resource allocation for MIMO wireless powered communication networks based on a non?linear EH model. IEEE Transactions on Communications，2017，65(5)：1984-1999.
18 Wu Q Q，Li G Y，Ng D W K，et al. An overview of sustainable green 5G networks. IEEE Wireless Communications，2017，24(4)：72-80.
19 Yang D C，Wu Q Q，Zeng Y，et al. Energy tradeoff in ground?to?UAv communication via trajectory design. IEEE Transactions on Vehicular Technology，2018，67(7)：6721-6726.
20 Yang Z H，Xu W，Shikh?Bahaei M. Energy efficient UAV communication with energy harvesting. IEEE Transactions on Vehicular Technology，2020，69(2)：1913-1927.
21 Zeng Y，Zhang R. Energy?efficient UAV communication with trajectory optimization. IEEE Transactions on Wireless Communications，2017，16(6)：3747-3760.
22 Tan R J，Zhou J，Du H B，et al. An modeling processing method for video games based on deep reinforcement learning∥2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference. Chongqing，China：IEEE，2019：939-942.
23 Moreno?Vera F. Performing deep recurrent double
Q?learning for Atari games∥2019 IEEE Latin American Conference on Computational Intelligence. Guayaquil，Ecuador：IEEE，2019：1-4.
24 Mnih V，Kavukcuoglu K，Silver D，et al. Human?level control through deep reinforcement learning. Nature，2019，518(7540)：529-533.
25 Yi M J，Wang X J，Liu J，et al. Deep reinforcement learning for fresh data collection in UAV?assisted IoT networks∥IEEE Conference on Computer Communications Workshops. Toronto，Canada：IEEE，2020：716-721.
26 Abd?Elmagid M A，Ferdowsi A，Dhillon H S，et al. Deep reinforcement learning for minimizing age?of?information in UAV?assisted Networks∥2019 IEEE Global Communications Conference. Waikoloa，HI，USA：IEEE，2019：1-6.
27 Tong P，Liu J，Wang X J，et al. Deep reinforcement learning for efficient data collection in UAV?Aided internet of things∥2020 IEEE International Conference on Communications Workshops. Dublin，Ireland：IEEE，2020：1-6.
28 You C S，Zhang R. 3D trajectory optimization in Rician fading for UAV?enabled data harvesting. IEEE Transactions on Wireless Communications，2019，18(6)：3192-3207.
[1] 王军, 申政文, 李玉莲, 潘在宇. 分离表示学习下的严重缺失静脉信息高质量生成[J]. 南京大学学报(自然科学版), 2021, 57(5): 810-817.
[2] 张礼, 王嘉瑞. 基于图同构网络的自闭症功能磁共振影像诊断算法[J]. 南京大学学报(自然科学版), 2021, 57(5): 801-809.
[3] 张强, 杨吉斌, 张雄伟, 曹铁勇, 梅鹏程. 基于生成对抗网络的音频目标分类对抗[J]. 南京大学学报(自然科学版), 2021, 57(5): 793-800.
[4] 李娜, 段友祥, 孙歧峰, 沈楠. 一种基于样本点距离突变的聚类方法[J]. 南京大学学报(自然科学版), 2021, 57(5): 775-784.
[5] 倪斌, 陆晓蕾, 童逸琦, 马涛, 曾志贤. 胶囊神经网络在期刊文本分类中的应用[J]. 南京大学学报(自然科学版), 2021, 57(5): 750-756.
[6] 陈磊, 孙权森, 王凡海. 基于深度对抗网络和局部模糊探测的目标运动去模糊[J]. 南京大学学报(自然科学版), 2021, 57(5): 735-749.
[7] 卢锦亮, 吴广潮, 冯夫健, 王林. 基于联合轨迹特征的徘徊行为识别方法[J]. 南京大学学报(自然科学版), 2021, 57(5): 724-734.
[8] 薛峰, 李凡, 李爽, 李华锋. 基于域分离和对抗学习的跨域行人重识别[J]. 南京大学学报(自然科学版), 2021, 57(5): 715-723.
[9] 贾霄, 郭顺心, 赵红. 基于图像属性的零样本分类方法综述[J]. 南京大学学报(自然科学版), 2021, 57(4): 531-543.
[10] 吴礼福, 徐行. 融合韵律与动态倒谱特征的语音疲劳度检测[J]. 南京大学学报(自然科学版), 2021, 57(4): 709-714.
[11] 颜志良, 丰智鹏, 刘丹, 王会青. 一种混合深度神经网络的赖氨酸乙酰化位点预测方法[J]. 南京大学学报(自然科学版), 2021, 57(4): 627-640.
[12] 王发旺, 陈睿, 伏云发. 基于DBN和RF的跨被试情绪识别研究[J]. 南京大学学报(自然科学版), 2021, 57(4): 617-626.
[13] 段建设, 崔超然, 宋广乐, 马乐乐, 马玉玲, 尹义龙. 基于多尺度注意力融合的知识追踪方法[J]. 南京大学学报(自然科学版), 2021, 57(4): 591-598.
[14] 杜淑颖, 侯海薇, 丁世飞. 基于多层次特征的深度集成聚类算法[J]. 南京大学学报(自然科学版), 2021, 57(4): 575-581.
[15] 黄鹤, 吴琨, 宋京, 王会峰, 茹锋, 郭璐. 融合全局与区域大气光值图的暗通道图像去雾方法[J]. 南京大学学报(自然科学版), 2021, 57(4): 551-565.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!

版权所有 © 2021 《南京大学学报(自然科学)》

地址：江苏省南京市鼓楼区汉口路22号，南京大学学报编辑部，210093

电话：025-83592704 , 传真：025-83592704        技术支持：北京玛格泰克科技发展有限公司