key: cord-0588458-y0sralzj
authors: Joloudari, Javad Hassannataj; Alizadehsani, Roohallah; Nodehi, Issa; Mojrian, Sanaz; Fazl, Fatemeh; Shirkharkolaie, Sahar Khanjani; Kabir, H M Dipu; Tan, Ru-San; Acharya, U Rajendra
title: Resource allocation optimization using artificial intelligence methods in various computing paradigms: A Review
date: 2022-03-23
journal: nan
DOI: nan
sha: b26a142b9beebf5ce71269a2e7c7ef71922d84f1
doc_id: 588458
cord_uid: y0sralzj

With the advent of smart devices, the demand for various computational paradigms such as the Internet of Things, fog, and cloud computing has increased. However, effective resource allocation remains challenging in these paradigms. This paper presents a comprehensive literature review on the application of artificial intelligence (AI) methods such as deep learning (DL) and machine learning (ML) for resource allocation optimization in computational paradigms. To the best of our knowledge, there are no existing reviews on AI-based resource allocation approaches in different computational paradigms. The reviewed ML-based approaches are categorized as supervised and reinforcement learning (RL). Moreover, DL-based approaches and their combination with RL are surveyed. The review ends with a discussion on open research directions and a conclusion.

. Conventional cloud computing model. Conventional cloud computing is limited by time delay, particularly with long-distance data transfer, which may degrade the quality of service. A new method, fog computing, has recently emerged that by mediating between the IoT and cloud levels, can facilitate data preprocessing and resource management, as well as shorten data transfer delay and reduce network traffic overhead. The computing environment can be conceived as a three-layered framework comprising IoT, fog calculations, and cloud computing [13] [14] [15] [16] (Figure 2 ), in which fog computing extends cloud services to the edge of the network nearer to the end-user to effect the reduction of data processing time and network traffic overhead. This can enhance service provision, especially for devices and applications requiring real-time interactions [17] . The basic entity in fog computing is the fog node, which executes the IoT application [18] . Any device with a network connection, computing, and storage can become a node, e.g., switches, routers, hubs, industrial controllers, surveillance cameras, etc. With the capacity for a large number of server nodes, fog computing offers one-step customer-server communication and real-time interactions with definable security, low jitter, and reduced time delay. Whereas fog calculations process information starting from where they have been generated to where they are stored, edge computing, a subset of fog computing, is concerned only with processing information close to where it has been created. IoT applications include many services requested by users of a system that must be responded to in real-time by the fog and cloud layers. Resource allocation on local area networks, as in edge computing, confers the advantages of short distances and shorter time delays for users close to the network edge [19] . AI methods such as supervised and reinforcement learning (RL), especially deep reinforcement learning methods (DRL), can be exploited to optimize resource allocation using various computing paradigms [20] [21] [22] . Machine learning (ML) and deep learning (DL) are increasingly exploited in cloud-based systems for resource management and virtualization. They do not require initial state transition and workload modeling. In particular, RL agents can learn to assign resources autonomously to run a cloud system [11] . We were motivated to perform an updated and comprehensive review on AI-enabled resource allocation in various smart computing environments. Many of the published reviews in the literature, which have been summarized in Table 1 , do not focus on AI and are limited to specific computing paradigms.

In contrast, a wide variety of computing paradigms have been considered in this paper, including cloud computing, vehicular fog computing, wireless network, IoT systems, vehicular network, 5G networks, machine-to-machine communication, train-to-train communication network, peer-topeer network, mobile cloud computing, cellular, and wireless IoT networks. From Table 1 , it is apparent that there are gaps in investigative research into problem issues such as high latency, high jitter, lack of location awareness, limited mobility support, and lack of support for real-time interactions. Of note, there is a secular trend for researchers to study resource allocation in nascent edge computing, fog computing, 5G mobile network, and wireless network environments. On account of their good performance, the adoption of ML and DL methods for automated decisionmaking with different computing paradigms has burgeoned. This study dissected many of these new computing paradigms to provide a comprehensive update on resource allocation issues in the contemporary computing landscape. Table 1 . Summary of articles related to resource allocation.

Cloud computing Examined the schemes based on cloud computing resources by using effective features, e.g., optimization goals, optimization methods, design approaches, and useful functions N/A Yousefzai [ 32 ] Edge computing Categorized reinforcement learning and heuristic learning methods for public safety communications on 5G networks ML Atman [24] Fog computing Examined categories of resource management: application placement, resource scheduling, task loading, load balancing, resource allocation, and resource provisioning for the computing environment; and approaches for resource allocation: auction, and optimization N/A Ghobaei-Arani [ 32 ] Cloud computing Energy efficiency for resource allocation problem N/A Hameed [ 32 ] Cloud computing Discussion on advancements achieved in energy-efficient computing N/A Beloglazov [ 32 ] Cloud computing Analyzed mechanisms to control and coordinate data center resources for energy-efficient operations N/A Shuja [ 32 ] Cloud computing Focused on resource monitoring in the cloud computing environment N/A Aceto [ 32 ] Cloud computing Developed conceptual framework for cloud resource management; recognized challenges of cloud: provision of predictable performance for cloud-hosted applications, achieving global manageability, scalable resource management, understanding economic behavior, and pricing N/A Jennings [ 23 ] Cloud computing Discussed implementation details of parallel processing frameworks, e.g., Google MapReduce and Microsoft's Dryad; focused on security issues in cloud systems N/A Goyal [ 23 ] Cloud computing Presented working process for commercial cloud computing service providers and opensource deployment solutions N/A Hussain [ 23 ] Cloud computing Examined dynamic resource allocation problem task scheduling strategies; examined operation mechanism of system with a SaaS-based cloud computing service under existing infrastructure N/A Huang [ 22 ] Cloud computing Studied virtual machine migration optimization features underlying cloud data center service operators N/A Ahmed [ 23 ] Cloud computing Studied virtual machine migration optimization features underlying cloud data center service operators N/A Ahmed [ 22 ] Cloud computing Analyzed the classification of strategy types and challenges related to resource allocation and their effects on cloud computing; focused specifically on CPU and memory resources N/A Vinothina [ 22 ] Cloud computing Examined resource allocation techniques in cloud computing; made a comparison between merits and demerits techniques, and their examined strategy consisted of prediction algorithms for resource requirements and resource allocation; identified efficient resource allocation strategies with effective use of limited resources N/A Anuradha [ 22 ] Cloud computing Examined resource management, i.e., resource allocation and monitoring strategies; examined problem-solving approaches of resource allocation in the cloud environment N/A Mohamaddiah [ 22 ] Cloud computing Provided strategies for resource allocation and their applications in the cloud; explained resource allocation in a cloud environment based on dynamic proportions N/A RamMohan [ 22 ] Wireless networks Provided a comprehensive overview of various methodologies to achieve common optimization tasks in the downlink of multi-user multiple-input communication systems N/A Casta˜neda [ 33 ] Cloud computing Examined resource management methods, e.g., resource provisioning, resource allocation, resource matching, and resource mapping; provided an overall overview of methods for IaaS in cloud computing N/A Manvi [ 33 ] 5G wireless networks Examined techniques and models of resource allocation algorithms in 5G network slicing; expressed ideas on software-defined networking and network function virtualization and their tasks in network slicing; presented the management and orchestration architecture of network slice N/A Su [ 33 ] Wireless, 5G, IoT, edge, fog, cloud, and vehicular fog computing Reviewing machine learning and deep learning methods for resource allocation in different computing paradigms

The contributions of this article are:

 Comprehensive literature review on ML-and DL-based methods for resource allocation problems in emerging computing environments.  Comparison of AI methods used to solve resource allocation problems.  Discussion of gaps and future research challenges in resource allocation in multilayer computing environments.

The rest of this article is organized as follows. In Section 2, the search strategy for the review is described. In Section 3, the findings of the literature review are presented. In Section 4, open research challenges and future works will be discussed. The conclusion is presented in Section 5.

We performed a literature search for publications up to 20 th January 2022 in Google Scholar using combinations of the following terms: "resource allocation", "efficient power consumption", "machine learning", "deep learning", "cloud computing", "edge computing", "fog computing", "Internet of Things", "wireless network", and "mobile edge computing". The logic that we adopted to combine these search terms using AND and OR operations is depicted in Figure 3 . The initial search yielded 460 papers, further reduced to 58 papers after excluding unsuitable papers. The exclusion criteria were non-English papers with low citation counts outside this review's scope. Five of the authors of this paper reviewed all the publications based on the aforementioned eligibility criteria, and only papers with at least three positive votes from authors were considered in this review. Additional publications were obtained by manual scrutiny of the references of the 58 selected papers. A final collection of 46 IEEE, 24 Springer, 16 Elsevier, and 23 miscellaneous (from other publishers) publications were included in the review. The taxonomy of the reviewed papers [43] is illustrated in Figure 4 . 

Shi et al. [44] applied the Markov decision process (MDP) and Bayesian learning to the study of optimal dynamic resource allocation in the cloud computing environment [45] . They found the former helpful for the allocation of cloud resources for components of network function virtualization, while the latter was predictive of future resource utilization based on historical usage patterns. Their proposed method outperformed greedy techniques, such as dynamic scaling, cost modeling, and virtual machine placement, regarding the total cost of cloud resource allocation. Rohmer et al. [21] proposed a learning-based resource allocation framework for P2P video-ondemand streaming. Using real data [46] , they proposed a Bayesian method for predictive analysis of popularity, which enabled dynamic switching between resource allocation strategies in P2P systems. This outperformed fixed strategies like lowest popularity score, lowest critical score, highest uplink first, and greedy in terms of mean rejection rate, maximum rejection rate, etc.

RL, a popular AI field, has been applied to diverse domains with good results. In a typical RL workflow ( Figure 6 ), at every time step , the agent observes the environmental state from state space . Depending on , the agent chooses action from the set of possible actions . After the action execution, the environment provides the agent with a reward +1 as well as the next state +1 . The probability of observing +1 given that action is executed at the state is represented by ( +1 , +1 | , ). Action selection is made using a policy function ( | ) which outputs the probability of choosing action given that the observed state is . The objective is to learn an optimal policy * such that the expected sum of discounted rewards is maximized. Atman and Nayan [24] applied RL and heuristic learning to adaptive resource allocation for 5G mobile networks for public safety communications. In [47] , an evolutionary genetic algorithmbased RL model enabled fast convergence to the global optimum through iterative genetic optimization, yielding superior results for long-term network use that exceeded the naïve strategy by over 90%. In the RL model in [48] , an adaptive Q-learning algorithm was used for approximate dynamic programming, which achieved fast convergence toward maximum revenue generation for the network owner. Gai and Qiu [4] used the quality of experience level metric combined with RL to devise the RL mapping table method for updating/maintaining the cost table and RL-based resource algorithm for achieving quality of experience level to realize the Smart Content-Centric for Internet-of-Things in a cyber-physical system. While the number of computational nodes significantly impacted training time, training time could be shortened by grouping computational nodes with similar capabilities together. AlQerm and Shihada [20] developed a participatory online learning algorithm with power and modulation adaptation capability in 5G systems that solved cross-tier and co-tier interference problems for optimal allocation, yielding significant improvements in throughput and spectral efficiency, fairness, and outage ratio for different underlay edge transitions. It outperformed downlink spectrum allocation [49] , joint resource allocation and link adaptation algorithm [50] , and matching resource management schemes [51] . Hussain et al. [52] used the Q-learning algorithm for slot assignment in the machine-to-machine communication network and the k-means clustering algorithm [53] to overcome congestion. The experiments showed that Q-learning increased the probability of slot assignment by more than five times compared to Ethernet slot assignment protocols such as ALOHA and slotted ALOHA.

Hamidreza Arkian et al. [54] proposed the cluster-based vehicular cloud architecture with learningbased resource management (COHORT) architecture, demonstrating increased efficiency, stability, and reliability. An improved COHORT clustering plan, in which fuzzy logic was used for the eclipse selection [55] , was compared with two plans based on user-oriented fuzzy logic [56] and lowest-ID clustering schemes [57] . The experiments showed that by increasing the maximum speed from 60 to 120 km/h, the cluster head duration for the COHORT scheme decreased by 15% versus much lower decrements for both the lowest-ID and user-oriented fuzzy logic schemes. Compared with the disCoveRing and cOnsuming services WithiN vehicular clouds (CROWN) architecture [58] , COHORT exhibited fewer service discovery and consumption delays as the number of vehicles increased. Salahuddin et al. [19] compared MDP and greedy heuristics [59] for minimizing overhead in the vehicular cloud environment. MDP resulted in better longterm benefits and lower overhead for resource provision.

Moreover, between the MDP and myopic heuristic methods, the MDP method has a lower overhead for the same configuration selection. The MDP and greed heuristic methods might lead to the same configuration choice. Hence, MDP should perform the myopic heuristic method even in the worst-case scenario. Table 2 summarizes ML-based (supervised learning and RL) studies on resource allocation in various computing paradigms, many of which involved cloud computing [60] . For example, Shi et al. proposed a 1-step MDP for the dynamic allocation of cloud resources. As a result, their model had a slower response time (time taken to find resource allocation solution) than the genetic algorithm. But when resources must be allocated at specific deadlines, the 1-step optimization failed to foresee the future, which resulted in MDP incurring higher costs versus genetic algorithm. [63] Noticeable reduction in response time and internet traffic compared to cloud-based and fog-based approaches N/A Fog cloud computing-IoT AI-based task distribution algorithm Abedi [64] Efficient resource allocation before and after contingencies using multiple trained models; approach evaluated by the real-world interdependent infrastructure of Shelby County, Tennessee N/A Resource allocation for infrastructure resilience Artificial neural networks Alemzadeh [65] Tackled resource allocation in wireless networks using whale optimization algorithm; applied WOA to power allocation for secure throughput maximization, mobile edge computing offloading, resource allocation in 5G wireless networks, etc.

Whale optimization algorithm Pham [66] Minimization of the maximal computational and transmission delay for users requesting computational tasks; used RL to learn resource allocation policy based on users' computational tasks; reduced the maximal delay up to 18% among all users and up to 11.1% compared to the standard Q-learning algorithm N/A Wireless networks based on mobile edge computing Modified Q-learning Wang [67] Suitable gate allocation to airport flights within different time intervals; method evaluated on the actual data from Baiyun Airport; reduced airport management costs N/A Airport flight management data Improved quantum evolutionary algorithm based on niche co-evolution strategy and enhanced particle swarm optimization Deng [68] Eliminated the need to search in a high-dimensional space for service placement decisions; computational complexity linear growth in the number of users; scalable to large networks; achieved near-optimal performance in simulation N/A Mobile edge computing Alternating direction method of multipliers Lin [69] Handled unlimited incoming requests in a parallel and distributed manner while ensuring the quality of service; achieved lower (0.5 ms) average turnaround compared to ant colony optimization N/A Cloud computing Integrated artificial neural network-genetic algorithm Geetha [70] Offered wireless edge service for training/inference of machine learning tasks while considering limitations of edge servers; aims were energy consumption minimization while considering end-toend service delay and accuracy, learning accuracy optimization, and ensuring end-to-end delay and bounded average energy consumption N/A Edge machine learning Stochastic Lyapunov optimization Merluzzi [71] Reduced data drops and service delays to maximize data delivery MATLAB Mobile edge computing Blockchain-assisted data offloading for availability maximization, naïve Bayes Manogaran [72] Improved resource utilization and user quality of experience level with system quality of service guarantee; outperformed traditional greedy algorithm N/A Industrial IoT Hierarchical RL, semi-MDP Liang [73] Proactive, dynamic resource allocation scheme; resource deployment for upcoming traffic data processing; evaluated on real-world/artificial datasets; outperformed greedy algorithm MATLAB Open radio access network Evolutionary optimization algorithm Ruan [74] Combined hierarchical RL with meta-learning; significant resource management improvement in dynamic vehicular networks by adapting to different scenarios quickly N/A Dynamic vehicular networks

He [75] Determined optimal lockdown resource allocation strategies for Arizona, California, Nevada, and Utah in the United States; more flexible resource allocation strategies helpful for wise allocation of limited resources to prevent infection

Multi-agent recurrent attention actor-critic, a DRL method Zong [76] MDP, Markov decision process; RL, reinforcement learning

Some of the applications of DL methods in resource allocation are illustrated in Figure 7 . 

Karthiban and Raj [10] used a DRL algorithm based on Q-learning for fair resource allocation in cloud computing environments. The proposed approach outperformed first-in, first-out, and greedy methods in terms of the average response time and average waiting time, even with increasing requests while guaranteeing the quality of service. Excessive power consumption in cloud computing systems reduces system reliability and increases cooling costs. Therefore, balancing power consumption and performance is an important design factor in cloud computing platforms. Liu et al. [11] proposed a joint virtual machine resource allocation and power management framework consisting of a global tier that used deep Q learning to allocate virtual machine resources to servers and a local tier for distributed power management in local servers. A self-cryptographic neural network and weight-sharing scheme were employed to accelerate convergence speed and control the high-dimension mode space. Experiments were implemented using the methods mentioned in actual Google cluster-usage traces [77] . The proposed hierarchical framework was observed to optimize power/energy consumption significantly better than the base round-robin method without significant difference in terms of delay. Wang et al. [ 22 ] used deep Q learning to propose in their DRL resource allocation method for smart resource allocation in mobile computing. The method was designed to minimize the expected service time of requests made by mobile devices distributed in different districts. Additionally, the computing load on each mobile edge computing server and network load on data links were balanced in order to achieve a better quality of service. The proposed method improved the average service time as the request aggregation district numbers increased compared to the open shortest path first method [ 22 ] .

Chen et al. [80] proposed a novel fog resource scheduling scheme based on the minimization of perception-reaction time. Perception-reaction time represents the time consumption of safetyrelated applications and is closely related to road security and efficiency. Due to the intractability of the formulated optimization problem, DQN was used in [81] to reduce overall delay in the fog computing environment for vehicular applications in the information-centric network Internet of Vehicles. DQN conferred better performance than Q learning, location greedy, and resource greedy algorithms. Vehicular fog computing combined with perception-reaction time criterion is more stabler than architectures such as no fog and no information-centric network. Ye et al. [82] focused on decentralized resource allocation in vehicle-to-vehicle communications for unicast and multicast vehicle communications. Their experiments revealed the superior performance of DQN for resource allocation in vehicle-to-vehicle communications and higher capacity in vehicle-toinfrastructure compared to the random method and dynamic proximity aware resource allocation [83] .

DRL can be used for resource allocation in vehicular networks, including methods that encompass observer, objective-oriented unsupervised learning paradigm, and learning accelerated optimization paradigm were examined (Figure 8 ). Each V2V agent observes the environment and then utilizes its local copy of the trained DRL agent to monitor the resource block selection and power control in a distributed way. Liang et al. [22] used DRL to solve wireless resource allocation problems in the vehicular wireless network environment. Deep deterministic policy gradient yielded the best results among the evaluated RL methods. They reported that their method outperformed weighted minimum mean-squared error [84] , significantly reducing computational complexity for the non-deterministic polynomial hard power resource allocation problem [85] . The feed-forward network and convolutional neural network [86] methods in the linear sum assignment programming problem could be used as a real-time solution. The performance of the two unsupervised methods in [87] using DNN was better than the heuristic weighted minimum mean-squared error. Zhao et al. [88] used distributed DRL for computing resource management, resource allocation, and system complexity reduction in vehicular fog computing environments. They proposed a contract-based incentive mechanism for resource allocation in the vehicular fog network. As the number of vehicles increased, the proposed mechanism incentivized more vehicle participation, improving quality, efficiency, and maintenance. This contrasts with the conventional offloading mechanism in which the computational load of non-cooperative vehicles is returned to the roadside unit, leading to an increase in pressure. Zhao et al. [89] used multi-agent DRL to reduce co-channel interference, prevent collisions, and increase system power in their proposed smart resource allocation method in train-to-train communications. The multi-agent deep Q-network method addressed the train-to-train resource allocation problem, which yielded successful data transfer and improvements in train-to-train connection throughput and overall system throughput. The proposed design was compared with the train-to-train communication resource allocation [90] and the random allocation scheme. In [90] , Stackelberg's game theory for power control and weight factors based on proportional fairness standard was proposed for channel selection to address resource allocation in train-to-train communications.

The ability to deal with continuous (as well as discrete) data and high dimensional problems has made DL the de facto standard in many learning problems [91] . DL methods are usually trained in a supervised manner using labeled training data [92] , which in practice, are usually available. It is advantageous to combine DL and RL, i.e., DRL, for resource allocation problems to exploit their strengths and attenuate their weaknesses [93] . Table 3 summarizes the results of the reviewed DLbased methods, which have been applied to various computing paradigms like cloud computing, vehicular fog computing, vehicle-to-vehicle communications, mobile edge computing, etc. DRL methods are commonly used, and DRL-based resource allocation yields good performance in cochannel interference, system and energy efficiency, latency, response time, and complexity. [10] Reduced perception-reaction time; lower average delay for non-safety N/A Vehicular fog computing DRL Chen [80] Increased vehicle-to-infrastructure capacity; optimal vehicle-to-vehicle latency N/A Vehicle-to-vehicle communications DRL Ye [82] Low energy usage, power/energy savings up by 16.12%; reduced latency by16.67%, N/A Cloud computing DRL-LSTM Liu [11] Accuracy: Hungarian algorithm (100%), CNN (92.76%); classifier accuracy: graph embedding for 1500 training samples (83.88%) N/A Vehicular networks DRL Liang [22] Reduced system complexity; improved computing power and entire system performance Python Vehicle fog computing Distributed DRL + Adam optimizer Zhao [88] Improved throughput of the train-to-train link; reduced co-channel interference in the system effectively N/A Train-to-train Multi-agent DRL Zhao [89] Increased minibatch size leading to faster convergence of DRL resource allocation algorithm Python Mobile edge computing DRL resource allocation Wang [ 22 ] Hierarchical resource management framework for network slicing to offer diversified services; outperformed greedy resource management Python + Tensorflow 6G wireless networks DQN Guan [94] CNN-based power allocation in industrial IoT applications; less network residual energy vs. IEEE 802.11; even distribution of power resources N/A Secure industrial IoT network CNN Goswami [95] Comprised an improved cat swarm optimization algorithm-based short scheduler for task scheduling that minimized make-span time minimization and maximized throughput maximization; a group optimization-based DNN for efficient resource allocation given bandwidth and resource load constraints; and a lightweight authentication scheme named NSUPREME; outperformed first come, first served and round-robin approaches in resource utilization, energy consumption, and response time N/A Cloud computing Resource allocation with task scheduling using hybrid machine learning Bal [96] Handled complex optimization in fog computing-based radio access; predicted channel quality change using LSTM; achieved higher guaranteed quality of experience in terms of high average bit rate, low rebuffering ratio, and average bit rate variance Python Fog computingbased radio access AI-aided joint bit rate selection and radio resource allocation Chen [97] LSTM-based traffic forecasting algorithm for resource allocation in network function virtualization; applied different weightings for overprovisioning and under-provisioning; reduced cost by 40% compared to methods based on symmetric cost minimization of prediction error N/A Cloud resource allocation LSTM Eramo [98] Variant of multiobjective evolutionary algorithm based on decomposition (MOEA/D-DU) was combined with ensemble fitness ranking with ranking restriction scheme to achieve better balance between the convergence and diversity in multiobjective optimization; outperformed state-of-the-art methods on test suite problems Python Wireless communications systems DNN Lee [99] Two-level resource allocation and incentive mechanism design that relied on evolutionary game theory to model cluster selection process dynamics at a lower level; DL-based auction mechanism for evaluation of clusters heads' services; achieved unique and stable evolutionary game as well as revenue maximization for cluster services N/A Edge computing DL Lim [100] Dynamically allocated DNN inference computation to multiple vehicles using edge server; allocation optimized using chemical reaction optimization; achieved lower overall latency and failure rate compared to competing schemes: edge, local, and neurosurgeon Python Integrated mobile edge computing and vehicular edge computing DNN Wang [101] Novel power migration expand resource allocation and allocation requests to servers with EESA; 26% less energy consumption of mobile edge server, improved service rate by 23%, compared with other algorithms; 70% EESA accuracy for allocating the resources of multiple servers to multiple users N/A Mobile edge computing DL: power migration expand + EESA Ali [102] Enforced security and privacy between IoT devices and edge computing nodes by combining blockchain and DRL, i.e., A3C; used A3C to allocate resources; evaluated method on simulation with three data service subscribers and three edge computing nodes He [104] Used cooperative caching with DQN [105] , a DRL-based resource allocation approach, to transmit contents with low latency; evaluated method in a layered fog radio access network; less average network delay compared with cooperative caching with popularity [106] , distributed caching with least recently used [107] , and no-cache N/A Layered fog radio access network DQN Fang [108] Used LSTM in an integrated resource prediction/allocation approach comprising monitoring agent, prediction/allocation agent, and reconfiguration and placement agent; was superior to methods that performed resource prediction and allocation processes separately 

Resource allocation is usually formulated as an optimization problem. Accordingly, it is susceptible to intractable optimization problems [110] , in which problem constraints would have to be relaxed at the cost of risking sub-optimal solutions. ML/DL methods have the potential to come up with acceptable near-optimal solutions in reasonable amounts of time for challenging optimization problems. To this end, researchers' interest in interdisciplinary approaches has increased, and various AI methods (ML/DL-based) are under investigation for resource allocation [111] . In this work, we have comprehensively reviewed the findings of studies of AI approaches to the resource allocation problems in diverse computing paradigms and have discussed the successes and shortcomings. Future work is still needed to develop new methods capable of handling resource allocation with reasonable computational complexity and performance. This comprehensive review will constitute a valuable reference for researchers in the field.

The performance of AI methods is dependent on the availability and quality of training data. There is a huge problem of noisy and unlabeled data in heterogeneous platforms such as IoT, mobile edge computing, etc. Many AI methods rely on supervised training, which requires labeled training data with good quality. Preparing such data may be difficult and time-consuming, which poses a limitation to applying AI methods. For example, one fast-growing use case is the Internet of Medical Things, which concerns remote healthcare services. Researchers' ultimate goal is to develop AI-based healthcare systems that dispense with the need for human intervention. However, there is extremely low tolerance for erroneous decisions in safety-critical domains such as healthcare. Therefore, high-quality labeled training samples are mandatory for training AI models. Such data are challenging to obtain since the labeling process is typically carried out fully manually by medical experts.

Another factor affecting the performance of complex AI models is hyperparameter tuning. For a DL model to be trained for a specific task, apart from model parameters that will have to be adjusted during the training process, model hyperparameters such as the number of hidden layers, learning rate, regularization coefficient, etc. are critical to successful training. Hyperparameter tuning requires searching a complex multidimensional space, which is onerous and can be confusing. Hyperparameter optimization tools like Wandb [112], Comet [113], etc., may facilitate the process by keeping track of the conducted optimization experiments. As an illustration, Figure Figure 9 . Sample output of hyperparameter optimization using Comet tool. Each sinusoid curve corresponds to one training experiment. The color scale on the right side represents the validation F1-score, with warmer and cooler colors corresponding to higher and lower scores, respectively.

Another important challenge regarding applying AI methods in different computing paradigms is the ability to make the models contextual. Existing literature on AI methods and DL is mostly about designing and training a model specialized at doing a single task [111] . However, in computing paradigms such as IoT, the tasks may dynamically change, and training DL models to adapt to new changes will impose high computation costs, which is impractical. Existing works have utilized conditional generative adversarial networks to interpolate between different 3D objects [114] or between different ages of a human face [115, 116] . Following the footsteps of these studies, it might be possible to train generative adversarial networks to output appropriate parameters set for another DNN based on a set of input conditions. This way, the training effort is devoted to the generative adversarial network, and no further training will be required when running the application.

Effective resource allocation approaches must be able to withstand unforeseen resource shortcomings. For instance, should a cloud computing server temporarily loses some of its computation resources due to a cyber-attack, it will be necessary to change the resource allocation priorities. Tasks with higher priorities will be granted access to the available resources. Dynamically changing the resource allocation strategy is indirectly related to the contextual models mentioned in the previous paragraph.

Communication between different systems, including smart devices, storage, servers, communication networks, is an undeniable part of daily life. Optimizing and increasing the efficiency of this communication is an important consideration, and resource allocation is a critical bottleneck. Researchers are using innovative AI methods to optimize resource allocation according to the data flow during network operation to solve the challenge of resource allocation. These measures have moved the industry towards automated resource management on a large and complex scale. This article has reviewed various AI methods used to solve the resource allocation problem in different computing environments and summarized the performance in terms of response time, energy efficiency, throughput, cost, service consuming delay, convergence time, latency, etc. New resource allocation methods are continually being developed, and the computing environments have shifted from cloud to fog and edge. While resource allocation in the cloud environment has been the subject of much research in the last decade, recent attention has focused on resource allocation at the level of smart devices.

An uncertain multi-objective programming model for machine scheduling problem

A survey on FinTech

Analyzing the cyber-physical impact of cyber events on the power grid

Optimal resource allocation using reinforcement learning for IoT content-centric services

Cyber physical computing for IoT-driven services

Application of artificial intelligence in intelligent decision-making of human resource allocation

A hierarchical edge cloud architecture for mobile computing

Its-cloud: Cloud computing for intelligent transportation system

VANET-cloud: a generic cloud computing model for vehicular Ad Hoc networks

An efficient green computing fair resource allocation in cloud computing using modified deep reinforcement learning algorithm

A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning

An intrusion detection and prevention system in cloud computing: A systematic review

Dynamic resource allocation for load balancing in fog environment

Game theory and evolutionary optimization approaches applied to resource allocation problems in computing environments: A survey

A systematic mapping study of cloud policy languages and programming models

Wireless Transmissions, Propagation and Channel Modelling for IoT Technologies: Applications and Challenges

The fog computing paradigm: Scenarios and security issues

Hybrid energy aware clustered protocol for IoT heterogeneous network

Reinforcement learning for resource provisioning in the vehicular cloud

A cooperative online learning scheme for resource allocation in 5G systems

A learning-based resource allocation approach for P2P streaming systems

Deep-learning-based wireless resource allocation with application to vehicular networks

Cloud resource allocation schemes: review, taxonomy, and opportunities

Efficient admission control and resource allocation mechanisms for public safety communications over 5G network slice

Resource management approaches in fog computing: A comprehensive review

A survey and taxonomy on energy efficient resource allocation techniques for cloud computing systems

A taxonomy and survey of energy-efficient data centers and cloud computing systems

Survey of techniques and architectures for designing energy-efficient data centers

Cloud monitoring: A survey

Resource management in clouds: Survey and research challenges

University of British Columbia Technical Report for CS

A survey on resource allocation in high performance distributed computing systems

Survey on Resource Allocation Policy and Job Scheduling Algorithms of Cloud Computing1

Virtual machine migration in cloud data centers: a review, taxonomy, and open research issues

A survey on virtual machine migration and server consolidation frameworks for cloud data centers

A survey on resource allocation strategies in cloud computing

A survey on resource allocation strategies in cloud computing

A survey on resource allocation and monitoring in cloud computing

Resource Allocation Techniques in Cloud Computing--Research Challenges for Applications

An overview on resource allocation techniques for multi-user MIMO systems

Resource management for Infrastructure as a Service (IaaS) in cloud computing: A survey

Resource allocation for network slicing in 5G telecommunication networks: A survey of principles and models

Application of Photoplethysmography signals for Healthcare systems: An in-depth review

MDP and machine learning-based cost-optimization of dynamic resource allocation for network function virtualization

A survey of network virtualization

Statistics and social network of youtube videos

Slice as an evolutionary service: Genetic optimization for inter-slice resource management in 5G networks

Optimising 5G infrastructure markets: The business of network slicing

Downlink spectrum allocation in 5g hetnets

A joint resource allocation and link adaptation algorithm with carrier aggregation for 5G LTE-Advanced network

Resource management in dense heterogeneous networks

Resource allocation and congestion control in clustered M2M communication using Q-learning

Information Theoretic Weighted Fuzzy Clustering Ensemble

A cluster-based vehicular cloud architecture with learning-based resource management

FcVcA: A fuzzy clustering-based vehicular cloud architecture

User-oriented fuzzy logic-based clustering scheme for vehicular ad-hoc networks

Multicluster, mobile, multimedia radio network

Finding a STAR in a Vehicular Cloud

Software-defined networking for rsu clouds in support of the internet of vehicles

Adaptation, learning, and optimization

Urban water resource management for sustainable environment planning using artificial intelligence techniques

Artificial intelligence-based Kubernetes container for scheduling nodes of energy composition

Machine Learning-Based Resource Allocation in Satellite Networks Supporting Internet of Remote Things

Resource allocation in combined fog-cloud scenarios by using artificial intelligence

Resource Allocation for Infrastructure Resilience using Artificial Neural Networks

Whale optimization algorithm with applications to resource allocation in wireless networks

A machine learning approach for task and resource allocation in mobile-edge computing-based networks

A Novel Gate Resource Allocation Method Using Improved PSO-Based QEA

Optimizing AI service placement and resource allocation in mobile edge intelligence systems

An advanced artificial intelligence technique for resource allocation by investigating and scheduling parallel-distributed request/response handling

Wireless edge machine learning: Resource allocation and tradeoffs

Artificial intelligence and blockchain-assisted offloading approach for data availability maximization in edge nodes

Reinforcement learning enabled dynamic resource allocation in the Internet of vehicles

Evolutionary Optimization for Proactive and Dynamic Computing Resource Allocation in Open Radio Access Network

Meta-Hierarchical Reinforcement Learning (MHRL)-based Dynamic Resource Allocation for Dynamic Vehicular Networks

Reinforcement learning based framework for COVID-19 resource allocation

Google cluster-usage traces: format+ schema

Smart resource allocation for mobile edge computing: A deep reinforcement learning approach

Divide and conquer: Partitioning OSPF networks with SDN

A machine-learning based time constrained resource allocation scheme for vehicular fog computing

Playing atari with deep reinforcement learning

Deep reinforcement learning based resource allocation for V2V communications

Dynamic proximity-aware resource allocation in vehicle-tovehicle (V2V) communications

Multi-agent deep reinforcement learning for dynamic power allocation in wireless networks

Learning to optimize: Training deep neural networks for interference management

Deep power control: Transmit power control scheme based on convolutional neural network

An iteratively weighted MMSE approach to distributed sumutility maximization for a MIMO interfering broadcast channel

Contract-Based Computing Resource Management via Deep Reinforcement Learning in Vehicular Fog Computing

Intelligent Resource Allocation for Train-to-Train Communication: A Multi-Agent Deep Reinforcement Learning Approach

Train-to-train communication resource allocation scheme for train control system

Uncertainty-Aware Semi-Supervised Method Using Large Unlabeled and Limited Labeled COVID-19 Data

Combining a convolutional neural network with autoencoders to predict the survival chance of COVID-19 patients

Fusion of convolution neural network, support vector machine and Sobel filter for accurate detection of COVID-19 patients using X-ray images

Customized slicing for 6G: Enforcing artificial intelligence on resource management

A Neural-Network-Based Optimal Resource Allocation Method for Secure IIoT Network

A Joint Resource Allocation, Security with Efficient Task Scheduling in Cloud Computing Using Hybrid Machine Learning Techniques

Artificial intelligence aided joint bit rate selection and radio resource allocation for adaptive video streaming over F-RANs

Proposal and Investigation of an Artificial Intelligence (AI)-Based Cloud Resource Allocation Algorithm in Network Function Virtualization Architectures

Intelligent resource allocation in wireless communications systems

Decentralized edge intelligence: A dynamic resource allocation framework for hierarchical federated learning

Dynamic resource allocation for jointing vehicle-edge deep neural network inference

A deep learning approach for mobilityaware and energy-efficient resource allocation in MEC

Asynchronous methods for deep reinforcement learning

Blockchain-based edge computing resource allocation in IoT: a deep reinforcement learning approach

Deep Q-Learning-Based Cooperative Caching Strategy for Fog Radio Access Networks

Cache in the air: Exploiting content caching and delivery techniques for 5G systems

Performance analysis of in-network caching for content-centric networking

Deep Reinforcement Learning Based Resource Allocation for Content Distribution in Fog Radio Access Networks

Application of an Innovative Convolutional/LSTM Neural Network for Computing Resource Allocation in NFV Network Architectures

Dynamic Resource Prediction and Allocation in C-RAN With Edge Artificial Intelligence

Machine learning for resource management in cellular and IoT networks: Potentials, current solutions, and open challenges

Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling

Face aging with identity-preserved conditional generative adversarial networks

Face aging with contextual generative adversarial nets