key: cord-0434861-55w52b4f authors: Okunevich, Iaroslav; Trinitatova, Daria; Kopanev, Pavel; Tsetserukou, Dzmitry title: DeltaCharger: Charging Robot with Inverted Delta Mechanism and CNN-driven High Fidelity Tactile Perception for Precise 3D Positioning date: 2021-07-22 journal: nan DOI: nan sha: 2baf53aad85f54872e56ef8fb643fc790c915679 doc_id: 434861 cord_uid: 55w52b4f DeltaCharger is a novel charging robot with an Inverted Delta structure for 3D positioning of electrodes to achieve robust and safe transferring energy between two mobile robots. The embedded high-fidelity tactile sensors allow to estimate the angular, vertical and horizontal misalignments between electrodes on the charger mechanism and electrodes on the target robot using pressure data on the contact surfaces. This is crucial for preventing a short circuit. In this paper, the mechanism of the developed prototype and evaluation study of different machine learning models for misalignment prediction are presented. The experimental results showed that the proposed system can measure the angle, vertical and horizontal values of misalignment from pressure data with an accuracy of 95.46%, 98.2%, and 86.9%, respectively, using a Convolutional Neural Network (CNN). DeltaCharger can potentially bring a new level of charging systems and improve the prevalence of mobile autonomous robots. N OWADAYS, the number of mobile robots is constantly increasing. They are applied in different areas such as logistics [1] , [2] , stocktaking [3] , assembling furniture [4] , education [5] , and etc. In the face of COVID-19, a lot of companies and research groups have created autonomous mobile robots for disinfection [6] , [7] . However, all mobile robots have a limited operation time [8] , which defines the range of mobile robots and depends on their energy capacity. This factor is not significant for indoor robots because increasing working space can be compensated by a larger number of docking stations. On the other hand, for outdoor robots that explore an unknown area (e.g., coal mine rescue robots [9] ), the operation time problem is crucial. There are many studies on increasing the working range of mobile robots. One of the most effective approaches is to apply renewable energy as a power source of the robot, such as solar energy [10] . This makes it possible to have an unlimited working range of mobile robots. However, there are disadvantages, such as sensitivity to external conditions and low charging speed. The most common way to boost the range of a mobile robot is to increase the battery capacity. However, in this case the size and weight of a mobile robot are also increasing dramatically. An alternative approach could be taken from the aviation and space industries. In order to increase the range of a plane, additional fuel tanks or refueling from a tanker are used. Similarly, a new mobile robot, which would be simpler and cheaper than the original one, could recharge the main robot and increase its range. The idea of autonomous charging systems has been applied in some prototypes. The common approach is autonomous docking of the mobile robots with the charging station [11] . Another approach is the development of auto-charging robots [12] , [13] . One of such charging systems is applied for the underwater robot [14] . The most advanced technology is using a mobile autonomous system for charging. The target object for the charging could be mobile devices [15] or a swarm of robots [16] . Some concepts and prototypes of mobile robots for recharging larger systems exist, however, they are applied for working with electric vehicles in public places [17] . The most famous are "Laderoboter" 1 by Volkswagen with a robotic arm (Universal Robot) as actuator and "EVAR" 2 by Samsung spin-off, both are fully autonomous. Wireless [18] or conductive types of charging implement the energy transmission in the charging system. Advantages of wireless power transmission include eliminating the electrical contact wear and providing isolation from the environment [19] . However, the conductive charging system is more suitable for charging mobile robots, since powerful DC charging system allows to achieve higher power transmission performance than the wireless system [20] . The actuator of the charging system has to dock with the target robot. The size of the actuator defines the size of the whole autonomous charging robot. Thus, it has to be compact and should be placed inside the mobile platform. Applying a robotic arm as an actuator helps to dock with the main robot at a distance of about 0.2-1.2 meters. However, it will significantly increase the cost, size, weight, and power supply of the charging robot. Another solution is using a Cartesian manipulator as it is implemented in "EVAR". However, it requires a lot of space and has a complicated structure for manufacturing. The Delta mechanism is more convenient to manufacture than the Cartesian mechanism. On the other hand, it requires more space inside the robot. An Inverted Delta actuator developed by us is more compact than the traditional Delta actuator and can be easily integrated inside the robot body. The main requirement for charging is a reliable and safe connection between electrodes. The angular misalignment between the electrodes on the charger and the target robot leads to a short circuit. The horizontal or vertical misalignment leads to a decrease in the contact area of the electrodes, which can cause damage to them. The same problem is solved by misalignment-sensing coils in inductive power transfer systems [21] . The tactile perception system can potentially solve the misalignment issue in the conductive case. In the current work, we propose DeltaCharger, an Inverted Delta actuator for autonomous charging of the mobile robot, which can provide a robust connection between the electrodes through the use of high-fidelity tactile sensors. The prototype of the DeltaCharger is shown in Fig. 1 . The design of DeltaCharger was inspired by DeltaTouch haptic interface [22] . The DeltaCharger mechanism consists of a setup ring with motors and a moving platform with two electrodes (end effector). The setup ring is mounted on a mobile charging robot using three wings. The mechanism is actuated by three servomotors (Dynamixel MX64). The Inverted Delta mechanism comprises three interchangeable kinematic limbs that allow the moving of the end effector as shown in Fig. 2 . The actuator is manufactured from polylactic acid (PLA) material and hollow aluminum tubes. The end effector consists of a platform with two electrodes, six magnets, and two tactile sensors. Electrodes are made of duralumin for greater strength. They withstand a current of 18 A. Magnets keep the charger's electrodes in constant contact with a discharged robot. Tactile sensors are applied to detect a misalignment between the electrodes on the end The system configuration of DeltaCharger with 3 blocks is shown in Fig. 3 . The first block, i.e., Computing Unit, includes Intel NUC computer and OpenCM 9.04 Dynamixel Controller. The communication between the Intel NUC and the microcontroller was implemented through UART. A computer solves inverse kinematics problem for Inverted Delta robot and analyzes the data from sensors. The second block is the Inverted Delta Mechanism. It is an actuator, which has a setup ring, servomotors, and end effector. OpenCM board controls servomotors through UART. Additionally, the controller collects data from tactile sensors by means of an analog interface. The power supply voltage for tactile sensors equals 5 VDC. Power Supply block consists of a battery and two DC-DC regulators. The output voltage of the battery equals 22.7-29.8 VDC. Servomotors require 12 VDC and Intel NUC requires 19 VDC. DC-DC regulators are applied for reaching these voltages. Electrodes in the end effector are electrified by the battery to transfer energy. Mobile robots often perform their tasks on uneven surfaces. This can cause problems during the charging process because electrodes on the end effector of the charger must have the same orientation as electrodes on the target robot. If the angle between these electrodes is too large, the process of charging will not be effective. In a normal case, each electrode of DeltaCharger contacts with one electrode on the main robot, whereas in a tilt case, each electrode can contact with two electrodes, leading to a short circuit (Fig. 4) . To solve this problem, one can measure the angle of misalignment between electrodes. If the angle is lower than a critical angle, charging can take place. To measure the angular misalignment, the system was equipped with tactile sensor arrays [23] . The end effector with tactile sensors is shown in Fig. 5 . Misalignment angle is the rotational angle around the Z-axis in the coordinate system of the actuator. Each array is capable of sensing the maximum frame area of 5.8 cm 2 with a resolution of 100 points per frame. The sensing frequency equals 120 Hz (frames per second). With these sensors, the system can precisely detect the force applied to the edges of the charger's terminal. The range of force detection is of 1-9 N . The high resolution of the sensors allows to make a correlation between the obtained force diagrams and different misalignment angles. As a result, the prediction system can be developed to detect the angles of misalignment by contact patterns. In addition, the docking precision of end effector is not high. The coordinates of the required position for the end effector are received from the computer vision (CV) system. It includes an RGB-D camera and detects the electrodes on the main robot. After that, it returns the coordinates of the electrodes in the actuator coordinate system. The disadvantage of this method is the working range for depth camera. The CV system cannot detect the coordinates of the docking point with high precision in proximity to the target robot. In this case, the electrodes may not touch, or the contact will not be over the entire area, leading to rapid wear of the electrodes. We propose to navigate the end effector after docking by a tactile system that is used to predict the misalignment angle. The horizontal and vertical misalignments would have different contact patterns, which depend on the value of misalignment. Machine Learning (ML) methods could be applied to predict the value of the end effector shifting to correct its position. DeltaCharger will be part of a mobile charging robot with the following operating strategy. The mobile charger will wait for the message with coordinates of the discharged robot from the robot itself or from the base station. After that, it will move to the area of obtained coordinates and will start to search for the electrodes by the RGB-D camera-based CV system. If the charging robot cannot find the electrodes, it should return to the base station. If the electrodes are recognized, the charger moves to the electrodes and docks with them according to Algorithm 1. The adjustment of the end effector position along the Zaxis is carried out by measuring the value of the current consumed from the motors. The position of the electrodes calculated by the CV system can be farther than the real position of the electrodes. In this case, the end effector leans on the electrodes with a high torque value, and the consuming current reaches a value of several A. The current over 0.5 A is too high for the continuous work of the servomotors, which will lead to overheating after a couple of minutes of operation. The prediction of the end effector position error by consuming current is not sensitive enough. A reliable method is to move the end effector back a few cm and move it forward with small steps. If the consuming current is higher than 0.4 A and lower than 0.5 A, the end effector rests on the electrodes, and servomotors can hold the end effector for 4 hours without overheating. The tactile sensors are not applied to detect the error of end effector position along the Z-axis since the difference between the obtained pressure data for Algorithm 1: Algorithm of safe docking with the target robot. receive the coordinates of electrodes from the visual system; move the end effector to the received position; if the consuming current of servomotors is more than 0.5 A then decrease the Z-coordinate of the end effector on 3 cm while current consumption of servomotors is less than 0.4 A do increment the Z-coordinate of the required position on 5 mm measure the misalignment angle; if the misalignment angle is less than the critical angle then while the horizontal and vertical misalignments are not equal to zero do detect the value of misalignments; move the end effector to align it; start charging; else return the end effector to the initial position; charging is failed; high and normal consuming current is not large. Thus, it is not possible to predict a case when the servomotors are close to overheating. After the end effector is aligned along Z-axis, the misalignment angle is measured by the tactile system. The end effector will be equipped with a rotational mechanism to align the end effector with the electrodes in future work. In the current work, if the misalignment angle is more than the critical one, the charging process will be stopped. The last step of the algorithm is the end effector alignment along the X and Y axes. The tactile system detects the values of vertical and horizontal misalignments. After that, the end effector moves to decrease the misalignment value. When the misalignment values equal zero or the alignment operation has been repeated more than 15 times, the end effector alignment process is over, and the charging process is started. The limitation of 15 loop executions is needed to prevent an infinite alignment cycle. In order to estimate the angular, vertical, and horizontal misalignments, a classification problem was solved. Our first step was to find the roll angle constraints and determine the possible classes (misalignment angle ranges). Outdoor robots can tilt more than 30 • because they have to cross the rough terrain, while for indoor robots angle constraints are different. To determine the maximum possible roll angle of the robot, we have conducted the experiment with an indoor robot. In our case, the robot had a maximum tilt angle of 5 • . The critical angle for the shape of electrodes is 12 • . It is not enough for charging very tilted outdoor robots. However, it is sufficient for a target indoor robot. Therefore, 6 classes of angular misalignment were defined with a range of 1 • . Furthermore, predicting vertical and horizontal misalignments has been identified as a classification problem. Ideally, the actuator can rotate the end effector so that the misalignment angle would be 0 • after the successful detection. In this case, the prediction of the position misalignment had to be calculated only for the angle of 0 • . The pressure input data from tactile sensors do not have such strong differences to detect the misalignment with a precision of 1 mm. However, the misalignment with the step of 5 mm can be detected by contact patterns. The error of the electrode detection by the camera in proximity to the target robot equals 1 cm. Thus, we were changing the end effector position to obtain contact patterns for horizontal and vertical misalignments from -10 to 10 mm with a step of 5 mm. Therefore, 5 classes of misalignment for each direction were defined. If absolute vertical or horizontal misalignment is more than 10 mm, tactile sensors cannot touch the electrodes of the main robot after the docking. In the case of receiving zero input arrays from tactile sensors, the charging process will end. For solving the classification problem and determining the optimal model for the misalignment prediction, shallow and Deep Learning (DL) approaches were tested. DeltaCharger has two tactile sensor arrays with a resolution of 100 points per frame each. Therefore, the dimension of the input vector for the classification problem is 200. We have explored 7 popular classification models: 1) Linear classification models: • Logistic Regression (LR) Classifier with crossvalidation estimator; • Support Vector Machine (SVM) with stochastic gradient descent (SGD) learning. 2) A supervised neighbors-based learning methods: Deep Learning (DL) methods apply non-linear processing layers for discriminative or generative feature abstraction and transformation for pattern analysis. We can use a regular neural network (NN) with an input vector of 1x200 or transform the tactile data to a 10x10 array with 2 channels. This transformation allows to represent the obtained force values as an image and apply a convolutional neural network (CNN) for the prediction. We developed the regular NN and CNN to compare them with each other and to achieve a better performance than shallow learning methods. The developed architectures are shown in Fig. 6 . The regular NN consists of fully connected linear layers with ReLU (Rectified Linear Unit) nonlinear activation functions and batch normalization. Batch normalization acts as a regularizer and speeds up the training of classification models [25] . In addition, with batch normalization, gradients used in training are more predictive and well-behaved, which enables faster and more effective optimization [26] . CNN includes 2 convolutional layers with a 3x3 kernel and 3 fully connected linear layers with ReLU nonlinear activation functions and batch normalization. For training and validation of classification models, 2 datasets were collected. The first one includes 600 data pairs from tactile sensor arrays. The main robot was tilted at an angle from 0 to 5 • . Per each angle span of 1 • , the end effector was docked with the main robot, and measurements were collected from 2 tactile sensors 100 times. Therefore, the collected data was balanced. The initial position of the end effector was varied from -20 to 20 mm along the X and Y axes with a step of 4 mm. Fig. 7 shows a set of obtained patterns for different values of misalignment angle ϕ. The second dataset includes 500 data pairs from tactile sensor arrays. The actuator was installed opposite to the counterpart with electrodes so that the coordinates of the electrodes in the actuator coordinate system were equal to (0; 0; 80) (coordinates are given in millimeters) at a zero misalignment angle. After that, the robot performed 20 attempts to dock with the electrodes. In a similar way, the target point for the actuator was changing with a 5 mm increment along the X and Y axes. As a result, the coordinates of the end effector changed from (-10; -10; 80) to (10; 10; 80). Both datasets were split into training (67% of data) and validation sets (33% of data) to avoid resubstitution validation or resubstitution evaluation [27] . For shallow learning models and regular NN, the data was flattened to one-dimensional vectors with 200 elements. For CNN, input data consisted of tensors with 2 channels (since we had two data pairs per sensor measurement) and with the shape of 10x10. All classification models were trained on the Intel NUC Mini PC (parameters are listed in the Table I ). 1) Classification models: All shallow learning models except XGBoost, which has its own python package, were trained using the Scikit-learn 3 machine learning library. DL models were constructed using a PyTorch 4 open-source machine learning framework. The cross-entropy loss was used as the criterion. It combines the LogSoftmax function with the negative log-likelihood loss. This criterion is commonly used for multilabel classification problems. Besides, we used the dynamic learning rate reducing scheduler ReduceLROn-Plateau. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. DL models for each case were trained during 50 epochs. 2) Combined results for angular misalignment prediction: The experimental results of all applied classification models for the misalignment angle prediction are shown in Fig. 8 . The collected datasets from two tactile sensors were balanced. Therefore, we chose accuracy as a quality metric for the final evaluation of all models for misalignment prediction. RF Classifier showed the best performance for the end effector misalignment detection between all shallow learning models. DL methods showed higher accuracy than shallow learning approaches, and CNN had the highest accuracy of 95.46% in the misalignment angle prediction problem. The disadvantage of CNN is the lowest speed among all models. Considering the speed of each method, the CNN model was significantly slower than the other models with an execution time of 38 ms. However, this time is still acceptable for the docking task. In conclusion, with CPU utilization, RF Classifier and regular NN showed similar performance considering model accuracy and execution speed. However, taking into account model learning time, which was 0.017 s and 6.597 s for RF Classifier and regular NN, respectively, in the misalignment angle prediction problem with CPU utilization, RF Classifier showed the best overall performance among all classification methods. However, in the next iteration of this project, we are going to use NVIDIA Jetson Nano with GPU acceleration. Therefore, CNN is an optimal classification model in this regard. It provides higher accuracy than other models, and with GPU acceleration it will be significantly faster. 3) Combined results for vertical and horizontal misalignment prediction: The DL and shallow learning models were retrained on the second dataset with the values of horizontal and vertical misalignment. The experimental results of all applied classification models for vertical and horizontal misalignment prediction are shown in Fig. 9 . The DL models showed higher accuracy than shallow learning approaches. However, the performance of the former for the prediction of vertical and horizontal misalignments is different. CNN showed a similar performance as ensemble methods, and regular NN had the highest accuracy of 87.9% in the horizontal misalignment prediction. For the prediction of vertical misalignment, CNN showed the highest accuracy of 98.2%. (a) Accuracy of models for the vertical misalignment prediction. (b) Accuracy comparison for the horizontal misalignment prediction. The CNN-based angular and vertical misalignment detection had the maximum accuracy of 95.46% and 98.2%, respectively. And for the detection of the horizontal misalignment, regular NN showed the highest accuracy of 87.9%. The force patterns mainly represented the positions of the electrodes of the main robot, which were pressing on the area of the sensor. The accuracy of predicting the angular and vertical misalignments was high due to differences in contact patterns depending on the position of the electrodes. The distribution of pressure points on the electrodes changed with the end effector orientation and shifted up or down on the obtained patterns. In the case of horizontal misalignment, the main difference between patterns was the force magnitude, while the shape of the patterns did not differ much. It is the main reason why detection of horizontal misalignment had a lower accuracy. One approach to solve this problem is applying the special shape of the electrode on the main robot, which can help to create unique patterns for horizontal misalignment. For example, if the electrodes have holes visible on the contact patterns, then the horizontal displacement will be easier to predict. In this work, we presented DeltaCharger, the Inverted Delta manipulator with embedded high-fidelity tactile sensors for charging mobile robots. The Inverted Delta mechanism is proposed to achieve a compact and reliable structure. The CNN-driven tactile perception system using data from tactile pressure sensors can accurately recognize the angular, vertical, and horizontal misalignments between the electrodes of the charger and the electrodes of the main robot. Based on the estimated angle, the proposed system decides whether it is safe to charge the main robot or the current position of the electrodes will cause a short circuit. Depending on the vertical and horizontal misalignments, the system adjusts the end effector position to reach the entire area of contact between the electrodes. The proposed CNN-based tactile perception system can predict the angle, vertical, and horizontal values of end effector misalignment from pressure data with an accuracy of 95.46%, 98.2%, and 86.9%, respectively. DeltaCharger can also be applied in charging of the heterogeneous robots, such as, mobile, quadruped, UAV, with different height of charging electrodes. The future work will be aimed at developing and assembling a mobile robot for charging. Besides, the mechanism of the end effector will be improved to allow electrode rotating in order to make it possible to charge the mobile robot tilted at the critical angle. HanGrawler: Large-Payload and High-Speed Ceiling Mobile Robot Using Crawler Phollower-The Universal Autonomous Mobile Robot for Industry and Civil Environments with COVID-19 Germicide Addon Meeting Safety Requirements WareVision: CNN Barcode Detection-Based UAV Trajectory Optimization for Autonomous Warehouse Stocktaking IkeaBot: An autonomous multi-robot coordinated furniture assembly system Mona: an affordable open-source mobile robot for education and research Exploring the Applicability of Robot-Assisted UV Disinfection in Radiology Ultrabot: Autonomous Mobile Robot for Indoor UV-C Disinfection Battery charge scheduling in long-life autonomous mobile robots via multi-objective decision making under uncertainty Efficient laser-based 3d slam for coal mine rescue robots Mission planning for a multi-robot team with a solarpowered charging station Vision-based autonomous docking and recharging system for mobile robot in warehouse environment Design of a cable-driven auto-charging robot for electric vehicles Concept design and load capacity analysis of a novel serial-parallel robot for the automatic charging of electric vehicles An autonomous charging system for a robotic fish Development of wireless charging robot for indoor environment based on probabilistic roadmap Swarm robots long term autonomy using moveable charger Autonomous robot-like mobile chargers for electric vehicles at public parking facilities Design of a contactless battery charger for micro-robots An overview of technical challenges and advances of inductive wireless power transmission Automated robot-based charging system for electric vehicles Automated Alignment with Respect to a Moving Inductive Wireless Charger TouchVR: a Wearable Haptic Interface for VR Aimed at Delivering Multi-modal Stimuli at the User's Palm A System of Tactile Transmission on the Fingertips with Electrical-Thermal and Vibration Stimulation XGBoost: A Scalable Tree Boosting System Batch normalization: Accelerating deep network training by reducing internal covariate shift How does batch normalization help optimization Model evaluation, model selection, and algorithm selection in machine learning The authors would like to thank Professor Hiroyuki Kajimoto, University of Electro-Communications, Japan for providing tactile sensor arrays, and researcher Jonathan Tirado, Skolkovo Institute of Science and Technology, for helping with setting up the tactile sensors.