1 Introduction

Insects are fundamental in ecosystems as pollinators, herbivores, detritivores, nutrient cyclers, and food sources for other species [1]. Native bees are pollinators related to various plant foods consumed [2]. Bees are fundamental in world agriculture to improve crop quality and yields [3]. Furthermore, they actively contribute to the recovery of degraded ecosystems [4], can be used in biovectorization - technology that uses insects as biocontrol agents [5], and can also play a crucial role in sustainable development projects and programs [4].

Studies report biodiversity losses among native bees in tropical regions [6, 7]. The main factors responsible for your decline are infections disseminated by parasites and pathogens, lack of genetic variability, stress due to the seasonal movement of hives to pollinate fruits and vegetables, toxic pesticide residues found in pollen, nectar, and hives (mite control), the low nutritional value of agro-landscapes dominated by monocultures (such as corn, soybeans, and cotton), and the most adverse weather conditions in recent decades [1, 6, 8,9,10].

The decline of insects results in adverse effects on ecosystems. Thus, preserving its abundance and diversity is fundamental for ecology [1]. Individuals with little entomological knowledge are incapable differentiates categories of insects, their stages of maturity, and their behavior. Therefore, it is necessary to develop faster and more effective approaches that solve these problems [11]. The issues addressed in the insect recognition and classification process are to quickly detect the animal of interest positioned on a complex background, accurately distinguish insect species with high similarity, and effectively identify patterns of interest in the behavior of the classes. Artificial Intelligence (AI) tools can have a mutualist relationship with methods applied in ecology. The hives monitoring can help combat the decline of bees related to foraging, pollen characteristics, hive behavior, and invasive species.

Attempting to help monitor hives, we used an object detector to detect and classify Tetragonisca angustula bees. We incorporated a simple tracking system based on the object detector to describe the movement of insects, generating information about count and their direction concerning the hive entrance. We also tested a more robust tracking system, in an attempt to treat occlusions, compared to the previous one.

To next section contains recent work on artificial intelligence assisting insect management and conservation. Section 3 presents the methods of detection, classification, tracking, and optical flow of the species. Section 4 contains the results obtained by the steps highlighted in anterior with your corresponding discussions. Finally, we present the conclusions and future perspectives in Sect. 5.

2 Related Works

Some works developed in the last six years stand out in an attempt to automate processes applied to Biological Sciences involving the Class Insecta of the Phylum Arthropoda (Kingdom Animalia). AI techniques used to monitor insects from images can provide an alternative to reduce manual activities and human errors [12]. In this sense, there is a tendency to use computer vision approaches for insect detection and recognition.

Object detection techniques such as Faster R-CNN [13], VGG19 [11], YOLO + SVM [14], and ResNet (mAP around 90%) [15] proved adequate. Such approaches became possible to identify, classify and count insects in plants, leaves, photographs, traps, and grains [13, 16]. The insects include wasps and moths [11].

Liu and his team [17] developed a system called PestNet for field pest detection. Monitoring the number of pest species prevents the indiscriminate use of pesticides that result in crops that are harmful to health. The acquisition equipment is a multispectral light trap with an HD camera to capture pest specimens in a tray. Its best performance achieved an mAP of 75.46%.

Abiotic and biotic factors - such as social interactions - are used to understand social insects. Individuals in a colony engage in various tasks, such as foraging and hive care, depending on social contexts and interactions [18]. Challenges arise when monitoring these colonies because individuals are small and numerous [19]. Tracking systems have been used for this function, as insect movements can provide information about their cognition and decision-making [20].

Most tracking uses classical machine learning techniques [21]; however, computer vision can automate tracking, featuring velocity, straightness, direction, and exploration of trajectories [22]. Markers make the tracking system based on images more robust, making it possible to measure locomotor activities [18] of members of an ant colony - in the laboratory - to study their circadian activity [19]. Some authors, such as [23], try to integrate real-time data analytics and monitor them under natural conditions, such as in a tropical forest [22] without markers. Bioluminescence and behavior studies of insects such as fireflies [24] and pest counting in automatic traps with live insects also use tracking [25].

Concomitantly emerged were works involving the Apidae Family of the Hymenoptera Order. Bee foraging studies using RFID tags [2] merged with temperature, barometric pressure, and solar irradiance data define a Recurrent Neural Networks (RNN) architecture [8] that performs best in predicting bee behavior.

There are also systems with deep learning for automated monitoring of beehives (DeepBees) [10]. In this case, the footage of the bees takes place at the entrance to the hive. DeepBees monitors health and hygiene, attacks and invasions (wasps), pollen, bees dead, and drones. However, there are still challenges in the detection, direct mapping of activities, and with diseases or mite infestation; caused by lack of data.

Tracking for pollination studies targets bees. Monitoring natural habitats are essential for behavioral studies, preferably with non-invasive techniques. Hybrid Detection and Tracking (HyDaT) map interactions between bees and wildflowers; kNN segments the image background and foreground, and YOLO locates bees [20, 26]. Perez-Cham [27], Sun [28], and their teams use 3D tracking to analyze and reconstruct the bees’ flight paths.

This work presents the following contributions: It uses non-invasive techniques, without sensors in the hive and markers in the insects; Uses a one-stage object detector to classify and detect Jataí; It only uses tracking based on detections; Compares the performance of two tracking algorithms to perform the task; Contains specific implementation for the optical flow of bees and; Compares the performance of computer vision with humans. In this way, we believe it is possible to contribute to the ecological preservation of the species, combating the decline of insects and facilitating their management.

3 Methods

3.1 Custom Dataset – Tetragonisca angustula (Jataí)

The Jataí (Fig. 1) is a bee native to Brazil, and its geographic distribution occurs throughout the Brazilian territory. Considering its natural environment, this species has the habit of nesting in tree hollows and rock cavities. Its adaptability promotes nesting in hollow walls, light boxes, letter boxes, stone walls, and other unusual places, considering an urbanized environment.

The colony has a queen mother and an average of 5000 (from 2000 to 8000) workers. Guards are a caste with larger sizes than other workers. During the day, the workers remain positioned over the inlet tube or hover close to it. On average measures 4 mm and can forage 600 m from the hive.

Fig. 1.
figure 1

Tetragonisca angustula. Adapted from A.B.E.L.H.A.

Full size image

It is a significant pollinator of crops such as acapu, avocado, coffee, carrot, cupuaçu, guava, orange, mango, watermelon, strawberry, moressuma, pepper, cucumber, tangerine, umbu, and annatto. Therefore, proper management of hives ensures the productivity of such crops and protects the colony from natural enemies.

The custom dataset aims to monitor automatically of native bee hives using non-invasive techniques. It consists of videos of the entry of several colonies for tracking identified as XXXY-Y, where XXX identifies the hive location and Y-Y the recording interval. We identified images taken from the videos for object detectors as the complement of the previous XXXY-YZ…, where Z… is an id associated with each image. Each video and image contains an information file with the metadata.

Knowledge about the Jataís showed that monitoring on cold, rainy days and at night is unnecessary. Also, we avoid windy days. In this way, the data represent reality and are valid only for Jataís.

We used the SONY HDR-AS20 camera, with a resolution of 1920 × 1080p and a recording speed of 60 fps. In an attempt to represent the natural environment of the hives, the recordings took place at different times of movement at the hive’s entrance. We withdraw videos of a beehive from the BeeKeep CS database, with a resolution of 960 × 540p and a recording speed of 30 fps. We acquired videos of natural and artificial hives in urban and rural environments relatively close, but climatic conditions are different.

We positioned the video camera, trying to optimize the capture of movements around each hive. Therefore, there is only variation in the capture angle for different beehives. The dynamic in the hive’s entrance guarantees the variability of the poses of the bees.

The custom dataset is structured as required by the family of object detectors known as YOLO (test, valid, and train). These directories have a separation between images, labels, and information. The info folder contains the metadata. Incomplete files are due to data not provided by BeeKeep CS (16.7%). The custom dataset has 26487 instances in 2100 images, with 19000 in 1500 images for training, 3719 in 300 for validation, and 3768 in 300 for testing.

Climate variations, preferably days with the sun between clouds (83.2% of the images), periods of the days (16.7% obtained in the morning and 66.7% in the afternoon), and seasons (16.7% obtained in winter and 66.7% in spring), during the recordings provided differences in lighting. We noticed differences in lighting in 45.6% of the images.

The correlogram between labels contains a bimodal distribution that indicates a mixture of two trends with different labels, with a predominance of the right trend with the highest number of observations; A Chi-square distribution inferring that the labels are homogeneous; and two Poisson distributions indicating that the label sizes are random but repeat at a defined average rate. This way, the labels have adequate precision and cover all the bees in the images.

The custom dataset has no duplicates and leaks, considering the SSIM [29] of 0.97. As the bees occupy a small part of the image, for being small, the differences between images are subtle.

Fig. 2.
figure 2

Custom dataset samples. 001 (−20.381099, −43.506333)* - Natural hive in an urban environment. 002 (−20.502591, −43.520515)*, 003 (−20.502507, −43.520330)* and 005 (−20.345640, −43.608437)* - Natural hive in a rural environment. 004 (−20.345266, −43.608529)* - Artificial hive in a rural environment. 006 - BeeKeep CS artificial beehive. *Geographical coordinates obtained from Google.

Full size image

There is a background for each beehive present (Fig. 2), totaling 0.29% of the images. When we do not obtain images without bees, use data augmentation to remove them (0.24% of them). We fill in the resulting space of an unlit bee with the texture of its nearest neighbors.

1737 images have occlusions, that is, 82.7% of them. We observed occlusions due to: the overlapping of bees, by background item, per entrance pipe, by light beams, bees that touch each other, and bees cut by frames.

The perceived anomalies are shadows and distortions during bee flight. The incidence of sunlight causes shadows that can be confused with objects by computer vision models. Bee shadows are monitored but have only been ignored so far (361 images have bee shadows, i.e., 17.2%). Distortions during bees’ flight occur due to a higher flight speed than the capture capacity of the camera.

Not possible to infer over class distribution because the custom dataset only has one class so far. We did not observe intraclass variations, but they need monitoring during updates.

3.2 Detection and Classification of the Native Bee

YOLO [30] (You Only Look Once) is an object detection model used in computer vision. The model treats object detection as a regression problem using a single Convolutional Neural Network (CNN). It does this by splitting the image into a grid, making multiple predictions for each grid cell, filtering these predictions, and removing overlapping boxes to produce its final output. Its evolution includes YOLOv2 [31], YOLOv3 [32], YOLOv4 [33], YOLOv5 [34], and YOLOv6 [35]; each incorporates contributions to the model.

The ResNeXt backbone and the dual head of YOLOv7 [36], combined with the Label Assigner engine that assigns flexible labels, allow the model to learn from data more effectively. It presents a new multiscale training strategy and a technique called Focal Loss, designed to solve the class imbalance problem.

The latest release, YOLOv8 [37], surpasses previous by incorporating advances such as a new backbone, a head split without anchor, and new loss functions.

3.3 Tracking and Optical Flow of Jataís

Tracking is obtaining an initial set of objects, creating a unique Id for each one, and accompanying each object as they move in frames, maintaining the assignment of Id’s. An ideal tracking algorithm should detect objects only once, be fast, control when the tracked object disappears or moves out of frame, be robust with occlusions, and recover lost objects between frames.

This work uses a detector-based tracking similar to [23]; it depends on the Euclidean distance between existing and new centroids. Bounding boxes are the basis for calculating centroids with Id’s assigned to the initial set of centroids. After that, it computes the Euclidean distance between the existing and new centroids, associating new centroids with the old ones. Then, minimize distances and update your coordinates. Finally, there is the registration of new centroids and the cancellation of the registration of objects that leave the video. This a simple and fast method that controls when the tracked object disappears or leaves the frame and recovers lost objects between frames.

We attempt to improve the tracking with the algorithm ByteTrack [38] (BT) to it does the same as the previous one, solve occlusions, and detect objects only once. The algorithm performs associations between the detections of the current frame concerning the previous frame. For that, BT uses similarity calculations consider the distances between the resources of appearance (Re-ID) and IoU resources. Then the Hungarian algorithm finalizes the match based on similarity.

Optical flow is used in computer vision to characterize and quantify the movement of objects in a video stream, often for motion-based object detection and tracking systems. When estimating optical flow between frames is possible to measure an objects speed in the video. It is also possible to follow the movement of Jataís, count the bees that leave and enter the hive, and your total. This approach can generate information about hive nutrition.

For this, we propose using the centroids of the detections containing the same Id and a displacement limit from the current frame to the previous frame. We define a circle containing the entrance of the hive as a limit. A Jataí has entered the beehive if the Euclidean distance between the circle center and detection centroid is greater than your radius in the previous frame and smaller than the current frame. Otherwise, the Jataí leaves the hive.

4 Results and Discussion

4.1 Detection and Classification of Jataí

Results from YOLOv8 [37] trained on the custom dataset are in Fig. 3. In just 300 epochs of training, the YOLOv8x model converged to a precision of 0.97, recall of 0.957, mAP50 of 0.984, and mAP50–95 of 0.744 (Fig. 3a). Considering the test data, the confusion matrix indicates that: every time the model predicts a class and is background, the reason is Jataí; 98% of Jataí predictions are Jataí and; 2% of background predictions are Jataí. The precision (0.935) – recall (0.937) curve for mAP50 of 0.969 and mAP50–95 of 0.682 (Fig. 3b) shows that most of the classifications that the model made are correct, and most of the predicted bounding boxes are as expected.

The YOLOv7 [36] model converged to a precision of 0.952, recall of 0.948, mAP50 of 0.979, and mAP50–95 of 0.634 (Fig. 3c). Considering the test data, the confusion matrix is identical to the previous one. The precision (0.922) – recall (0.901) curve for mAP50 of 0.955 and mAP50–95 of 0.579 (Fig. 3d) also shows the same as the previous one.

We noticed that both present acceptable results for the application, but the YOLOv8 metrics reach better scores than the YOLOv7 ones. We believe this is due to differences in their backbones. In this way, we use YOLOv8 for the tracking implementations.

Fig. 3.
figure 3

Training results and Precision-Recall curve with YOLOv8 (a, b) and YOLOv7 (c, d), respectively.

Full size image

4.2 Tracking and Optical Flow of Detected Bees

In the tracking algorithm called EuTrack (ET), which depends on Euclidean distance, a parameter can be changed to control when the tracked object disappears or leaves the frame and its ability to recover lost objects between frames. We set the parameter as 20 for all experiments, i.e., an object is only considered for a new Id if it is not detected for 20 consecutive frames.

The BT [38] is a robust system in treating occlusions because it performs similarity analysis, adopts the Kalman filter to predict the object’s movement, and uses additional Re-ID models to improve the long-range association. We set its parameters for the whole experiment as track_thresh equal to 0.05, track_buffer equal to 20, match_thresh equal to 1.0, aspect_ratio_thresh equal to 3.0, min_box_area equal to 1.0 and mot20 equal to True.

In Fig. 4, one can view the bounding box, detected class, and probability. The Tracking algorithms generate the Id and mark the centroid of each bounding box. The optical flow, implemented from the centroids and Id’s, informs about movements in and out of the hive (delimited by the red circle). The bee trails, with a displacement limit equal to 120, were hidden to facilitate the visualization of the other considerations. Also, observe that Jataí with Id 7 (Fig. 4 - 001) entered and the one with Id 14 (Fig. 4 - 003) left their respective hives. Timestamps can collaborate with this information to generate possible monitoring of the hive entrance.

Results for a video of each hive are in Table 1, along with their absolute error and mean relative error. We use a process similar to pseudo-labeling (Pseudo) to count bees in the videos (we call it a procedure with the aid of technology). We used ET to aid in counting. Note in Fig. 4, specifically at 003 (Id 2 and 5), 005 (Id 4), and 006 (Id 0 and 9), the presence of soldiers hovering around the hives entrances. These soldiers generate errors in counting the bees that enter and leave the hive. Therefore, we remove duplicates from these counts (RD) to improve this aspect.

Summarizing Table 1, we have that ET has superior results in the bee count, but we can also use BT. It is best to remove duplicates of bees to monitor the hive entrance. The error is less when BT counts the bees that get out of the hive and when ET counts those that enter. In this way, there is a tie considering the best cost benefit.

We repeated the same procedure for six videos of hive 001 (Table 2), plus a count performed by a person from the biological sciences area without using technology.

Fig. 4.
figure 4

Results of the tracking and optical flow. Detailed information in Fig. 2.

Full size image

Summarizing Table 2, we have that BT has superior results in bee count. It is best to remove duplicates of bees to monitor the hive entrance. ET performs better in accounting for incoming and outgoing bees. In this way, ET obtains a better cost-benefit concerning the movement at the hive entrance and BT to count bees. We also noticed that using the system to assist human activity reduces errors caused by fatigue and distractions.

Table 1. Results of the trackings and humans with the aid of technology.
Full size table

The errors in Tables 1 and 2 are caused by some limitations. ET does not resolve Id’s swaps caused by crossing their positions and occlusions caused by soldiers and the hive tube. The movement in the hive tube is responsible for most of the exchanges of Id’s, and the model presents difficulties for bees that touch each other (Example in Fig. 4 - 005 Id 5 and 6).

Table 2. Results of the trackings and humans with and without the aid of technology.
Full size table

In agitated hives, agglomerations of bees occur. This fact splits a bounding box containing one bee into others and unifies bounding boxes into one containing several bees. This process contributes to the acquisition of new Ids, along with occlusions.

The transition of Jataís between frames and occlusions causes the resumption of Id by different bees. The camera is not steady in 006, and the systems recognize some anomalies in 002 and 004. We monitored anomalies, but it was not possible to extinguish them in the field.

The speed of the videos, frames per second, cannot be too low because the centroids must be close between the frames. BT relocates Id’s in the agglomerations with difficulties, causing the assignment of new Id’s and generating duplicates. The same error of dividing and merging bounding boxes also occurs in BT, contributing to new Id’s assignment.

There are difficulties in treating occlusions as well. The authors of [38] show that BT is robust for occlusions where one of the objects does not move during occlusion. This type of occlusion is rare with Jataís, as the occluded bee also moves. When the occlusion ends, the occluded object may be in a different position in rotation or translation. The movement causes the bee to have different characteristics from when it was occluded.

The similarity between individuals in the hive also impaired the similarity analysis used in BT. There are no physical differences between workers and castes in the same or different beehives. The bees themselves only differentiate through pheromones.

However, using the system in the field can help to combat the decline of bees related to foraging and hive behavior. Generating reports and doing repetitive work brings advantages to the approach, thus reducing errors of human origin. But the use in practical applications must consider the BT to count the bees with an average relative error of 11%. ET to monitor bees that enter with an average relative error of 9% (21% if there are outliers). Both to observe leaving bees, BT with an average relative error of 18% if there are outliers and ET with an average relative error of 33% otherwise.

5 Conclusion

We believe it is a proposal capable of detecting, classifying, and generating tracking information for Jataí in the field. In this way, it is possible to help combat the decline of bees related to foraging and hive behavior. We must consider a mixture of both systems in practical applications. BT counts bees with an average relative error of 11%. ET monitors incoming bees with an average relative error of 9% (21% if there are outliers). Both monitor bees that leave, BT with an average relative error of 18% if there are outliers, and ET with an average relative error of 33% otherwise.

The biggest challenge in this type of implementation is to obtain conditions and data of the species of interest in the field to compose the custom dataset. Harsh conditions common in natural environments provide difficulty in hardware use and maintenance. The context also introduced issues mainly caused by occlusions and outliers that we must overcome to reduce the relative error.

In future work, we intend to optimize the custom dataset by adding greater diversification to the data - considering invasive species, acquiring videos of different hives, adding data for tracking to the custom dataset presented, and completing the metadata of the videos with pseudo-labeling for counts and the same performed by biologists. To implement the MOT20 format [39] in the tracking custom dataset introducing evaluation metrics. To compare results with other Tracking algorithms in an attempt to reduce the limitations of this work. Finally, we will implement pollen identification in monitoring to generate information about hive nutrition. (Additional information at https://github.com/Rodolfoloc/Native-bees).