Deep Convolutional Features for Fingerprint Indexing

Abstract

Automatic fingerprint identification systems (AFIS) are among the most used people identification solutions. As the size of fingerprint databases is continuously growing, studying fingerprint indexing mechanisms is desirable to facilitate the search process in a large-scale database. This work presents a method for fingerprint indexing, which uses both exact and approximation methods of nearest neighbors (ANNs), which are very efficient in terms of runtime, even if they sacrifice a little accuracy by presenting approximate solutions. In the presented approach, searches with ANN methods are made from deep embedding vectors extracted from image databases using a convolutional neural network (CNN). In this work, a CNN ResNet18 was used to extract the deep feature embeddings vectors, and the vectors vary in size between 64, 96, and 128. The ANNs methods tested for the query step were ANNOY, NGT, HNSW, and Nanoflann. The results were quite promising when using the FVC fingerprint databases (2000, 2002, and 2004), once we reached 100% hits in the searches with a penetration rate of 1%, with very low run times.

1 Introduction

The use of fingerprints as a form of authentication and identification has proven to be a robust and efficient method over the past few years. It is assumed that fingerprints are unique to each individual and do not change their shape and texture throughout life. Another positive point for using this type of technology is the low cost and the high quality of fingerprint capture sensors, compared to other biometric sensors. Lines or ridges form fingerprints, and the spaces between these lines are called valleys. Based on features found in the flow of these lines, most automatic fingerprint identification systems (AFIS) can distinguish two fingerprints.

Lately, fingerprint inclusion and the massive use of technological devices by governments and private institutions have generated AFIS with large databases registered with many individuals. In this context, a simple identification query to a system with millions of people can take a completely undesirable time. In this context, AFIS applications usually employ indexing mechanisms when identifying individuals by searching in a subset of the database instead of searching in the entire registered database. Fingerprint indexing can be done using different fingerprint features, such as singular points [25], minutiae [7], and texture [12]. In the latter case, some works have used Convolutional Neural Networks (CNNs) to extract features from fingerprints in recent years.

Generally, indexing mechanisms consist of two steps: indexing and querying. When indexing, the system extracts different kinds of features from fingerprints and stores them according to the method. In the query step, according to a queried fingerprint, the collection of features is generated so that a search is made based on similarity to return the k fingerprints most similar to the query, where k is a previously defined number [8].

In this work, we used approximate nearest neighbor search algorithms (ANN) as an indexing and search technique based on fingerprints’ features extracted using a CNN architecture, ResNet18. We used ANN to calculate the k best candidates (i.e., the k most similar) fingerprints. We conducted experiments to evaluate the performance of the proposed solution using FVC (Fingerprint Verification Competition) databases and four ANN methods (ANNOY [2], HNSW [18], NGT [11], and Nanoflann [4]). Our main goal was to study how different parameters influence the system’s performance, which is evaluated with time and hit rate metrics.

This work is organized as follows: In Sect. 2, the database used for searches with the presented method was shown. In Sect. 3, concepts to understand this work were described; they are indexing, evaluation, and performance metrics. In Sect. 4, the related works were presented. In Sect. 5, the method presented in this work was described. In Sect. 6, we have the preparations for the performance evaluation tests. In Sect. 7, the results were presented. Furthermore, finally, in Sects. 8 and 9, we have the conclusion and acknowledgments, respectively.

2 FVC Databases

The FVC 2000, 2002 e 2004 datasets, described in [21], has a total of 10,560 fingerprint samples divided into different databases (DB1, DB2, DB3, DB4). Each DB has 110 different fingers, and each finger 8 different finger captures, each capture representing a different image of the same finger. Another important piece of information is that within the same year, the same 110 fingers were captured between the different DBs, making it possible to compare the same finger with different equipment, just comparing the same finger id with different DBs of the same finger same year.

There is an internal subdivision for one of the DBs, where for the 110 sets of samples, those numbered from 1 to 100 correspond to DBa, and those numbered from 101 to 110 are part of DBb. Researchers commonly use such division as the split into training and test sets, where DBa is used for training and DBb is used for testing, allowing the results to be compared fairly.

Table  1 presents detailed information about the FVC dataset, such as image dimensions, sensor type for each DB in each year. As we can see in the table, there are optical, capacitive, and thermal sensors, and each type of sensor model results in images with different dimensions. DB4 databases were synthetically generated by the tools described in the “sensor type" column. Figures 1, 2 and 3 shows one sample from each database described in the Table 1.

Table 1. Description of FVC databases with the list of sensors.

Fig. 1.

FVC 2000: (a) 2000_DB1; (b) 2000_DB2; (c) 2000_DB3; (d) 2000_DB4.

Fig. 2.

FVC 2002: (a) 2002_DB1; (b) 2002_DB2; (c) 2002_DB3; (d) 2002_DB4.

Fig. 3.

FVC 2004: (a) 2004_DB1; (b) 2004_DB2; (c) 2004_DB3; (d) 2004_DB4.

3 Indexing, Evaluation, and Performance Metrics

The construction of indexes in the fingerprint storage can follow different strategies, among the main ones based on minutiae, singular points, and, more recently, deep features. The indexing process consists of two main steps, the indexing of features and the recovery strategy. When indexing a database of fingerprints, the objective is to optimize searches in large collections of registered individuals, making the identification process less complex, reducing the 1:1 verification needed until finding the searched fingerprint, thus saving considerable time to get an answer.

The set of features consists of a collection of discriminating characteristics extracted by the strategy adopted to differentiate and store the fingerprints in the representation of the index. The recovery step consists of recovering a considerably smaller subset of previously registered fingerprints using some similarity function. Such a smaller subset forms the list of candidate fingerprints, where it is expected that among these is the sought fingerprint.

According to [8] the challenges in indexing fingerprints are: features with high dimensions, features of varying type, unorderable indexes, no pattern in catches of the fingerprints, especially concerning lighting and occlusion, images used in the search may be of low quality or be rotated, and finally the interoperability between capture devices as can be seen in the description of the FVC datasets.

There are different metrics used to evaluate indexing techniques, among which we can mention the hit rate, the penetration rate, and the search time. Defined in the Eq. 1 the hit rate is the percentage of genuine matches between the queried fingerprint and the set k returned by the recovery method concerning all queries. The penetration rate is how much of the search set equals the value of k, and usually, this rate varies between $1\%$ and $10\%$ [13, 17, 22]. Moreover, the search time is equivalent to the total execution time of the respective approximate method responsible for identifying all images from the test set.

$$\begin{aligned} Hit\_rate = \frac{hits}{queries}\times 100\% \end{aligned}$$

(1)

4 Related Works

In [6], the authors proposed an indexing mechanism based on CNNs, more precisely using Inception V3 [23]. In the proposed mechanism, fingerprints are aligned in a unified coordinate system, and a dictionary of maps is created from a base of 440 thousand fingerprints. The CNN is used to extract fixed-length features to be indexed and uses consensus similarity metric when querying. Tests were performed using NIST’s SD4 and SD14 datasets and demonstrated better performance than state of the art published so far.

An indexing solution was proposed in [22], which is composed of three different CNNs. Being the first network called FingerNet [24] to enhance the fingerprints and extract minutiae, with the minutiae divided by locations in smaller blocks. With another CNN, features are extracted based on texture and minutiae, called Minutia-centred Deep Convolutional (MDC). Finally, the last CNN aggregates the features of the descriptors in the representation end name AMDC or aggregated LCD. The similarity between the indexed features is calculated directly via AMDC, where only a normalization and scalar product between two AMDC features is used. The experiments were carried out with FVC2000 DB2, FVC2000 DB3, NIST SD4, NIST SD4 natural, and NIST SD14, preventing the method’s superiority in several studies. However, there were no comparisons with techniques based on CNN.

Some works use variations of Delaunay triangulation for indexing. In [14] this is used with the k-means clustering algorithm for search, performing experiments on the FVC2000, FVC2004, FVC2004, NIST SD4, and NIST SD14 datasets. In [13] an extended triangulation is used and partitioning the triangles into groups, presenting results on the FVC2002 and FVC2004 databases. Improving on the results of these papers, in [17] a new index vector is proposed whose values are spread in index space, with experiments using the datasets FVC2002 and FVC2004, outperforming the state-of-the-art.

The authors of [1] studied fingerprint indexing using information from the fingerprint pores, which requires high-quality images and sensors with a lot of capture capacity. Complete and partial fingerprint pore information is extracted and clustered using K-means. The recovery of fingerprints is made possible by searching for the nearest cluster based on the pores detected in the query image. The results are satisfactory compared to other techniques based on minutiae.

5 Proposed Indexing Method

The objective of this work is to evaluate a solution for fingerprint indexing. When performing tests with the FVC 2000, 2002, and 2004 databases, it is possible to evaluate results for different types of sensors. In the following subsections, both the solution and the techniques used in its composition will be detailed.

5.1 Architecture

The system to be evaluated consists of two different stages, the fingerprint indexing stage and the indexed fingerprint query stage, like the other indexing systems. It is considered that the input images were captured with fingerprint sensors and preprocessed so that they are used both for indexing and for queries.

Fig. 4.

Indexing system architecture

As can be seen in Fig. 4 in the indexing process, the fingerprint is given as an entry in a CNN so that the features can be extracted so that they are stored in an indexing base. The result of this step is an indexing database duly completed and ready to carry out consultations. The networks are trained with different fingerprints than those used in the production environment and evidently need to be trained in a stage before using the approach.

The second and complementary step of the solution is to perform searches with fingerprints so that the subset of fingerprints is considerably smaller than the registered base is returned so that the task of fingerprint identification is more efficient. Each fingerprint consulted is extracted from the embeddings with CNN, and then searches are made using approximate ANN algorithms to define the fingerprint k most similar to the query.

5.2 Deep Metric Learning

The generation of embeddings, that is, a set of features is carried out through a convolutional neural network (Convolutional Neural Network, CNN). The chosen CNN is the ResNet18 because it is an architecture with a good trade-off between performance and computational cost. The embeddings are extracted from the neck of the last completely connected layers, with a dimensionality ranging between 64, 96, and 128.

The ResNet18 is a residual neural network, as shown in Fig. 5. Altogether the neural network has 11511784 trainable parameters, requiring a total of 2 GFLOPS for activation. The residual layers, obtained through shortcuts, facilitate the convergence of training and improve the accuracy [10].

Fig. 5.

ResNet18 architecture.

The implementation of ResNet18 present in the machine learning DLIB C++ library was used [15]. The loss function used is Coupled Loss (CL), which is the DLIB default, defined in the Eq. 2 [26]. The CL is defined in terms of a global distance threshold d and also by a margin $\epsilon$ added to decide when two embeddings n-dimensional, $f (x_1)$ and $f ( x_2)$, of two fingerprints, belong to the same identification. Therefore, if the distance between the embeddings is greater than d, then they belong to different fingerprints.

$$\begin{aligned} || f(x_1) - f(x_2)|| < d \end{aligned}$$

(2)

The training is carried out with N elements of an image base forming the load tensor, with a dimensionality of 224$\times$224 for the entry of CNNs, resizing the images to this dimension. When using backpropagation, a neural network can only learn from cases where the condition imposed by the loss function is violated. Therefore, example mining similar to [3] is used at the load loss computation level, so the training procedure benefits from difficult negatives, which typically improves quality and convergence. However, difficult negatives are also combined with random specimens, which, unlike [16], also take into account easy negatives.

5.3 Approximate Nearest Neighbors

The principle behind Approximate Nearest Neighbors (ANN) is to get a significant improvement in search performance, which comes at the cost of a loss in accuracy, so an approximately correct but fast result is obtained [2, 19]. This means that ANN methods can be used for various tasks, such as classification, labeling, and recommendation. In addition, there is evidence in the literature that, for some cases, ANN can also impact an unexpected improvement in results [5, 9]. Three state-of-the-art ANN methods have been chosen that are widely used in both research and commercial products: ANNOY, HNSW and NGT. In addition to these, Nanoflann was also used.

ANNOY (Approximate Nearest Neighbors Oh Yeah) [2] was developed by Erik Bernhardsson and is used by Spotify to recommend songs. Its implementation is summarized in a Binary Space Partition (BSP) tree using hyperplanes, similar to those used to render old 3D graphics. A random forest of BSP trees is built to improve search results: the more trees, the greater the accuracy of the results and the less “performance (time) for higher accuracy (quality)". Standard configurations use $2 * D$, where D is dimensionality.

HNSW (Hierarchical Navigable Small World Graphs) [18] is a graph-based method that separates links between nodes in a search graph according to their size scales in different layers, therefore performing searches in a multilayer graph. Consequently, the estimate of the nearest node occurs only in a necessary fixed portion of the connections for each element. The main advantages of the HNSW method are its robustness and its support for continuous incremental indexing.

NGT (Neighborhood Graph and Tree for Indexing Highdimensional Data) is a graph-based ANN method initially proposed to work on a large scale, high-dimensional vector space [11]. NGT improves the graph of k-nearest neighbors (KNNG) by adjusting the path and degree of edges derived from a KNNG, that is, optimization in terms of the number of edges entering and leaving influencing the accuracy and search time, respectively. In particular, the authors describe how an acceleration in performance can be achieved by removing edges that can be replaced by alternative search paths. NGT also supports adding and removing items from the index, which is an important feature for real-world applications. Experimental evaluations show that NGT can compete or even surpass other methods [2, 11].

Blanck and Kumar [4] propose nanoflann, based on FLANN [19, 20], one of the first libraries to provide ANN methods. This includes several algorithms to search for the nearest neighbor in metric spaces, such as KD-Trees variants. The nanoflann method was proposed for the construction of KD-Trees from databases describing point clouds and rotation groups. However, nanoflann does not focus on searches for approximately closer neighbors but on the exact performance of the closest neighbor based on optimizations, such as adopting quadratic distances and customizable memory access policies.

6 Experimental Evaluation

The experiments were carried out on a computer with Linux operating system (Ubuntu 20.04.2 LTS), equipped with an Intel(R) Core(TM) i7-9750H CPU @ 2.60 GHz$\times$12, 16 GB RAM memory, a 512 GB SSD, and NVIDIA GeForce RTX 1660TI graphics card.

For the generation of embeddings, ResNet18 was trained using a synthetic dataset, with 100 identities with ten fingerprint images each, totaling 1000 images. This decision was made to verify the efficiency of generalizing the generation of embeddings to other datasets without the need for new training. The fingerprints were generated using a proprietary tool of a project of which the authors of this paper are members¹. Figure 6 shows an example of a fingerprint from the synthetic dataset.

Fig. 6.

Examples of fingerprints from our synthetic dataset.

Regarding the generation of embeddings for search with ANN models, tests were performed using all FVC BDs. First, the bases were separated into search and test sets. The search set corresponds to the sample in which the ANNs models will consult and calculate the nearest neighbors in order to identify each of the elements of the test set.

For comparison with related works, the DBa of the FVC DBs corresponded to the search set, and the DBb was used for the test set. For each of the search and test sets, three sets of embeddings were generated, with their respective sizes of 64, 96, and 128.

Finally, the ANNs models were tested in each of the search and test sets in order to verify which combinations of these models and embedding vector size yield better hit rates. As for the penetration rate, also in order to compare to related works, a minimum of 1% and a maximum of 10% were used. In this work, the penetration rate is directly related to the k value of nearest neighbors used by the ANN models, that is, a penetration rate of 1% indicates that a value equivalent to this rate was used as k. So, as the search set has $100*8=800$ elements, the value corresponds to k with 1% penetration is 8, and the value corresponds to 10% is 80.

Search time was also a metric calculated in the experiments in order to verify the efficiency of the approach in generating good results quickly. It equals the sum of all searches performed in the test set.

7 Results

First, the results will be presented regarding the training of ResNet18 to generate deep feature embeddings using the synthetic image dataset. Figure 7 shows the evolution of the curve of mean loss values concerning the CNN training.

Fig. 7.

Evolution of average loss for ResNet18 training over our synthetic dataset.

An early stopping strategy was used for stopping training in order to avoid overfitting. Therefore, when stagnation in the loss curve evolution was detected for many epochs, training was manually interrupted. We validated this strategy empirically using increasing volume cuts from the training base. It was found that, in fact, this situation led to a model of poor performance. The results observed from queries based on the deep feature embedding vectors generated by our best model provide evidence that the stopping criterion was well applied.

Table 2 presents the best results obtained in queries with the ANN methods in each of the FVC image databases. The best results were considered in relation to the hit rate of $1\%$ penetration to $10\%$. That is, whoever obtained better results from $1\%$ penetration is ahead, as the idea is to use the least value of nearest neighbors k for searches. In case of a tie, the combination of the ANN model with the smaller embeddings vector was highlighted since the larger the embeddings vector, the more computational resources are allocated to generate it. If the tie persists, the following criterion was the search time for each of the penetration rates. In all cases, the NGT method was superior in relation to the other ANNs, regarding the Hit rate, therefore, all the results of Table 2 were obtained with the NGT method.

Table 2. Results of the best combinations of embeddings vector size for each of FVC ’DBs.

Generally, the larger the vector of embeddings, the better the results found by searches of the ANNs methods, but the longer the search time tends to be. Search time in each search performed presented values so low that the difference between these can be considered disregarded.

Complementing the previous data, Table 3 presents the best results obtained by each of the ANN models.

Table 3. Best results obtained by each of the ANNs models.

Interestingly, the best results for each of the ANN models were all relative to searches over FVC’s 2004 DB3. There is evidence that this dataset has the most uncomplicated images to identify by the proposed method. As much as the NGT method was superior, it is possible to notice that the other indexing methods do not fall so far behind in terms of accuracy because they still present very competitive results.

The results obtained in this paper were compared with previous results from the literature. For doing so, evaluation was carried out using the same image bases used in the related works. It is worthy of mentioning that all the works considered used the same search and test set separation scheme. The comparisons were made considering the best results of each one for the queries in the respective image database. These comparative results are depicted by Figs. 8, 9, 10 and 11. Bear in mind that some works do not appear in all comparisons, as the authors did not present tests regarding the fingerprints database in their work.

Fig. 8.

Comparison with other indexing methods with FVC 2000_DB2 dataset.

Fig. 9.

Comparison with other indexing methods with FVC 2000_DB3 dataset.

Fig. 10.

Comparison with other indexing methods with FVC 2002_DB1 dataset.

Fig. 11.

Comparison with other indexing methods with FVC 2004_DB1 dataset.

The search method presented in this work using ANN models presented better results in most cases. Unfortunately, none of the related works discussed in this paper present information on search times for comparison, which would probably enhance the proposal of this work even more. After all, the search times were negligible.

8 Conclusion

This work presented a method for fingerprint indexing based on deep embedding vector extraction using CNN and ANNs. We performed several experiments varying the embedding vector size and using four ANNs methods (ANNOY, NGT, HNSW, and Nanoflann). The feature extraction CNN was trained using a synthetic dataset that was applied out-of-the-box to the evaluation datasets constructed with different sensors for real-world fingerprints (see Table 1). Experimental results indicate that the proposed method is competitive compared to other works found in the literature.

On top of that, for several FVC databases, the method achieved 100% hit rate at a low penetration rate of 1%.

It is important to highlight that the CNN used (ResNet18) is simple and fast. This model can be leveraged to run the proposed method on resource-constrained devices, such as smartphones, as well as stated by the tiny search times.

As for future work, it is intended to use other CNNs to compare with the performance of ResNet18. It is intended to use other databases for training CNNs for the indexing stage and also for the query stage. Finally, it is planned to test the solution on smartphones, at least the query step, in order to verify performance on mobile hardware.