key: cord-0275488-jr9cz14h
authors: Bodduna, Kireeti; Weickert, Joachim
title: Removing Multi-frame Gaussian Noise by Combining Patch-based Filters with Optical Flow
date: 2020-01-22
journal: nan
DOI: 10.1117/1.jei.30.3.033031
sha: 72aaeeee63d02ee9eca1764df4af8671cb292938
doc_id: 275488
cord_uid: jr9cz14h

Patch-based approaches such as 3D block matching (BM3D) and non-local Bayes (NLB) are widely accepted filters for removing Gaussian noise from single-frame images. In this work, we propose three extensions for these filters when there exist multiple frames of the same scene. The first of them employs reference patches on every frame instead of a commonly used single reference frame method, thus utilizing the complete available information. The remaining two techniques use a separable spatio-temporal filter to reduce interactions between dissimilar regions, hence mitigating artifacts. In order to deal with non-registered datasets we combine all our extensions with robust optical flow computation. Two of our proposed multi-frame filters outperform existing extensions on most occasions by a significant margin while also being competitive with a state-of-the-art neural network-based technique. Moreover, one of these two strategies is the fastest among all due to its separable design.

Restoring images corrupted with various types of noise degradations is a classical image processing problem. Additive white Gaussian noise (AWGN), Poissonian and mixture noise types are the most studied noise models. AWGN elimination is particularly important because it combined with variance stabilizing transformations 1-3 for also removing the latter two types of noise.

In the single-frame AWGN elimination scenario, 4-9 BM3D 6, 7 and NLB 8, 9 produce superior results. Early contributions to Bayesian non-local denoising can be found in the works of Awate and Whitaker. 10, 11 For a comprehensive survey on image filtering, we refer to Milanfar 12 . BM3D and NLB are non-local patch-based methods which utilize the similar information available at distant regions in the image. More precisely, they filter a 3D group of similar patches. BM3D in particular is a quasi-standard for modern denoising algorithms. It is used as a benchmark in articles that involve both neural network-based techniques 13 and traditional approaches. 8 Multi-frame filters, [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] on the other hand, utilize information from multiple frames of the same scene to compute the final denoised image. In this work, we concentrate on the fundamental problem of finding general approaches that can optimally extend single-frame patch-based methods such as NLB and BM3D to the multi-frame scenario.

There already exist two types of extensions [24] [25] [26] [27] [28] for BM3D and NLB. Methods from the first category search for similar 2D patches from all the available frames. However, they use just one reference frame for filtering purposes, thus making limited use of the available information. 24, 25 Extensions from the other category take privilege of having more data in 3D spatio-temporal patches. [26] [27] [28] Nevertheless, techniques which utilize 2D patches on multiple reference frames and those which seperately filter information in the spatial and temporal dimensions, have not been studied. The latter can reduce undesirable interactions between regions of dissimilar greyvalues. Furthermore, a careful and systematic evaluation of these extensions is also missing.

Our Contribution. In order to address the above problems, in our recent conference paper 33 we introduced three extensions which can be divided into two categories: Firstly, we employed the 2D patch similarity approach of Buades et al. 24 and Tico 25 but using every frame as a reference one for filtering purposes. This ensured that we made use of the complete available information. Secondly, we introduced two other extensions which benefit from separately filtering the different types of data in the temporal and the spatial dimensions. The first one performs a simple temporal averaging followed by a single-frame spatial filtering, while the other reverses this order.

In the present work we additionally introduce three novel contributions: Firstly, we also consider non-registered data. In contrast to our conference work, 33 we combine our multi-frame filters with robust optical flow methods for dealing with the inter-frame motion. Such a study is really interesting as the utilisation of motion compensation was avoided by Arias and Morel 28 for circumventing motion estimation errors. In fact, contrary to most works on multi-frame denoising, we juxtapose the filter performance simultaneously for perfectly registered and for non-registered data. For the latter scenario, we pay special attention to parameter optimisation of the optical flow approaches. Such an analysis provides valuable additional insights into the importance of well optimized motion estimation in multi-frame denoising.

Secondly, we provide the first comprehensive evaluation of general strategies how to extend single-frame filters to multi-frame ones. In our previous work 33 we applied all the proposed extensions to just BM3D. In this paper, we also include the NLB denoising filter. Our evaluations include very high AWGN noise levels. Such large amplitudes of noise, which are consistenly ignored in the literature, are very relevant for microscopic and medical imaging applications.

Last but not least, we propose better parameter selection strategies for our filters than in our conference paper. We shall see that this will even change the order in our experimental rankings. For the sake of completeness, we also include three state-of-the-art multi-frame denoising solutions 27, 28, 34 in the evaluation part, which was missing in our preceding paper. 33 The neural network-based approach presented in 34, 35 is one among the many learning-based multi-frame filtering strategies 36, 37 adopted nowadays.

Paper Structure. In Section 2 we first review the central ideas behind the design of NLB and BM3D filters. We then introduce the five multi-frame extensions including our proposed techniques, along with the existing robust optical flow methods employed for registration. In the ensuing Section 3, the new optimal parameter selections for our extensions are presented. We also showcase the results of several denoising experiments along with detailed explanations behind the observed ranking of various techniques. Finally, in Section 4 we conclude our work with a summary and an outlook.

NLB 8, 9 and BM3D 6, 7 are non-local patch-based denoising methods which consider similar information from distant regions in the image. Both single-frame filters are two step approaches which combine the denoised image of the initial step with the noisy image in order to derive the final noise-free image. Furthermore, both of these steps are split into three sub-steps each, namely grouping, collaborative filtering and aggregation.

Grouping: In order to exploit the advantage of having more information, for every noisy reference patch considered, one forms a 3D group of similar patches using L 2 distance.

Collaborative Filtering: The term "collaborative" has a literal meaning here: Each patch in a group collaborates with the rest of them for simultaneous and efficient filtering. In NLB, one uses Bayesian filtering (in both main steps) to denoise the 3D groups. In BM3D, a hard thresholding (first main step) and Wiener filtering (second main step) are employed.

Aggregation: In order to derive the final denoised image, one computes a weighted averaging of the several denoised versions of every pixel.

In this section, we describe five multi-frame extensions for the above mentioned single-frame filters, in detail. For a better comprehension, we arrange all the five of them in an increasing order of design complexity.

In the multi-frame scenario, there exist slightly different types of data in the temporal and spatial dimensions. Thus, in order to combine them carefully the first two extensions break down spatio-temporal filtering into two separable stages.

Proposed Extension -Average then Filter (AF): First, we average all the frames registered using optical flow. Then we employ a single-frame filter for removing the remaining noise in the averaged frame.

Proposed Extension -Filter then Average (FA): Here, we first denoise every registered frame by using a single-frame filter and then average the denoised frames.

The above two approaches differ from some previous methods 29, 30 in the following fundamental aspect: Irrespective of the quality of registration, we utilize a temporal average and spatially filter strategy. This is different from a temporal average or spatially filter technique that depends on the registration error. While the first two extensions FA and AF perform a separable spatiotemporal filtering, the subsequent three employ combined filtering ideas. The first two among the three techniques utilize 2D patches and the final strategy considers 3D spatio-temporal ones. Let us discuss them in more detail now.

Existing Extension -Single Reference Frame Filtering (SF): 24, 25 Here, a single frame among all available ones is considered as the reference frame. One selects reference patches from just this frame. For every reference patch, a group of similar patches is formed using information from all the frames but not just one.

Proposed Extension -Multiple Reference Frame Filtering (MF): The fourth extension differs from SF in three different aspects. Firstly, in order to make complete use of the available information we consider all frames for reference patches. Secondly, we perform an aggregation of denoised pixels in such a way that after the first main step we have as many denoised frames as there are initial ones. This paves the way for the final difference: For every reference patch we find similar patches from all frames in the second main step also. We cannot do this in the second main step using SF because it has considered reference patches from just one frame initially. We can thus formulate the final denoised image u final which is obtained from a combination of the registered noisy data f and the initial denoised image u initial , as

Here, x denotes the 2D position vector. We represent the set of most similar patches to the reference patch P belonging to frame , using P(P ). For every patch Q in the set P(P ), we have χ Q (x) = 1 if x ∈ Q and 0 otherwise. The symbol u wien Q,P (x) denotes the estimation of the value at pixel position x, belonging to the patch Q. We derive this estimation through Wiener filtering (with coefficients w wien P ) a combination of f and u initial . In similar spirit to (1), we can formulate the NLB aggregation process:

Here, the superscript bayes implies Bayesian filtering. 8, 9 By restricting the total number of frames to one in (1) and (2), we obtain the original single-frame BM3D and NLB algorithms. This implies that MF encompasses the single-frame filters.

While grouping and filtering stages produce noise-free patches, aggregation computes the final denoised image from them. Employing 3D spatio-temporal patches gives an advantage of having more information at the patch denoising steps itself, even before employing the aggregation process. This exact idea is employed by the final extension.

Existing Extension -Combined Filtering (CF): [26] [27] [28] One fixes 3D spatio-temporal patches and searches for similar volumes instead of patches. Then, a 4D filtering technique is employed, which removes noise using all the considered similar volumes. Such ideas are in accordance with the single-frame NLB and BM3D filters, where one considers a 2D similarity measure combined with a 3D denoising technique. Table 1 serves as a look up table for the above five extensions and presents the chief characteristics of each one of them. By combining the five multi-frame extensions and the two single-frame filters, we have ten filters in total. As an example, we will abbreviate one of these combined techniques as BM3D-MF, if it is a combination of single-frame BM3D with extension MF. Due to space constraints, within the experimental results that are going to be presented in the upcoming subsections, we sometimes use shortforms for NLB-MF as NL-MF and BM3D-MF as BM-MF. Moreover, we use the abbreviation TA to denote temporal averaging. For non-registered data, TA denotes averaging after optical flow-based registration. Input: Noisy non-registered dataset f nr Main Algorithm: 1. We employ an optical flow technique for obtaining registered data f from f nr . Options for the optical flow methods include SOF-1, SOF-2 or SOF-3. 2. We utilize a combination of single-frame denoising filters with their multi-frame extensions for producing the final denoised output u final using registered data f . Options for the single-frame filters are NLB or BM3D. They can be combined with extensions AF, FA, SF or MF. Output: Denoised data u final Table 2 : A general algorithm of the proposed denoising scheme.

As already mentioned, we perform experiments on both perfectly registered and non-registered datasets. In the latter scenario, we need to first register the images before applying the above multi-frame extensions. Thus, we have employed three robust discontinuity preserving optical flow methods. [38] [39] [40] These motion estimation techniques perform better than some classical strategies. 41, 42 In all the three approaches, one minimizes a similar energy functional to determine the motion vector w = (w 1 , w 2 , 1) between frames f 1 and f 2 :

Here, x = (x, y, t) T denotes the spatio-temporal location, Ω is the 2D image domain and ∇ is the spatio-temporal gradient. The above energy penalizes deviations in both gray values and gradients. One enables interactions in between neighboring pixels through the smoothness term. The parameters γ and α represent the gradient and smoothness term weights, respectively. Moreover, applying Ψ(s 2 ) = √ s 2 + 2 results in a robust convex energy functional with = 0.001 ensuring strict convexity of Ψ. The smoothness function Φ(∇f 1 , λ) with parameter λ specifies the regu- larisation strategy. The three optical flow methods that we use in this work differ in the choice of this particular function. We abbreviate these three techniques as SOF-1, -2 and -3 (SOF means sub-optimal flow). In SOF-1, one employs a decreasing scalar function Φ(∇f 1 , λ) to preserve image driven flow discontinuities. The second and third optical flow strategies try to avoid blob like artifacts using two different approaches. SOF-2 performs a minimum isotropic diffusion even when the gradient is very large. In SOF-3, one utilizes an automatic selection strategy for λ. The same numerical procedure is adopted to compute the solution in all the three methods.

We use the above mentioned optical flow strategies for the first four extensions. The algorithm in Table 2 describes the main ideas behind the denoising framework of our approaches. The fifth method CF uses its own motion compensation techniques. The difference in the various motion estimation approaches used should not be an issue as we are also performing experiments on perfectly registered data. This finishes the modeling and theory part of this work. Now, we move on to the experimental demonstrations.

For creating perfectly registered data, we have considered multiple AWGN realisations of the classical House, Peppers and Bridge (http://sipi.usc.edu/database/) images with fourteen datasets each. They are obtained by a combination of σ noise = 10, 20, 40, 60, 80, 100, 120 with five-and ten-frame datasets. In a similar spirit, we have also created non-registered data by corrupting the Grove2, 43 Shoe and Bird House 44 images with AWGN. It has to be noted that we have not clipped the dynamic range of the images after degrading them by noise. 

Optical Flow Parameters: For the Grove2 dataset, we have optimized the optical flow parameters with respect to the ground truth flow for all three methods. We then choose the best method to register every dataset. For Shoe and Bird House datasets we have optimized the SOF-3 parameters with respect to the final denoised image directly as the ground truth flow was not available. Table  5 shows more details.

Denoising Parameters: Various studies [6] [7] [8] [9] 45 have contributed in making the single-frame filters BM3D and NLB parameter selection-free, while retaining the quality of the denoised images as much as possible. In a similar spirit to the above works, in this paper we use better versions of two extensions introduced in our conference paper. 33 Firstly, at the time of application of the filter in the first extension AF, the noise distribution has already changed due to temporal averaging. Since we are using an AWGN model, we know that the standard deviation of noise is reduced by a factor √ L for a dataset with L frames. We can improve the performance of type-AF extensions if we select the filter parameters corresponding to the new standard deviation.

The second improvement is to optimize the number of patches in a 3D group using both the original single-frame BM3D filter as well as the BM3D-MF technique. The threshold parameter on L 2 distance and the parameter which decides the maximum patches in a 3D group together Table 4 : PSNR values after denoising ten-frame datasets with various methods. Abbreviations as in Table 3 . control the total number of patches one employs for filtering purposes. Our experience suggests that the gain in quality due to the threshold for low amplitude noise elimination, is relatively lot less when compared to the deteriotion because of it in case of large noise levels. Since one of the main objectives of this paper is to concentrate on large noise amplitudes as well, for simplicity reasons we refrain from using the threshold parameter in any of the first four BM3D extensions. Moreover, Data α γ S10 25 1.5 S20 75 2.5 S40 95 1.5 S60 110 0.5 S80 85 0.5 S100 95 0.5 S120 90 0.5 Data α γ BH10 100 0.5 BH20 130 0.5 BH40 135 1.0 BH60 135 0.5 BH80 130 1.5 BH100 100 1.5 BH120 90 1.5 Table 5 : Optical flow parameter values used for different datasets. Left: Grove2 dataset with the best among SOF-1, SOF-2 and SOF-3 methods. We have considered the tenth frame as the reference frame since ground truth flow information was available between frames 10 and 11. Centre: Shoe dataset with SOF-3 approach. Right: Bird House dataset with SOF-3 technique. We have utilized the fifth frame as the reference frame for the latter two datasets and then employed frames 4-6 for optimizing the optical flow parameters. Also, we have used BM3D-MF and BM3D-FA as denoising filters for optimizing SOF parameters for these two datasets, respectively. Table 6 : PSNR values of denoised Grove2 images after using a combination of denoising methods and optical flow. Top: Four-frame datasets (frames 9-12). Bottom: Eight-frame datasets (frames 7-14). Frame size: 640 × 480.

in the multi-frame scenario we have more similar patches, when compared to the single-frame layout. We thus check in the upcoming sections, whether the best performing extension (BM3D-MF) in our conference publication, 33 can give even better results by increasing the maximum number of patches in a 3D group through doubling. We label this particular parametric choice as BM3D-MFO, where O stands for an optimized version.

For the results of perfectly registered noisy data using SF and CF techniques, we have always presented the best peak signal to noise ratio (PSNR) value among all frames. This ensures a fair comparison with the remaining three extensions. For experiments on non-registered datasets, we have calculated the PSNR value by leaving out a border of fifty pixels on all sides of the reference frame at which different frames were registered. We do this in order to mitigate the ill-effects due to unavailable information at the borders of registered images. This also makes sense for several multi-frame imaging applications where we capture the region of interest in the centre of the frame. Tables 3 and 4 showcase the PSNR values of the denoised images, and Figure 1 displays the visual results after we have applied all ten methods. It is clear from these results that extensions of type-AF outperform all other techniques. They are superior to type-MF approaches (which is in contradiction to our conference paper 33 ) as we account for the change in the noise distribution due to temporal averaging.

In the category-FA extensions, we directly apply the single-frame filters on every frame. This is a sub-optimal solution because we do not have enough signal on each of the frames. Techniques belonging to type-SF do not make use of the complete available information as they just consider a single reference frame.

In the MF and CF filters, we avoid the disadvantages of both FA and SF. However, they fall behind type-AF methods for two reasons: Firstly, we separate out temporal and spatial filtering in category-AF techniques. This is advantageous since we have noisy versions of the same original gray value in the temporal dimension for perfectly registered images. In the spatial dimensions we have noisy versions of approximately equal gray values in general. This outperforms simultaneous non-linear filtering of the MF and CF techniques, where we combine the information in all dimensions at one go. Such a strategy proves to be inferior even though we use a non-linear filtering in Table 9 : PSNRs after denoising 10-frame datasets with various methods. Left: Perfectly registered datasets. Right: Non-registered layout. Abbreviations are as in Table 3 . Moreover, G stands for Grove2, S for Shoe, and BH for Bird House.

the temporal dimension when compared to the linear temporal averaging of category-AF filters. Interestingly, a similar result was observed in a single-frame scenario in the work of Ram et al. 46 By adopting a simple linear filtering on a smoothly reordered set of pixels they could produce results almost equivalent to the sophisticated BM3D filtering. The reason behind such observations is that linear averaging of different noisy versions of the same pixel intensity does not create artifacts like a non-linear combination of dissimilar intensities does. This is also the reason why averaging is preferred in electron microscopy. 47 Moreover, the linear nature of temporal averaging helps in computing the new standard deviation of noise after temporal filtering through theoretical knowledge. The second reason why MF and CF types fall behind category-AF is the following: The latter extension computes the initial grouping on the less noisy averaged image. In all the other categories we do this on the noisy initial images, which makes the grouping error-prone. The overall better performance of type-AF filters does not mean we can immediately reject the next best MF and CF categories. We must remember that we assumed AWGN noise and perfect registration. In the first scenario, we were able to optimize the denoising ability of NLB-AF and BM3D-AF easily for AWGN. Its signal independent nature helped in easier selection of filtering parameters which account for the change in noise distribution after temporal averaging. For noise of Poissonian type for example, AWGN elimination methods are normally combined with variance stabilizing tranformations for noise elimination. These transformations have the property of inducing a bias while stablizing the variance in the data. In another recent paper, 48 we evaluated the first four BM3D extensions in the Poissonian noise scenario and observed similar results as for our Gaussian noise study: 33 BM3D-MF outperformed BM3D-AF. Apart from not accounting for the change in noise distribution due to temporal averaging, the above mentioned bias problem was also a reason behind this. We conjecture that employing more sophisticated stabilisation frameworks 49, 50 could help in this respect. The second scenario where we cannot reject methods from categories other than type-AF is for imperfect registrations. We will examine this situation in the upcoming section where we consider non-registered datasets.

Furthermore, BM3D-AF is superior to NLB-AF (from Tables 3, 4 and Figure 1 ) because BM3D is a better single-frame denoising method than NLB for gray value images. We infer that the usage The BM3D-AF+ variant uses a σ noise value that corresponds to the raw noisy images. It does not consider the change in noise distribution due to temporal averaging. It produces a result that is inferior by 1.74 dB on a 10-frame dataset with σ noise = 80.

of the discrete cosine transform and the bi-orthogonal spline wavelet transform in the two main steps of BM3D, respectively, leads to superior anisotropic modeling.

Tables 6, 7 and 8 display the PSNR values of the denoised images while Figures 2 and 3 showcase the visual results. It can be clearly seen that NLB-AFand BM3D-AF outperform other approaches several times. However, for low amplitude noise situations NLB-CF, which is the current state-ofthe-art method, is competitive with the category-AF extensions and even superior to them at certain occasions. Let us explore these results a bit further. For all the three datasets, we have performed experiments on two kinds of data: One with less number of frames and the other with more of them. In the latter case it is highly probable that there exists large motion between the reference frame and others which can lead to high errors in motion estimation. Hence, if a particular approach is able to produce better quality results for a high number of frames, this indicates that it is robust to motion Tables 3 and 9. estimation errors. From Tables 6, 7 and 8, we can observe that CF is the only technique which does not even have a single instance where the PSNR value has decreased when more number of frames have been utilized. AF, MF, FA and category-SF filters could produce enough quality improvement for perfectly registered data. However, in the present non-registered layout we can find at least one instance for each of these extensions where the quality has deteriorated with an increase in number of frames. The only explanation behind this is the robustness of category-CF extensions with respect to motion. However, at regions where the motion registration is correct, the performance of AF-type techniques is so high that they can outperform category-CF approaches despite presence of motion estimation errors at other regions. Nevertheless, optical flow methods will continue to improve in the future. Thus, the philosophy of our proposed category-AF extensions will benefit from these advancements. As already mentioned, the BM3D-MFO variant employs twice the number of patches than BM3D-MF. The decrease in PSNR from BM3D-MF to BM3D-MFO in Table 7 for high noise am-plitudes and visual results in Figure 4 indicate the following: The black patches in darker regions of the image can be eliminated using BM3D-MFO. However, we must use the above strategy of increasing the number of patches only if we encounter black patches. Having too many them in a 3D group would instead give rise to an undesirable blurring.

In order to emphasise the critical nature of noise standard deviation selection as well as optical flow-based registration, we have performed two small ablation studies. Figure 5 illustrates the importance of selecting the correct noise standard deviation. One might also argue that there is no need for an optical flow-based registration in category-MF extensions. They inherently possess a patch-based search algorithm which can compensate for motion within frames. However, such a strategy assumes a translatory motion. The optical flow approaches, on the other hand, are applicable for any type of motion. Figure 6 shows the significance of optical flow-based registration. In Table 10 , we present a more detailed ablation study for non-registered data.

Thus, we can draw two conclusions from our results: The latest robust optical flow methods are also capable of extending the best performing nature of type-AF filters from the perfectly registered layout to the non-registered scenario. Secondly, in the future we should concentrate on approaches which separate the filtering in spatial and temporal dimensions for ideal as well as practical situations, like BM3D-AF and NLB-AF.

In recent years, learning-based denoising solutions have gained a lot of attention. In order to finish a comprehensive evaluation of our proposed technique, we have also compared its performance with a state-of-the-art neural network-based filter -VNLNET. 34, 35 Table 9 shows the PSNR values of this evaluation. The results show that our strategy outperforms VNLNET in the perfectly registered scenario and is competitive with it in the non-registered layout.

All the above results show that type-AF filters are among the best performing methods irrespective of whether there is any motion or not in the image dataset, what criteria have been used to optimize the optical flow, and what kind of optical flow technique has been employed. In future, BM3D-AF and NLB-AF can be combined with occlusion handling, 31 deflickering and sharpening 27 strategies. One could also replace the present denoising and motion estimation techniques with better ones for further pushing the state-of-the-art standard.

The AF-type frameworks are also the fastest among all extensions as they employ separable spatio-temporal filtering. Since temporal averaging can be performed in real time, their net complexity is just a combination of the optical flow method and the 2D single-frame filter employed on the temporally averaged frame. Although all the experiments in this paper were performed using a CPU (Intel(R) Core(TM) i7-6700 CPU @3.4 GHz using C++ and OpenMP) implementation, we also have a GPU (NVIDIA GeForce GTX 1070 graphics card using ANSI C and CUDA) version of BM3D-MF. We have already shown that BM3D-MF encompasses the original single-frame BM3D algorithm mathematically. Thus, the same GPU implementation can also be employed for BM3D-AF by just changing the number of frames to one and using the new standard deviation of noise after temporal averaging, as input. With such an approach, we have observed that BM3D-AF is 7.25 times faster than BM3D-MF for a 4×640×480 sized dataset. It consumes just 1.82 seconds for the filtering process after motion compensation, despite employing a naive patch matching algorithm. Also, the CPU 2 implemetation of BM3D-AF is over 50 times faster than NLB-CF, which is a current state-of-the-art technique.

We have optimized the usage of NLB and BM3D filters for the multi-frame scenario. We can conclude from the experiments that our proposed following sequential process gives the best results in most cases: They register the images with robust optical flow methods, temporally average the registered noisy images, and then apply the single-frame filters with optimal parameters corresponding to the new noise distribution after temporal averaging. This is true for both NLB and BM3D, an observation which has surprisingly not been recognized for many years. This re-affirms the fact that sometimes the simpler solutions are the most powerful ones and can also be competitive with sophisticated neural network architectures. Furthermore, we achieve this significant quality improvement at the cost of zero additional parameters and far less computational time. The technique also preserves a large amount of detail even when the images are corrupted with noise of very high amplitude. Thus, the category-AF extensions in combination with robust optical flow methods can be employed in practice for many multi-frame image processing applications.

Combining BM3D-AF and NLB-AF with variance stabilizing transformations, deflickering, sharpening and occlusion handling techniques will be considered in our future research. We will also use type-AF extensions as regularizers in PDEs for robust image reconstruction applications; c.f. [51] [52] [53] [54] 

The transformation of Poisson, binomial and negative-binomial data

Optimal inversion of the Anscombe transformation in low-count Poisson image denoising

Optimal inversion of the generalized Anscombe transformation for Poisson-Gaussian noise

A non-local algorithm for image denoising

A review of image denoising algorithms, with a new one

Image denoising by sparse 3-d transform-domain collaborative filtering

An analysis and implementation of the BM3D image denoising method

A nonlocal Bayesian image denoising algorithm

Implementation of the non-local Bayes (NL-Bayes) image denoising algorithm

Nonparametric neighborhood statistics for MRI denoising

Feature-preserving MRI denoising: a nonparametric empirical Bayes approach

A tour of modern image filtering: New insights and methods, both practical and theoretical

Nonlocality-reinforced convolutional neural networks for image denoising

Fast Haar-wavelet denoising of multidimensional fluorescence microscopy data

Multiframe SURE-LET denoising of timelapse fluorescence microscopy images

Denoising low-dose CT images using multiframe blind source separation and block matching filter

Patch-based nonlocal functional for denoising fluorescence microscopy image sequences

Low-rank tensor approximation with Laplacian scale mixture modeling for multiframe image denoising

Robust tensor approximation with Laplacian scale mixture modeling for multiframe image and video denoising

A patch-based low-rank tensor approximation model for multiframe image denoising

Sparsity based denoising of spectral domain optical coherence tomography images

Efficient joint Poisson-Gauss restoration using multi-frame L2-relaxed-L0 analysis-based sparsity

Multiple view image denoising

Denoising image sequences does not require motion estimation

Multi-frame image denoising and stabilization

Nonlocal transform-domain filter for volumetric data denoising and reconstruction

Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms

Video denoising via empirical Bayesian estimation of space-time patches

A note on multi-image denoising

Multi Image Noise Estimation and Denoising

Patch-based video denoising with optical flow estimation

Video denoising by sparse 3d transform-domain collaborative filtering

Enhancing patch-based methods with inter-frame connectivity for denoising multi-frame images

A non-local CNN for video denoising

Video denoising by combining patch search and CNNs

Deep burst denoising

Burst denoising with kernel prediction networks

A PDE model for computing the optical flow

Regularization strategies for discontinuitypreserving optical flow methods

Robust discontinuity preserving optical flow methods

High accuracy optical flow estimation based on a theory for warping

A duality based approach for realtime TV-L1 optical flow

A database and evaluation methodology for optical flow

Dense 3D reconstruction with a hand-held camera

Image denoising by sparse 3-D transformdomain collaborative filtering

Image processing using smooth ordering of its patches

Electron Tomography: Methods for Three-dimensional Visualisation for Structures in the Cell

Poisson noise removal using multi-frame 3D block matching

A closed-form approximation of the exact unbiased inverse of the Anscombe variance-stabilizing transformation

Variance stabilization for noisy+estimate combination in iterative Poisson denoising

Motion-compensated spatio-temporal filtering for multi-image and multimodal super-resolution

Enhancement of noisy and compressed videos by optical flow and non-local denoising

Evaluating data terms for variational multi-frame superresolution," in Scale Space and Variational Methods in Computer Vision

Hough based evolutions for enhancing structures in 3D electron microscopy

After working as a high school physics teacher in Hyderabad, India, for about an year, he joined the Saarbrücken Graduate School of Computer Science in Germany. Here, he first completed graduate level coursework and is now a Ph.D. student at the Mathematical Image Analysis Group

Joachim Weickert has developed many models and efficient algorithms for image processing and computer vision using partial differential equations and variational methods. He has served in the editorial boards of ten international journals or book series and is Editor-in-Chief of the Journal of Mathematical Imaging and Vision

We thank Prof. Karen Egiazarian from Tampere University, Finland. A valuable discussion with him has helped to improve the evaluation part of this work. We also thank Dr. Matthias Augustin and Dr. Pascal Peter for useful comments on a draft version of the paper.