Deep learning-based single image super-resolution for low-field MR brain images

BackgroundGiven a low-resolution 2D image ({mathbf {y}}), our purpose is to accumulate its high-resolution counterpart ({mathbf {x}}). The relationship between ({mathbf {x}}) and ({mathbf {y}}) may be modeled as follows:$$start{aligned} {mathbf {y}}= {varvec{{mathscr {F}}}}_{LR}^{-1}{mathbf {D}}{varvec{{mathscr {F}}}}_{HR}{mathbf {x}}+ {mathbf {n}}, finish{aligned}$$
the place ({varvec{{mathscr {F}}}}_{LR}^{-1}) is the inverse FFT operator utilized within the LR regime, ({mathbf {D}}) is an operator that selects solely the low-frequency elements in k-space, which, in our case, yields a matrix of measurement (64times 64), ({varvec{{mathscr {F}}}}_{HR}) is the FFT operator for the HR regime ((128times 128)) and ({mathbf {n}}) is an (unknown) noise vector. The aim of super-resolution is to seek out an approximate inverse of the operator ({varvec{{mathscr {F}}}}_{LR}^{-1}{mathbf {D}}{varvec{{mathscr {F}}}}_{HR}). We be aware that the usual super-resolution drawback is mostly posed in a different way, i.e., the HR image is assumed to endure blurring and downsampling, culminating in an LR image. However, utilizing Eq. (1) follows the low-resolution MRI acquisition course of extra precisely.Convolutional neural communityWe selected a convolutional neural community of the SRDenseNet structure for our software. Our alternative was motivated by SRDenseNet’s good efficiency, mixed with its manageable variety of parameters. We be aware that, as there’s a huge literature on deep learning-based strategies for super-resolution, different networks could also be relevant as well24,26,27. SRDenseNet was launched by Tong et al.25 It consists of blocks of densely linked convolutional layers (“dense blocks”). In each dense block, which is in line with the DenseNet architecture33, every convolutional layer receives as enter the concatenated outputs of all previous convolutional layers, as proven in Fig. 1. By reusing characteristic maps on this manner the training of redundant options is prevented. Instead, the present layer is pressured to study supplemental data. As within the unique paper, we are going to use 8 dense blocks of 8 convolutional layers every, the place every convolutional layer produces 16 characteristic maps, which signifies that every dense block yields 128 characteristic maps. In every convolutional layer, the kernel measurement is 3×3. After the ultimate dense block, a bottleneck layer with convolutional kernels of measurement 1×1 is used to scale back the variety of characteristic maps to 256, adopted by a transpose convolutional layer (which is commonly known as a deconvolution layer) which upsamples the image to HR house. Note that on this work, the upsampling issue is the same as 2 and therefore we use just one single transpose convolutional layer with a stride of two, versus the two transpose convolutional layers within the unique SRDenseNet which was used for an upsampling issue of 4. Finally, one other convolutional layer with a 3×3 kernel is utilized to scale back the output to a single channel. All layers besides for the ultimate convolutional layer use a nonlinear ReLU (Rectified Linear Unit) activation perform. Additionally, skip connections are employed to feed the output of every dense block to every of the next dense blocks, as is in line with the SRDenseNet_All structure showcased within the unique paper25. The full structure, which has 1,910,689 trainable parameters, is proven in Fig. 2.Figure 1A dense block, which is a basic part of the SRDenseNet structure, incorporates eight convolutional layers that obtain the outputs of all previous layers as enter.Figure 2The SRDenseNet convolutional neural network25 that’s used to hold out single image super-resolution on low-field MR images.Dataset and coachingIn this work, we centered on 2D images, nevertheless it must be famous that this method may be prolonged to 3D. We generated a coaching and validation set utilizing 2D images obtained from the publicly out there NYU fastMRI Initiative database (,35. As such, NYU fastMRI investigators supplied information however didn’t take part in evaluation or writing of this manuscript. A list of NYU fastMRI investigators, topic to updates, may be discovered on the aforementioned web site. The main aim of fastMRI is to check whether or not machine studying can assist within the reconstruction of medical images. The database consists of slices of T1-weighted, T2-weighted and FLAIR (fluid-attenuated inversion restoration) images, acquired utilizing 1.5 T and three T MRI scanners. By coaching on such a wide range of MR brain images, the ensuing community must be relevant to images acquired utilizing totally different sequences as properly, with out the necessity to re-train the community every time the parameter settings change. We be aware that, even when we have been planning on making use of the skilled community to, for instance, T1-weighted low-field MR images solely, it might nonetheless make sense to coach the community on high-field MR images acquired utilizing totally different sorts of sequences, making it adaptable to totally different sorts of enter. The cause for that is that the relief instances range with subject energy and therefore, a T1-weighted image acquired utilizing a low-field scanner may look totally different from one acquired utilizing a high-field scanner. One parameter to watch out with, although, is the image measurement. We will use enter images and output images of measurement (64times 64) and (128times 128), respectively. Because of the purely convolutional nature of the community, it’s doable to make use of images of a special measurement as enter. The community ought to be capable to accommodate small deviations in measurement. However, it’s unlikely that it might generalize to images that deviate considerably in measurement from the images within the coaching set.The images within the database have totally different sizes. As we’re focused on HR images of (128times 128) pixels, all images have been resized to (128times 128) pixels. This was completed through the use of an FFT to transform the images to k-space information, deciding on the central a part of k-space and subsequently making use of an inverse Fast Fourier Transform (FFT), as in Eq. (1). After that, we downsample these HR images to LR images of (64times 64) pixels, by, once more, utilizing Eq. (1), i.e., we use an FFT to transform the image to k-space, choose the central a part of k-space (of measurement (64times 64)) and apply an inverse FFT to acquire an LR image. To acquire noisy LR images, we add advanced Gaussian noise in k-space, with the noise stage various from image to image. We used a variety of noise ranges in line with the low-field MR images we’ve got seen in apply. This step is critical to make the convolutional neural community generalize to images acquired utilizing a low-field MRI scanner, which, because of the weaker magnetic subject, yields indicators with a comparatively low SNR36. In this manner, 29,059 and 17,292 image pairs have been obtained from the coaching and validation units which are supplied within the dataset, respectively. We assigned 10,000 of the 17,292 image pairs within the validation set to our personal validation set, and the opposite 7292 to our take a look at set. Some examples of image pairs current within the coaching set are proven in Fig. 3. We be aware that the information was break up on the affected person stage, and therefore, no information leakage occurred.Figure 3Examples of HR-LR image pairs within the coaching set. The first row incorporates 4 totally different HR images, with the white squares denoting patches whose zoomed-in variations are proven within the second row. In the third row, the corresponding (noisy) LR images are proven, with the fourth row containing LR variations of the patches within the second row.Since SRDenseNet is a purely convolutional neural community, it’s doable to coach on patches as a substitute of full images, which requires much less reminiscence throughout coaching. Furthermore, utilizing patches permits us to generate extra information. Therefore, we used the HR-LR image pairs to create 190,000 pairs of patches to coach the community on, and 10,000 pairs of patches for validation, the HR and their corresponding LR patches having a measurement of (32times 32) pixels and (16times 16) pixels, respectively.The convolutional neural community was carried out in TensorFlow37. The Adam optimizer38 with a studying fee of (10^{-3}) was used to reduce the mean-squared error loss between the community output and the mannequin HR image patches. Additionally, we investigated two totally different loss features: (ell _1)-loss and HFEN (High-Frequency Error Norm) loss39. However, after visible inspection of the ensuing images, we discovered that the mean-squared error loss outperformed the others. We used a batch measurement of 20 and a complete variety of epochs of 74 as a result of this corresponded to the smallest worth of the validation loss. The coaching was carried out on a Titan X Geforce GPU (12GB) and took about 5 hours.Low-field MR image acquisitionTwo three-dimensional in vivo scans of the brains of two wholesome volunteers have been acquired utilizing the low-field MRI scanner described by O’Reilly et al.9 We will use totally different (2D) slices of the ensuing 3D images as our community enter. Both experiments have been carried out utilizing a turbo spin echo sequence. For the primary experiment, the next parameters have been used: FoV (subject of view) (224times 224times 175) (hbox {mm}^3), voxel measurement (1.75 instances 1.75 instances 3.5) (hbox {mm}^3), (T_R)/(T_E) (repetition time/echo time) = 500 ms/20 ms, echo practice size 4, acquisition bandwidth 20 kHz, no sign averaging, cylindrical k-space protection. The second experiment was carried out with a special set of parameters: FoV (180times 240 instances 180) (hbox {mm}^3), (1.5 instances 1.5 instances 3) (hbox {mm}^3), (T_R)/(T_E) = 400 ms/20 ms, echo practice size 5, acquisition bandwidth 20 kHz, no sign averaging. All strategies have been carried out in accordance with related pointers and rules.

Recommended For You