Comparative Analysis of Structure and Texture based Image Inpainting Techniques

There are various real world situations where, a portion of the image is lost or damaged or hidden by an unwanted object which needs an image restoration. Digital Image Inpainting is a technique which addresses such an issue. Inpainting techniques are based on interpolation, diffusion or exemplar based concepts. This paper briefly describes the application of such concepts for inpainting and provides their detailed performance analysis. It is observed that the performance of these techniques vary while restoring the structure and texture present in an image. This paper gives the limitations of each technique and suggests the choice of appropriate technique for a given scenario. Keywords—digital image inpainting, exemplar based inpainting, TV inpainting, isotropic diffusion, anisotropic diffusion. 1-INTRODUCTION A Photographic picture is a two dimensional image which can contain many objects. One may be interested in the object or scene that is hidden by another. For example, a beautiful picture may contain some letters written on it or a view of the Taj mahal maybe occluded or a historic painting may be torn or damaged. Here the picture below the letters, the occluded portion of the Taj mahal and the damaged portion of the painting needs to be restored. This problem is addressed under various headings like disocclusion, Object Removal, Image Inpainting etc. Retrieving the information that is hidden or missing becomes difficult when there is no prior knowledge or reference image. Here the information surrounding the missing area and other known area has to be utilized for the restoration. Usually, the user in the form of mask specifies the unwanted foreground or the object to be removed or the portion of image to be retrieved. Clone Brush tool of Adobe Photoshop restores the image when a sample of the image to be placed in the missing area is selected by the user whereas in inpainting the missing area is automatically filled in by the algorithm. Digital image inpainting is a kind of digital image processing that modifies a portion of the image based on the surrounding area in an undetectable way. The techniques rely majorly on the diffusion and the sampling process. It has a wide variety of applications in restoration of deteriorated photos, denoising images, creating special effects in movies, digital zoom-in and edge based image compression. 2. STATE OF ART The inpainting problem can be considered as assigning the gray levels to the missing area called as Ω with the help of gray levels in the known area Φ as shown in fig. 2.1. The boundary δΩ, between the two plays a major role in deciding the intensities in Ω. All the algorithms are iterative and try to fill in δΩ first and moves inwards successively altering δΩ each time. The algorithm stops when all the pixels in Ω are successfully assigned some values. The restoration of the structural information like edges or textural information like repeating patterns pose a major challenge for the inpainting techniques. Based on the nature of filling, the algorithms could be classified into structure based and texture based methods. 1063 Comparative Analysis of Structure and Texture based Image Inpainting Techniques ISSN-2277-1956/V1N3-1062-1069 Figure2.1. Digital image Inpainting problem Structure based methods uses geodesic curves and the points of isophotes arriving at the boundary for inpainting. Isophotes are the lines joining the same gray levels and geodesic curves are lines following the shortest possible paths between two points. When used in its primitive form it may result in disconnected objects. This is illustrated in Fig 2.2; while inpainting the black square in Fig 2.2a, a horizontal bar is expected but the algorithm results in two disconnected bars as in Fig 2.2b. The mathematical models for deterministic and variational PDE are explained in detail in [4] and [6]. A series of Partial differential equations are used to extend isophotes in to the missing area in [1],[2] and[11]. In [12] a convolution mask is used to extend the gray levels in to the inpainting area. The curvatures are extended into the inpainting area in [5]. The Texture based methods mainly rely on texture synthesis, [3] and [9] which grow a new image outward from an initial seed. Before a pixel is synthesized, its neighbors are sampled. Then the whole image is queried to find out a source pixel with similar neighbors. At this point, the source pixel is copied to the pixel to be synthesized which is the missing area. This is called as Exemplar based synthesis. Based on whether a pixel or a sub window is used for sampling it is further classified as pixel based sampling and patch based sampling. The patch size, matching criteria and order of filling varies between algorithms. Exemplar based inpainting is used in [7] and [15]. 3. DIGITAL IMAGE INPAINTING TECHNIQUES The digital image inpainting involves two major steps. First step involves the selection of area to be inpainted and the second is the inpainting algorithm which gives appropriate values for the selected area. 3.1 Inpainting area selection The area to be inpainted is selected by the user based on color, region selected by user or a binary image specifying the missing area. Color based selection is more flexible and it could be used for specifying the area irrespective of the shape, area and number of regions. Instead of looking for the exact color value, the color values closer to it is also taken, into account for the quantization effects. This method requires the missing area to be in a unique and different color from the rest of the image. The user can select the missing area through a free hand selection or polygon selection. This method is capable of selecting the missing area irrespective of the color. This could be used predominantly on black and white images. However the missing area cannot be precisely specified in this method. It becomes tedious to select more than one area as in the case of imposed text on the image. If the area to be inpainted remains constant across various images or the template of the damage is known, the missing area is specified in the form of a binary image with the same size of the input image. This method is best suited to specify the black text imposed on black and white images. In practice the missing area is selected using any image manipulation software and given a different color which is then used for the inpainting algorithm. The user selected area is usually called as the mask or the region to be inpainted. 3.2 Structure based inpainting These methods are based on the Partial differential equations which contribute to the structural information in an image. The differential equations which use the concept of Interpolation, Diffusion and Total Variational PDEs are discussed in this paper. IJECSE,Volume1,Number 3 S. Padmavathi and K. P. Soman ISSN-2277-1956/V1N3-1062-1069 3.2.1 Interpolation Based Inpainting The simplest method uses soap film PDE, where δΩ becomes its boundary conditions. A set of linear equations are formed with the known values of δΩ and unknown values of Ω in four major direction namely the north, south, east and west. Interpolation of the four neighbors is used to frame the equation. The δΩ forms the right hand side of the equations. The equations are solved to get the intensities of Ω. 3.2.2 Anisotropic Diffusion Based Inpainting Inpainting problem is considered as diffusion of gray levels from the boundary area δΩ into the unknown area Ω. The level set theory used to explain the diffusion boundaries during various periods. If the diffusion process does not depend on the direction or the presence of edges, it is called as isotropic diffusion. Interpolation technique is isotropic in this sense. Anisotropic diffusion[13] is used to avoid blurring across edges. Equation 3.1 shows the anisotropic diffusion where g represents a smooth function, K represents the curvature, ∇ I represent the gradient of the image and Ω represents the area other than Ω. The curvature is given by the equation 3.2.