Automatic Number Plate Segmentation and Recognition System

The authenticity of the driver & car is a key problem today because as the population of today grows, so do the number of automobiles on the road. Each car will require a significant amount of labor & time to be manually entered, and even then, accuracy is not guaranteed at 100%. We create a software system called Automatic Number Plate Recognition (ANPR) that automatically detects the number plate and records it in the database in order to solve this problem. This technology aids with traffic management, & security is one of its main uses. It aids in the monitoring of vehicles on the road as well as the access control of undesirable vehicles in zones & regions that are restricted. The nature of light makes it difficult to identify a license plate. This system looks into a portion of the input image that has a license plate. Since plate numbers might appear everywhere on an input image, it is challenging to check every area of the image. In the ANPR system, a method known as spectral analysis is utilized to acquire the image, extract the number plate region from the input image, and then match each extracted letter with the letters recorded in the system database using character segmentation.

This technique's primary benefit is its ability to capture moving vehicle images & send them for segmentation &recognition[1]. Now, if the registered number plate in the database matches the output number plate, it is permitted to proceed; alternatively, it is prohibited. This technique is applicable to toll booths, parking areas, society, & universities. Due to the color of the number plate characters & their background, it is difficult to distinguish a number plate from input photographs of a car in an open area. To identify potential number plate regions, gradients from real photos are first adopted[2]. As of right now, our algorithms rely on segmentation, morphological operation, &Canny edge detector. The algorithm for locating license plates includes a number of phases, including edge detection & morphological operations like dilatation & erosion. [3-6] address smoothing, character segmentation, & plate characters recognition.

The success of extracting number plates from the scene or image is absolutely necessary for the character segmentation stage. The solitary license plate may have problems with contrast, different lighting circumstances, or it may be angled differently. Before segmenting the characters in such a case, pre-processing procedures, de-skewing, de-blurring, or any other approaches might have to be used, depending on the circumstances of the number plate. Depending on the methodology used, this step may be completed either during the extraction stage or after obtaining an isolated candidate area. The isolated number plate is mapping onto a straight rectangular shape in [7], a preprocessing technique that deals with slanted license plate photos. In [8], the least square method is used to correct tilted license plates; it takes into account both vertical and horizontal tilts. The Karhunen-Loeve (K-L) transformation is used in [9] to arrange the character coordinates into a 2D variance matrix. The tilt of the image is then corrected for, both vertically & horizontally, by computing the angle of rotation or Eigenvectors. To calculate the angle of vertical tilt, three methods are suggested: K-means cluster based line fitting; least squares based line fitters; and K-L transform. Although the threshold application appears straightforward when converted to a binary image, this is actually a very difficult step in the process. A poor threshold value could lead to connected characters, either between characters or with the frame of the license plate, which would be challenging to segment [8]. Due to variations in image settings and illumination, a single value threshold may not be suitable for all photographs.

It is required to improve the image before binarizing it. The process of improving an image may involve removing image noise, boosting contrast, or using histogram equalization techniques. In [10], a method to feel the license plate following by the augmentation of the plate by grey color modification was proposed in order to execute gradient analysis across the full image. The Niblackbinarization technique in [11] modifies the picture threshold in accordance with the local mean & standard deviation. In [12], a local threshold approach is used for each pixel. The average grey levels in an m-n window positioned in the middle of the pixel are subtracted from the supplied constant value to get the threshold value.

A novel approach to lessen noise and character improvement was put out in [13]. It was estimated that the character size would be roughly 20% of the size of the license plate. The range of the grey scale's intensity is initially 0-100. Then, a scaling factor of 2.55 is applied to the 20% larger pixels. Characters are strengthened while noise pixels are reduced. The approach of adaptive local binarization is used since the image binarization cannot produce good results with a single global threshold. Based on the features employed, number plate segmentation techniques are discussed below.

NP Segmentation Utilizing Connected Components

In [14], segmentation is accomplished through pixel connection. In [15], connected component labeling is used to test 958 High Definition (HD) photos under various situations and achieve segmentation accuracy of 99.75%. Based on their connectivity, the character pixels in the binary image are identified, as their aspect ratio and size are compared to those of the number plate characters. For linked or broken characters, this strategy does not appear to be effective. A small dataset of 50 photos was evaluated in [16] utilizing linked component labeling and the morphological technique, yielding a 91% segmentation rate. Utilizing connected components analysis & HD photos in a range of lighting & weather circumstances, an accuracy rate of 99.5% was noted in [17]. To obtain the numbers in [18], the extracted plate area is binarized and labeled. Several label layout patterns were applied to help the labeled segment be recognized as a number. The system's overall recognition rate was 99%. For segmentation, [19] proposed a mix of connected components & blob coloring approaches. Their system's accuracy rate was 93.7%.

NP Segmentation Utilizing Vertical/Horizontal Projection

Different colors are used for the characters and backdrop of a license plate. The generated binary image has various values for the number plate's character & background. For character segmentation, [19] uses pixel projection in both the vertical and horizontal dimensions. In [20], projection techniques are employed. The binary number plate is projected vertically, then horizontally to extract individual characters by analyzing the start and end positions of each character. Vertical projection is used for character extraction in [21] in addition to noise removal examination of the character sequence. This method can process over 30,000 photos at a processing speed of 10–20 ms with an accuracy of up to 99.2%. The [22] profile projection method is examined using a database of 560 pictures. With an efficiency of detecting several number plates displayed in a single image, the segmentation rate of 95.4% was attained. After review, it is clear that the most straightforward and widely used method is the one that uses both horizontal and vertical pixel projections. Since the segmentation of characters using projection techniques is not position-dependent, the results are encouraging. The character count must be known in advance, though. The projection values may be influenced by noise & image quality.

NP Segmentation Utilizing Characters Features

Characters are easier to identify with basic understanding, which helps with vehicle plate segmentation. The RGB color extractor is used in [23] to do character isolation. For 255 color images tested, the segmentation rate is 98.5%. Segmentation is carried out in [25] using YOLO models, YOLOv2, Fast-YOLO, & Classification-Regression Network (CR-NET) [24], that are based on neural networks. [26] scans the binary image horizontally to identify the positions of the starting & terminating characters. The character's commencement point is considered to be when the ratio of character pixels to background pixels crosses the threshold level after briefly falling below it. According to Paliy et al. [27], resizing the retrieved car plate into a predetermined template proportion was suggested. Character positions in this design are all known. After scaling, the same places act as the characters. This method is renowned for being easy to use. However, when number plates are taken and moved, the result is background rather than text. [28] suggests a potential fix for severely damaged license plates. The vehicle plate may be located from the image using the color combination. For character segmentation, each character dimension is taken into account. The layout of a Chinese license plate is utilized to build the recognition classifier. Complementary color schemes may be seen in [29], where Taiwanese license plates have a white backdrop and black letters. When the number plate is scanned horizontally, there may be maximum of 14 color transitions from white to black or conversely & minimum of 6. Applying the Hough transform fixes the rotation issue. A hybrid binarization technique is used to separate the characters on dirty license plates. To maintain the parameters, the feedback technique is lastly used. For their experiments, about 332 different photographs are taken at various distances and with various lighting conditions. Segmentation &localisation rates are 96.4% & 97.1% overall, correspondingly.

NP Segmentation Utilizing Boundary Information

The modeling of contours can also be used to segment characters. [30] employs vertical edge detection along with long edge removal. The technique is closed curved. [31] uses a vertical histogram to segment characters. [32] proposes a segmentation technique on an adaptive morphology approach for recovering severely damaged license plates. The sections are determined and combined using a histogram-related technique. The algorithm of morphological thickening is employed in [33] to identify the reference lines that separate the overlapping characters. The morphological thinning algorithm finds the starting point for connected character segmentation. An image set of 1189 degraded photos was accurately segmented into about 1005 images, for an 84.5% segmentation rate. A method was explicitly shown in [34] for segmenting the numerical characters on a license plate. This is accomplished by using some dynamic programming. By employing the bottom-up methodology of the algorithm, this method operates quickly. Additionally, by removing aspects like edges & color that depend on the environment, this technique operates effectively. 97.14% of attempts to identify primary numbers are successful.

Recognition of the segmented characters is the last step in ANPR systems. Due to zooming and camera distance, the segmented characters may vary in size & thickness [27]. Noise may break, skew, or affect the characters. This section discusses several character recognition techniques.

Character Recognition Using Template Matching

Template Matching is the most basic character recognition technique. It is a crosscorrelation method that is extracted character's similarity to the template characters set is assessed. The candidate with the most similarities to one of the characters from the set of templates is chosen. These techniques are frequently utilized for binary images since a change in lighting conditions has a direct impact on the gray level intensities in the final image. The segmented characters in [35] are recognized using the Template Matching Technique, which has a success rate of 98.1%, 96.37%, 93.07%, and 92.52% for four different sets of Moroccan format license plates. [40] also presents the template matching technique for use. The literature contains numerous similar techniques, including Bayes judgment, Jaccard Technique, Hamming distance, &Hausdorff Distance. Character recognition in [36] uses normalized cross correlation. By scanning the templates column by column, the retrieved characters are compared with them. The character that is most closely matched has the highest correlation value. Template Matching can only successfully recognize characters that have not been damaged, skewed, had their typefaces changed, or have been scaled to a fixed size. To solve this problem, [37] makes use of numerous templates that are saved for a single character, including a tilt factor that takes into account various orientations.

Over 1200 photos with a width of 250 pixels that were taken in a variety of lighting and color situations had a high extraction rate of 100% and a respectable recognition rate of 90% in [38]. Over 1300 license plate-sized photos with a fixed aspect ratio are used in [39] under a variety of real-time circumstances. The system's overall efficiency for recognition is 92.12%, with a processing time of 1.2 s at 10 FPS, and the extraction rate is effective at about 98.35%.

Character Recognition Using Extracted Features

All pixels are used in the template matching procedure, which lengthens the processing time. Character recognition also employs an alternative feature extraction technique that speeds up processing by removing insignificant pixels. In [40], character recognition is accomplished using SVM. An enhanced SVM-based approach is suggested in [41] for difficult and complex license plates. In [42], vertical & horizontal binary character projection is used to create the features vector. In [42], the projections were each quantized into four distinct levels. Each character in [43] undergoes the Hoteling transformation to create the characteristic vector. The segmentation output is highly responsive to this change. In [44], the production of a characteristic vector is accomplished by dividing a binary character into 24 blocks of 3 4 pixels, totaling 24 blocks with individual values. A predefined template of characters is compared to each matrix, and according to the template matching procedure, the closest match is chosen for each block. Two distinct sets of characters are created after they have been identified, and they are evaluated further employing trained neural networks to increase the system's recognition accuracy. For further improving the recognition rate, Euler number & positioning check is done. The average time required by this method to recognize a Botswana number plate format was 2.41 seconds. Vowels, slanted or skewed plates, and veiled or occluded plates presented challenges. The technique is quite similar to the one in [45], with the exception that in [45], the elements with inclination degrees of 0, 45, 90, and 135 are counted. In [46], the centrally situated axis is used to implement features scanning. The horizontally bound lower and upper central moments are connected by this axis. Then, a distinctive vector for each character is formed by the transition between the number plate characters & backdrop as well as the space between them. Character rotation is consistent with this method. The distinctive vector is created in [47] by showing how to shape the overall character. To obtain the feature vector, quantization is applied to the output waveform. Because the character shape does not alter when the font or size changes, this approach can recognize characters with many fonts and variable sizes. Character extraction is carried out in [48] using the Gabor filter. The highest filter response will be found on character edges with the same orientation angle as the filter. It has the ability to create character-specific characteristic vectors. Kirsch edge detection is used in [49] to extract characters pointing in various directions from the character picture. In comparison to previous edge detection techniques, such as Wallis, Prewitt, and Frei Chen, this detection method for character recognition and extraction produced more acceptable results [50]. In [51], a binary image is used to extract the characteristic vector, which is then thinned to convert the direction of the character strokes into a unique code. The grey level values of the pixels in the 11 sub-blocks are presented to the neural network classifier as the characters in [52]. According to the experiment done in [54], a scene is evaluated in [53] by getting to the non-overlapped 5 5 pixels blocks and processing the overall image data to extract "spread" edge characteristics. While [55] describes the sub-image categorization in terms of the coarse-to-fine recognition strategy. In [56], an SVM is employed to perceive classification using three characteristic parameters: peripheral backdrop area, contour-crossing, & directional counts.

Figure 1 shows a block diagram of the general ANPR system.

Figure 1. System Block Diagram

A. Vehicle Image Browse/Captured by Camera

Using a 3.2 MP digital camera, the image of the car whose license plate is to be recognized is captured.

Figure 2. Captured Image

B. Extraction of Number Plate Location

In this method, the RGB image, or the captured image, is first converted to a grayscale image before the number plate is extracted. In this case, the region is sensed using numerical morphology, & Sobel operation is utilized to determine the threshold value. We then see a distorted image. The holes are then filled with filler material to produce what appears to be a binary image.

C. Segmentation & Recognition of Plate Character

The segmentation process now uses bound box exercises. The inherits of this precise image region are computed using the bounding box. Observing the plate's component is a crucial step in recognizing a vehicle number plate. Now that the image has been improved, it has been converted to gray scale so that we can get cover.

D. Display Vehicle Number

After performing the above processes, MATLAB produces the corresponding license plate.

Figure 3. User Interface

NPR IMPLEMENTATION USING MATLAB

The following describes the entire process of utilizing MATLAB to operate NPR:

A. Input Image

The major goal of this stage is to capture an image. A precise 3.2 megapixel digital camera is utilized in this proposed system. Image input dimensions are 120 x 160, 1200 x 1600.

Figure 4. Input Image

B. Extraction of Number Plate Location

The camcorder-captured images of automobiles are the inserts to the system. With order to aid in character extraction and speed up dispensing, RGB to Gray-scale conversion is used. A digital camcorder's RGB-highlighted image is converted back to a grayscale image:

Gray=0.114*R+0.587*G+0.299*B………(1)

Figure 5. Extraction of Number Plate Location (I)

The first step in understanding a car number plate is to take note of its size. Number plates often have a rectangular shape, so it is important to be aware of its boundaries. The region of focus is identified using theoretical morphology, & threshold value utilized to identify high-light regions with high edge magnitude & high edge variance is determined using the Sobel operator.

Figure 6. Extraction of Number Plate Location (II)

Figure 7. Extraction of Number Plate Location (III)

Thus, using a binary gradient mask, we can extract the lines from the elevated contrasted images. However, the object of interest's precise outline is not provided. Different joining lines that surround the item in the gradient mask can be seen when contrasted to the unique portion. However, linear gaps can be eliminated if the Sobel image is expanded using linear structuring elements. We represent the structuring element as matrices, that are a sign of a reliable structure or help in calculating the size of an image that will support subsequent picture distribution processes. We enhance the binary gradient mask using the upright structuring ingredient & straight structuring constituent.

The MATLAB function imfill (BW, "holes") allows for the filling of holes in binarized images. The dilated gradient mask accurately depicts the cell's drawing, but there are still some holes visible inside the cell. Outlet is known as the location for positioning pixels that haven't split due to the backdrop matching the figure's boundaries. Figure 5.2 shows what will happen if less than 100 related pixels are removed. Therefore, the MATLAB imfill function is utilized to fill these gaps.

C. Remove Connected Objects on Border

Although the area of interest has now been divided into several segments, not only one single object has been established. Utilizing the MATLAB unclear edge function, any items connected to the edge can be removed. To avoid diagonal connections & fill the hole needed to establish the plate part, the connectivity of this function was already set to 4 or 8. Genuine plate sector is discovered after the small pixel mechanism is removed.

D. Character Segmentation

The most crucial module in number region detection is character segmentation since it serves as the foundation for all auxiliary procedures that depend on it. A character might be indecently split into two characters or half pieces if segmentation collapses. Therefore, a technique known as the boundary box technique is used to resolve this issue. The bounding box process is used to establish the image section's attributes. Each character & number on the number plate are separated out for identification once a bounding box has been created for each one that is accessible. The operation's outcome is displayed.

Figure 8. Character segmentation

The images need to be of high quality to get a high accuracy rate for license plate recognition. An image's quality can be impacted by a number of things. These include the camera's manufacture & model, resolution, light, & orientation when the input photographs were taken. The raw augmented ANPR camera data can be turned into useful information & used to better analyze traffic flow, such as passenger & commercial mobility, with the aid of the appropriate tools & software. ANPR cameras have the capacity to add information about the types of vehicles to their capabilities.