Automatic Isolated-word Arabic Sign Language Recognition System Based on Time Delay Neural Networks: New Improvements

This research presents an improved automatic isolated-word recognition system for the Arabic sign language (ArSL) for the Jordanian accent. Our proposed system requires that the signer wears two gloves with different colors, he also should wear another colored mark on his head, this mark should have different color than the colors used in the gloves. In this paper the video for each sign is converted to a sequence of static images; each image is segmented for three colored regions and outlier according to the mean and covariance of each color region using the multivariate Gaussian Mixture Model (GMM) on the characteristics of the the Hue Saturation Lightness Model (HSL) color space. Tracking the hand motion trajectories of the right and left hand over time is conducted . Finally we identify a list of features to be used as an input to the time delay neural networks (TDNN) for the recognition step. Two different test collections were used in this research, the first data collection is used to prove that when using the head of the signer to determine the position of the centroid for the right and left hand instead of using the center of the upper area for each frame as reference can improve the recognition rate. The experimental results shows an improvement in the recognition rate from 70% to 71.66%. finally we used the second data collection to prove that our proposed categorization for the Arabic signs into four categories according to the overlap existence between head and hands can improve the recognition rate. The experimental results based on data collection number two shows an improvement in the recognition rate for the testing set, where the recognition rate increased to reach 77.43%.