Handwriting Recognition via Visual Observation of Pen Movements

Abstract

Handwriting recognition is a common problem in applications such as converting handwritten documents to digital documents and authentication. Existing approaches utilize the handwritten document as the most important clue in recognition process. In this work secondary observation is used instead of the written document. Secondary observation becomes the only source of information when handwritten document cannot be observed directly or additional clues are needed for authentication. Secondary observation refers to the observation of the movement of the pen or the hand during writing. Three deep learning models are proposed to predict what is written by observing only the pen movement. The proposed solutions are the first for this problem. The result of the experiments presents a comparison of the success of three proposed models and shows that secondary observation is useful to solve the problem. Among the proposed models, the best results are obtained by using LSTM architecture which is frequently used to learn from temporal data.