Free-Hand Stroke Approximation for Intelligent Sketching Systems

Free-hand sketching is used extensively during the early design phases as an important tool for conveying ideas, guiding the thought process, and serving as documentation [5]. It also provides a natural way of interaction desirable in the context of E21 and intelligent design spaces. Unfortunately there is little computer support for sketching. The first step in building a sketch understanding system is generating more meaningful descriptions of free-hand strokes. We describe a system that takes strokes described by an array of points along with timing data, and generates such concise descriptions in terms of geometric primitives such as circles, polylines, curves and their combinations. 1 System description 1.1 Issues Loss of information due to digitization, noise in the data, imprecision due to freehand sketching and sparseness of sampled data points – which may be as low as 4-5 dpi as opposed to scanned drawings that may have 1200-2400 dpi – complicate the stroke approximation task. 1.2 Feature point detection Stroke processing starts by looking for vertices. We want to avoid picking points on the curved regions as much as possible. Vertex localization is a frequent subject in the extensive literature on graphics recognition (e.g., [4] compares 21 methods). However these methods produce piecewise linear approximations. 1 Piecewise linear approximation algorithms don’t satisfy this requirement. 644 Tevfik Metin Sezgin Our approach takes advantage of the interactive nature of sketching by combining information from both curvature and speed data for detecting corners while avoiding a piecewise linear approximation. Feature points are indicated by the maxima of curvature data and the minima of the pen speed. Both speed and curvature data are noisy and simply picking the extrema introduces many false positives. Below we describe two methods for selecting the extrema in each case. Average based filtering The basic idea in average based filtering is confining our search for the vertices to only those regions where the curvature/speed data is above/below a threshold computed by the mean of the data. Then we search for the global extremum within each region to find the feature points. Scale space filtering Our experiments show that the above method is satisfactory for settings where noise is relatively small but doesn’t deal with extremely noisy data as in Fig. 1 very well, because it doesn’t differentiate between vertices due to the fine scale structure of the noise and those due to the high level structure of the stroke. The scale space based method deals with this problem by looking at the number of feature points present at different scales in the scale space. As seen in Fig. 2, the feature count graph has two distinct regions where the feature count drops with different rates. This behavior is typical for freehand strokes. Our task is selecting a scale where most of the noise is filtered out. This is done by modeling the feature count graph by fitting two lines to it: one to the region with the steep drop corresponding to places where the feature points due to noise disappear and the other to the flatter region where real feature points disappear. We take the scale corresponding to the intersection of these two lines as our scale. The results obtained by applying this technique to curvature and speed data are in Fig. 3. Generating hybrid fits The above methods separately may miss feature points, so we combine them to generate hybrid fits. We start the hybrid fit generation by letting the intersection of the generated fits 2 From this point on, by curvature we will refer to the absolute value of the curvature data defined as |∂d/∂s| where d is the angle between the tangent to the curve at a point and the x axis and s is the distance traveled along the curve. Free-Hand Stroke Approximation 645 Fig. 1. A very noisy stroke along with the fits generated using curvature and speed data with average based filtering. Both fits have picked many false positives due to noise. 0 50 100 150 200 250 300 350 400 0 100 200 300 400 500 600 700 800 0 100 200 300 400 500 600 700 800 0 20 40 60 80 100 120 Fig. 2. The scale space and feature count graph for the noisy stroke in Fig. 1. Fig. 3. Curvature and speed fits generated using the scale space approach. These fits contain 9 vertices for the curvature and 7 vertices for the speed fit compared to 69 and 82 vertices for the fits obtained using average based filtering in Fig. 1. 646 Tevfik Metin Sezgin to be our initial hybrid fit. At each iteration, we form a list of candidate vertices by picking the vertex with the highest certainty from each of the original fits. We augment the most recent hybrid fit by the candidate from this list that reduces the least squares error of the fit by the largest amount. 1.3 Handling curves Our system also supports strokes that may contain curves. First we detect curved regions by comparing the curve length l1 between the consecutive detected feature points to the Euclidean distance l2 between the two points. l2/l1 is significantly larger than 1 in curved regions. The curved regions are then approximated by Bézier curves. 2 Evaluation For purposes of evaluation, we built a higher level recognizer that takes the output of our system and interprets strokes in the mechanical engineering domain. Fig. 4 shows a number of free hand sketches and the output of the recognizer. Fig. 4. Input-output figures for the rough sketch of the direction reversal mechanism of a walkman. 3 Related work Related work can be found in [2, 1, 6, 3] but these systems either don’t support drawing arbitrary shapes or don’t do automatic vertex detec3 Some domain specific objects such as springs and ground symbols are recognized by the recognizer. Free-Hand Stroke Approximation 647 tion. Some of them require drawing modes making them unnatural freehand sketching interfaces.