Over the last year or so I’ve experimented with various ways to add or integrate ink into Internet Explorer. The SearchTIP is one of these projects. It’s a embedded control that contains an inkable canvas in which you can handwrite search queries. I wrote it because I wanted a way to Google without having to use a keyboard–something that I hoped would be useful on a slate Tablet PC–nor require the extra steps of using a pop-up TIP.
The downside about the SearchTIP is that the control is loaded when the page is displayed–which takes time. Hosting an inkable control like this is reasonable when a single query string or inkable surface is required, but what do you do if you want ink in a multi-field form in a browser page? This is still doable with a .NET solution, however, the more fields you get and the more interaction that is needed, the more you might as well think in terms of a rich client and/or a one-touch deployable solution.
But what if you have a handful of fields in an existing web page? Is there another way to fill in the form using recognized ink? Is the TIP the best approach? The TIP is definitely nice, particularly because of its correction features. But what else can be done.
Well, here’s another path I’ve been following. If you follow this link, you’ll see a Camtasia screencapture of an Internet Explorer extension that I wrote that is being used here to provide handwriting recogntion of ink placed over HTML edit fields. The recognition is constrained for each field based on external “rules” that I’ve defined for the page being displayed. The rules are currently stored locally, however, they could be on a remote server, embedded in the HTML, or possibly inferred.
Anyway, on the main Google page shown in the screencast you can see me handwriting the phrase “notice of address change”. The recognized text is placed in the edit field below the handwriting.
What’s more interesting, is the form which the Google query leads us to. Here there are several fields, such as the Name, Address, City, State, and ZIP, in a change of address form. To improve the quality of the recognition for each of these fields, additional recognition “rules” are applied to each field. The results are pretty good as you can see in the screencast. In many respects, this is similar to Microsoft’s context tagging tool, which is useful for providing context to existing dialog boxes.
There’s no app to release yet. The code is still too unstable to pass around. IE is a tempermental app. Further, there are several more events I need to figure out how to track within IE in order to make the experience natural.
So far, I like this path a lot. The next big hurdle is to figure a way to tie in a correction mechanism and possibly a list of alternates. As always, I’ll let you know how it goes.