Inking in Internet Explorer

0
139

InkInIESmall.pngOver the last year or so I’ve experimented with various ways to add or integrate ink into Internet Explorer. The SearchTIP is one of these projects. It’s a embedded control that contains an inkable canvas in which you can handwrite search queries. I wrote it because I wanted a way to Google without having to use a keyboard–something that I hoped would be useful on a slate Tablet PC–nor require the extra steps of using a pop-up TIP.

The downside about the SearchTIP is that the control is loaded when the page is displayed–which takes time. Hosting an inkable control like this is reasonable when a single query string or inkable surface is required, but what do you do if you want ink in a multi-field form in a browser page? This is still doable with a .NET solution, however, the more fields you get and the more interaction that is needed, the more you might as well think in terms of a rich client and/or a one-touch deployable solution.

But what if you have a handful of fields in an existing web page? Is there another way to fill in the form using recognized ink? Is the TIP the best approach? The TIP is definitely nice, particularly because of its correction features. But what else can be done.

Well, here’s another path I’ve been following. If you follow this link, you’ll see a Camtasia screencapture of an Internet Explorer extension that I wrote that is being used here to provide handwriting recogntion of ink placed over HTML edit fields. The recognition is constrained for each field based on external “rules” that I’ve defined for the page being displayed. The rules are currently stored locally, however, they could be on a remote server, embedded in the HTML, or possibly inferred.

Anyway, on the main Google page shown in the screencast you can see me handwriting the phrase “notice of address change”. The recognized text is placed in the edit field below the handwriting.

What’s more interesting, is the form which the Google query leads us to. Here there are several fields, such as the Name, Address, City, State, and ZIP, in a change of address form. To improve the quality of the recognition for each of these fields, additional recognition “rules” are applied to each field. The results are pretty good as you can see in the screencast. In many respects, this is similar to Microsoft’s context tagging tool, which is useful for providing context to existing dialog boxes.

There’s no app to release yet. The code is still too unstable to pass around. IE is a tempermental app. Further, there are several more events I need to figure out how to track within IE in order to make the experience natural.

So far, I like this path a lot. The next big hurdle is to figure a way to tie in a correction mechanism and possibly a list of alternates. As always, I’ll let you know how it goes.


^

1 COMMENT

  1. Loren you are a God and are quite welcome to eat my first-born child (if I acquire one). Please please please make finishing this your priority over and above working eating sleeping and breathing – it’s what I’ve wanted from my Tablet since I first heard about them.

  2. Chris, no need to give up your first born. This is just programming after all.

    In terms of priority, yeah I can see where this would be quite useful. I’ll see if I can sacrifice a couple hours sleep here and there to get this done, but food, on the other hand, I can’t do without. 🙂 I need my brain food.

    Ctitanic, thanks!

    Loren

  3. Doug, there are a couple things that may help.

    First, the SearchTIP requires the .NET 1,1 Framework on your system. Also, if you are running .NET 2.0, there is an update for ink that you’ll need. You’ll have to manually select this update from Windows update–it’s not part of the automatic updates.

    Second, you’ll need Internet Explorer to view the SearchTIP.

    Third, the most common problem people are having is with permissions within IE. The simplest approach is to add http://www.TabletPCBlogs.com as a trusted site within Tools | Internet Options | Security tab.

  4. Loren, what would be really cool is an app that allowed one to ink on anything.
    Sounds complex but how about this mechanism;
    Kind of like writing on a transparent layer on top of any app.

    Pressing a taskbar button activates the ink layer, deactivating the button, makes the app act in its normal fashion, while still seeing the ink layer on top.

    Do you think this is possible?
    Would other readers here find such funtionality desirable?

  5. Byron, a “write anywhere” solution makes sense. Several other people have asked for this too. I’ll see what I can do.

    The trick is in providing a good correction UI. The TIP is wonderful in this respect. The ink in IE tool shown in this post has similar issues.

    Casey, thanks!

  6. EverNote’s ritePen and OrangeGuava’s Desktop both provide a write anywhere feature that’s very much worth checking out. Thanks Kenneth for the link.

    The ink in IE sample shown here, however, is similar to write anywhere but the intent and implementation is a bit different. I’ll see about putting some more screencasts together so you can see how it’s different.