In case others trip over this questions. Yes, it's possible but painful. It'll take effort depending on the number of members signed up in WordPress.
Best process would be to copy all of the WP users into XF, making sure the user_id and ID were identical. Once done then enable XenWord, and XenForo takes over for login information, etc.
The opposite is extremely easy -- an established XF site and adding a new WP is a snap.