I suspect it's to do with fluency, at least partially. Have you looked into how it goes for very fluent typists? Problem is, I suspect you might get enough fluency that typing isn't distracting as compared to handwriting only at significantly faster speeds than most people consider the beginning of "fast".
It's not about distraction, actually. Since the mode of typing vs. writing is different; I can feel that my brain is working differently.
While I don't type lightning fast (~130WPM and beyond), I already type without thinking about it. I just think and it appears on screen, actually. On the other hand, when I'm writing, there's another sub-process which is evaluating whether what I'm writing makes sense or works in real world. It's not possible to do this while typing since the freedom provided by the pen, and the thinking process is completely different. Also, I can build a model of what I'm writing about in my mind better. In short, typing lends to shallower thinking while writing allows more depth and exploration on the subject.
This is also evident when I'm writing code. I design it on paper, and type that design to the IDE I use.
The research articles I started to collect also points to something similar. When using pen and paper, neurons fire differently and in larger networks, pointing to a different mode of thinking. Considering I started use pen/paper and keyboards almost at the same time, and able to verify that using a pen really makes my brain work differently, I find "you're typing it wrong" a flawed argument for the most part.
