Then you'd be wrong. Much of it is how a bike feels. In your head, the light weight and skinny tyres and position make a road bike feel much faster, but when you look at the physics it's just not the case.
Dont get me wrong though, a road bike is still much better for any sort of distance, because even small differences in efficiency and comfort actually make a big difference to how you feel on a bike, and therefore how far you can ride.
Here's an example, roughly simulating the OP's speed and power:
http://www.kreuzotter.de/english/espeed.htm
MTB 26lbs with 1.75" offroad tyres on a 1 percent slope and 5mph headwind @ 150W = 11.1mph
Road bike (hands on the tops) with skinny tyres 20.9lbs in same conditions @150W = 12mph
Same road bike on the drops = 13.4mph
Obviously there are more variables, and the more trained you become then the bigger the difference as the aerodynamics start to play a bigger part, but at the OP's current speed and power, 1 - 2mph seems about right.