Its a polite way of saying no (or they think it is), the ones who say it are inexperienced in bed and believe height has relation to a sausage, and the main one, small people are generally annoying - angry midget syndrome, being over the top to overcompansate size etc.
I hate stereotyping but...