Funnily enough I recently found out about this too. I can visualize stuff, but my friends sister can't.
Ok how about this one... could you draw a road map of your home town / village without referring to images or another map? If so, surely you can in some way 'see' your hometown...?
Without putting words in the OPs mouth from my experience talking with others that have this problem - yes, they likely could.
Typically people that can't visualize have coping mechanisms that mean it can go unnoticed (like the OP not evening knowing he was missing out!) drawing something from memory is still possible without visualizing but based on facts and memory.
I.e. They're able to drive home because they know that they arrive at a village sign with a church on the left, they drive past two roads (grey street and Stevens avenue) then past the school. They take the next right which heads off under a bridge and arrive at their house half way along the road.
Making a drawing of that might even result in a more accurate one than you can imagine because it's based on facts they've had to develop to cope.
That's how it works for some anyway!