It's just statistics. You are overestimating people's ability to drive/park, squeezing into a garage will cause more damage than vandalism on average unless you live in a really dodgy area. Then compared to street parking it has the added issue of being directly associated with a property in most cases, they take your keys from your house and then straight to the garage knowing the key will be for that car. Same risk for driveway parking of course, but less accidents.
The key issue I find with people being incredulous about insurance premiums, is they are often incredibly self-centred in their thought processes. They think "I am amazing at parking, so I will never ding my car". "My garage is massive and my car fits inside with acres to spare". And this may be very true. But the insurer doesn't know this, they just have a set of statistics, telling them how much money they stand to lose from garage parking compared to driveway parking on average.