The difference between the 4 and 4S is Siri and a slightly faster processor yes? That being the case, I don't understand why upgrading from 4 to 5 is much different than a 4S to 5.
The 4 was a single-core Cortex-8 class chip. The 4S was a dual core Cortex-9 class chip. So for a start there's a 2nd core so that you can be running two independent threads at once, which will help improve responsiveness, but then it's a more modern micro-architecture, so even though it's clocked around the same, it's probably 20-30% quicker at processing instructions.
The 4S to the 5 is moving to what many people consider a Cortex-15 class chip, which at a similar clock speed roughly doubles performance.
Add to that the new screen - yes, it's not much bigger, but when watching video the 16:9 aspect ratio means that you don't have to a) chop off the sides or b) have letterboxing when watching content which has been the biggest thing for me.