Just out of interest where are you based?
I'm strongly considering quitting my job in order to pursue an MSc in Statistics and depending how that goes I might attempt a PhD. Somewhere like say ETH Zurich would, at first glance, seem to be a fantastic place to study in terms of quality of life, funding etc... Is your experience common with other people you know in other depts/locations in Switzerland?
I am at the ETHZ's sister school in the french part of CH, ETH Lausanne (EPFL).
The quality of life (outside of university) otherwise I wouldn't be here. The funding is obviously fantastic relative to other countries.
My story is certainly not uncommon. At the moment I have an especially hard time due to a project deadline but even so - no one in the lab will work less that 60 hours a week, working weekends is normal, spending vacation time working is normal.
I do know other people that had much easier times, a friend goes climbing with his boss during the week etc. So if you are lucky it can be great, if you are unlucky it can really suck. Some other friends went to the university of Lausanne, there things are much more relaxed but you get much less pay, and obviously doesn't have the reputation.
Some things to note, in switzerland you have to doctoral courses. These are much the same as Master courses, with homework, projects and exams. You have to do a lot of teaching, 1/3rd of your pay is from teaching, so you should expect to spend 1/4 to 1/3rd of your time doing teaching related things.
Typically you have 4 years to do your PhD, but that is incredibly tight with the courses and teaching workload, so not everyone manages it. At EPFL they are moving to a US style 5-6 year PhD, which I think makes sense. At The moment we are trying to do a 5-6 year PhD in 4 years. That is something to bear in mind.
If you get a subject you love, a great prof, a free working environment and are lucky (make a breakthrough early and you are laughing) then it would be fantastic. To survive you will need incredible motivation though, so you have to love the subject and the academic lifestyle.