I like them, but I don't think I use them like most people, and I don't use them non-stop:
1. Doing pushups on them really works lots of stabilizing muscles and not just lower chest and triceps. Try it sometime... you'll be shaking real bad and might fall on your face if you're not careful.

If I'm on a ball, I can only do about half of the normal amount of pushups I can do.
2. Doing crunches on them are simply more comfortable in my opinion. But also, if you want to do low rep crunches where you hold the contracted position for a second, I like it better because you have more range of motion, because you can stretch back further and go beyond horizontal, and once again, it's just more comfortable. I feel more of a burn and get worn out easier on a ball than just on the floor and my back is lots more comfy.
3. I use it for.... crap, can't think of the name... basically where you get in a smith machine or something similar, put your feet out, then basically do horizontal pullups like a row. Instead of putting my feet on the floor, I put my feet up on a ball, which makes you stabilize your legs with your core and also gives you a little different angle on your pullups... it's actually harder like this.
4. I use it for hamstring curls. Basically you can lie on your back with a heel on the ball and flex your hamstring and butt up and it helps with muscle inbalances. This works more muscles and is alot more comfy than the typical hamstring curl while lying face down on a bench.
I think the key is the stabilizing muscles. Just depends on what you want to do. If you want to just look good and lift the most amount of weight possible, you're probably not interested. I think from a more athletic standpoint where you need stability in a particular sport, they're great.
Personally I think alot of guys shun them because they think they look girlie/gay/amateurish but alot of great athletes use them. I bet you'll never see a beefcake in a gym use one though. Just depends on your goals.