Not being able to stay on the swiss ball is a combination of the both; but more so just bad balance.
The more you do it the better you will get. I used to lay on the ball for abs and slide left and right. Now i can do full situps on the ball, with only one foot on the ground and the other one straight in front of me; it just took a lot of work. There are a lot of ways to work on balance and core.
For balance:
Do squats, without weight, while standing on a bosu ball. First with the flat side on the ground, then as you get better put the rubber half on the ground and stand on the flat side. Do ab work on the swiss ball when you can; and even do some of your weight training on the swiss ball (dumbell bench etc). When doing standing exercises, like bicep curls, stand on a stability pad or bosu ball. Between sets, when you're resting, stand on one leg on a stability pad for 30 seconds then switch.
For Core:
Read this article -