Yeah, the more weight/less reps scheme is supposed to result in increased strength, moreso than increased muscle size. However, I've put on significant size using 5x5.
Like Ride On said, if you're not eating at a surplus, you're not going to bulk up. That is a weighttraining myth that seems to keep women out of the weightroom - that if they lift weights, they'll get all big and muscular. Building muscle takes a lot of heavy lifting and a diet centered on a calorie surplus with a ton of protein (in addition to all the other nutrients a person needs).
So, to answer your question, go with a set/rep scheme that you feel comfortable with. The standard is 3 sets of 8 to 10 reps. You could also do 5 sets of 5 reps (where you'd have to increase the weight to keep it challenging) or 2 sets of 15 reps (with lighter weights of course. This scheme is considered endurance building, but I've never really tried this myself).
Also, some people really hate the idea of using a Smith machine for squats. Something about it not being the natural movement. I'm not expert so I'll refrain from commenting on that, but I do generally believe that, as long as you take the time to learn proper form in the interest in injury prevention, freeweight exercises are superior to any machine that assists in the lift as anything that makes an exercise easier kinda defeats the purpose of the exercise, at least to a small extent.