This is actually a good question and one that come up fairly often. In my experience, after being in and out of commercial gyms for nearly two decades (and working at many over that time as a trainer myself), my general experience is that most personal trainers are glorified rep counters. They can count to 8 (in-between looking around the gym or talking on their cell phone), make sure the client shows up, collect the check once per month and act primarily as salesmen more than doing anything related to training per se.
In fact, one gym I worked at in Austin, Texas had regular meetings for the trainers where their head 'expert' would come in. Was this to keep us up to date with the cutting edge of fitness science? Nope, it was to teach us how to close the deal and get people to sign up for more sessions.
Since most of what a personal trainer is going to do is handle what goes on in the weight room (you don't really need to spot a treadmill workout although many trainers will keep their clients company during cardio), I'm going to focus on that aspect of training in this article.
In my experience and in my opinion, most personal trainer certifications in the US are basically a joke. Generally requiring no more than a three-hour test (the NSCA CSCS exam is six hours) with no testing of hands-on competency in the weight room, it simply isn't that tough to get certified. While there are differences between them to be sure, the letters after someone's name would be one of the last things I'd look at in terms of determining if a trainer were any good.
At the end of the day, all of the letters in the world after someone's name don't mean jack squat. You can carry every national certification in existence and still not known a damn thing about how to train people in the weight room and you can be uncertified and be a true expert on weight training technique (e.g. my mentor).
Honestly, having been out of that aspect of the field for quite some time, I don't have much more to say about certification, I'm simply not up to date on what the individuals are doing (although a quick glace at the websites tells me that not much has changed).
Almost any trainer you'd approach will probably carry one or more of the major certifications but that guarantees, at most, very basic competency and the ability to have passed a written test at some point.
So what else can you use? A lot of people pick their trainer based on the trainer's physique. The logic seems to go that if the trainer is in shape, that they can get the client into shape. Sometimes true, all too often not.
In my experience, big male bodybuilders usually know how to train one person one way and that's themselves for bodybuilding. They'll put a beginner/general fitness client straight into hardcore advanced bodybuilding training and kill the person (I'll come back to this below).
True story: I had a client one time, older female who had had a double mastectomy. Had gotten a personal trainer at Gold's gym prior to working with me who put her through a 20 set chest workout on her first day; she couldn't lift her arms for like a week. Cuz, you know, you gotta bomb and blitz those pecs. This happens far more often than it should.
Yes, certainly, a trainer should work out and be fit. But looking solely at the trainer's physique is usually a mistake. It certainly shouldn't be the only thing that goes into a decision.
So if certifications don't tell you what you need to know and arm size doesn't, what can you use? One of the single biggest indicators that I personally pay attention to is what kind of form and technique a trainer has their client using. Is it good, horrible, a little bit sloppy or what?
In 15 years in commercial gyms, I can honestly count the number of trainers I've seen teaching proper form on anything on maybe one hand. Maybe not even that. Of course, when I see most of these guys train themselves, I see why: they don't know what good form is in the first place. Much less how to teach it to someone else.
Even if they do use semi-proper form themselves, it's clear that they either can't teach it or simply don't care to bother. The stuff I've seen clients put through by personal trainers just makes me cringe.
But that's probably the primary way that I'd generally determine if a trainer had their head up their ass or not, look at the kind of form they're teaching on key exercises. Yes, other stuff (program design, motivation) is clearly hugely important but unless someone can actually teach the exercises right, none of the rest will make much difference.
But that doesn't really help out beginners, who may or may not know proper form themselves, in terms of what to look for. Of course, there's endless information on the web about exercise technique. Most of it is shit mind you (Youtube is a never ending source of amusement in this regards, especially for powercleans), but there is some good stuff to be had. Some resources (e.g. exrx.net) are about half good and half bad. There's some rotten form on that site on some movements.
One thing I've suggested to folks before is to get a hold of the book "The Insider's Tell-All Guide to Weight Training Technique" by Stuart McRobert. It's an excellent book and examines technique for the majority of productive movements (squat, deadlift, bench press, etc).
Another excellent book in this regards is Mark Rippetoes "Starting Strength" which addresses proper technique for only 5 exercises (bench, squat, deadlift, powerclean, shoulder press) but does so in excruciating detail. I'm told the second edition describes a lot of other exercises but I haven't gotten round to buying it yet.