However, having said that, I think it's pretty pathetic - a big, lean "fit" dude who couldn't run a mile or swim a lake to save his life. I guess what I'm saying is, it's up to you whether you want to look fit, or you want to be fit.

If you only want to look fit, you can safely skip the "cardio".
+1 & agreed!!!
Last Thursday I went mountain bike riding with my trainer. Roz is 41 and probably one of the most fit & healthy-looking woman you've ever met, she looks like an IronWoman. She's cut, firm, muscular and can execute just about any exercise movement flawlessly.
We started-out and she
insisted that I ride my routine and she'll just keep up. After fiddling with a left shoe that didn't want to clip-in I finally gave-in to her demand that I just "do my thing"...so I said "okay, it goes a little something like this......" and I proceeded to get into my rhythm.
No, I did not try to blow her away....I just brought my heart-rate up to a cruising 150 and spun for about 8 minutes figuring she was on my heels.
When I stopped to look back, she was nowhere in sight...so I pulled-off to the side of the road and waited in the shade. Minutes later I began to wonder if something had gone bad (flat-tire, fallen, etc)....so I started cruising down the road...and down, and down and down! I finally saw her on the side of the road looking spent & exhausted. I asked what happened and she said her heart-rate monitor peaked at 193 and she just had to stop!
193?!!!!! No way.....I just paid for another 20 sessions and there's no way she's gonna pull some mortality/death thing to weasel-out on me!
We finally got her going again and I kept her in the lower gears, but even at a super-slow rate she was running HR 155-165....and what troubled me the most was that when we stopped to rest, her heart-rate wouldn't go down much below 145. Not good. My heart-rate during this time was about 116 AND my Garmin Edge 705 continually went into Auto-Pause mode because it thought we weren't moving! In all my riding, I've never ridden with such a weak person! I was shocked. Even some jogger going by commented to Roz "well at least you look really super-fit & healthy!".....
I mean, this woman looks like a stunning triathlete capable of kicking everyone's ass....and all she could repeatedly say is "looks can be deceiving" and "I do ZERO cardio!"...and ya know what, this is something I hear from soooooo many trainers! Many say they don't want to risk burning muscle so they avoid cardio...but it's an excuse and when pressed they generally admit it and then continue to admit they just don't like cardio. (I'm speaking of trainers I've met....).
I agree with G8 that it's sad as heck when you have these uber-fit looking people who can't run, bike or do any cardio much to speak of. I'm sure there are those who will argue, but you just don't get the circulatory, respiratory and other benefits (that come from cardio) as you do from weight-lifting. I've even had doctors say "if you enjoy weight-lifiting, fine....but the most beneficial benefits are derived from cardiovascular exercise"......
Now, before this spawns a huge battle-royal of arguement.....let me say this: I do both AND I know the best thing is to incorporate both weight-lifting AND cardio into your routine, but at no point is one a substitute for the other.
To the OP; you CAN lose weight w/o cardio, but don't sell-short the benefits of cardio...swimming in particular can really help develop upper-body and get you tone/cut very nicely too!