Plenty of people lose weight without counting calories, but ultimately it's the calories that make the difference
^ Exactly.
I think I've counted my calories fewer than a dozen X's, in 20 months I've been on my back-to-good-health journey, and that was just to get a rough idea, of where I was at.
Then, when it comes to body building, the big thing is, counting your macro balance (carbs, fats, and protein %'s) so I've done that a few X's too.
Of course in reality, it doesn't really make a difference if you are eating 1000 cals, or 10,000 cals > as long as your body is doing what it should be (for many here, losing weight) or, with body building, gaining lean muscle.
If your not losing weight (or, your actually gaining) you need to take in fewer cals, or burn more. Those are your only two options.
The one thing I do religiously, every day though, is weigh myself and record my daily weight. I don't really trip on daily fluctuations, in fact, I expect them, but once a week (every Friday) I add up all the days for that week, and divide by 7. This way of averaging, smooths out all the daily fluctuations, and gives me a much better idea of where I'm actually at... and where I'm headed.
Right now I'm at the end of a cut cycle (just about 2 more lbs to go

) but again, whether I'm eating 2000, or 9000 cals a day, doesn't really matter. If I'm still slowly losing (on a cut cycle) it's all good. If not, I need to tighten up on the cals, when planning my daily intake.
When I start bulking again, I'll bump up my cals as much as needed, and while watching the scale like a hawk, start gaining 1 to 1 1/2 lbs a week, while continuing to work out like a mad-man the whole time, of course
Peace,
Fish
PS, From the outside, someone might look at this whole process and think, "Wow ! What a PITA all that sounds like" ! But actually, once you figure out how to make your body do what you want it to do, it's very liberating, and can really become a lot of fun
