Every body fat test I try says I'm 11-12% body fat but my Tanita scale says I'm 21% I'm not sure why, but it is way off for me.
And if you were to lose fat and build muscle staying the same weight, you would still be proportioned differently so it is unlikely your clothes would still fit properly. (smaller waist and hips, but larger shoulders etc.)