I'd say that they both matter... Try to keep both low. Only 30% of the calories that you eat should be fat, so just look at that number and try to follow it pretty closely.
My personal preference is to look more at calories, though... It's just my obsession.