Tougher? No. Slower? Yes.
Managing expectations is a huge part of this process. If you have unrealistic expectations, chances of reaching your goals are slim to none.
The "secret" to building muscle is finding that sweet spot, calorically speaking, where you're eating enough to facilitate muscle growth but not so much that you're out-eating your abilility to build muscle, thus storing a bunch of fat.
Finding that sweet spot takes trial and error and that's why measuring progress is critical.
Rates of muscle growth will vary depending on how long you've been training, age, genetics, diet, training, etc but the rate is almost always much slower than most trainees realize or expect.