When I was taking immunology as an undergrad, my professor had a saying that "what doesn't kill you, makes you stronger". That is so true. Being exposed to a multitude of nonfatal germs, which the vast majority are, increases the number of antibodies in your system so that when you are exposed to that one particular nasty fellow, your body will have the defense mechanism against it.
So, to answer your question, matt182 is 100% correct in saying that being exposed to fewer germs is actually bad for you.