Since molds and fungus like cool, dark, and damp places to grow, and since used and sweaty gloves are therefore ideal places, I wouldn't try using just plain ol soap and water in a towel. Instead, try alcohol wipes and just wiping it all down on the inside.
I never tried this before but it makes sense. Go ahead and try it out, tell us how it goes.
