Official climbing season is only in July and August and that's also the only time when all the mountain huts and security people are up there. Without that it's very dangerous to go and also you might not be able to go all the way up because of too much snow, but then there are no crowds for sure ;P
I heard that beginning of July is still the best time, because Japanese schools are still not on vacation (I couldn't go because I had to work), but as it's still rainy season the weather can be very shitty, too! :(