Not being familiar with IIS, I don't know - but I'd be looking for 3 things if I had this problem myself.
a) Does IIS have some kind of flood protection/page requests per IP limter thingy and if so how do you alter the settings
b) Is your firewall being anal
c) How are other network applications involving serving doing (Kazaa etc)?
[edit]
d) Could it be caching frequently accessed stuff? If it's trying to cache MP3s or DivXs then there's going to be some damn big cache files about which would take a while to load.