Stock isolator goes bad fast and creates the clacking sound coming from the supercharger. Invest in a aftermarket solid isolator. Best mod for the money, I hated that clacking coming from my $70k Cadillac.
If you have a smaller pulley, who ever installed it for you shouild have changed it. The isolator bushing rides on the pulley shaft inside the snout. The factory one is worthless and is coposed of a large spring that rides on the metal shaft. Almost immediately, that springs starts to eat ythe shaft. My inital guess would be that this is your problem. Ask who ever did your smaller pulley (which involves taking off the snout) if they put a solid isolator bushing on. Its almost standard practice for anyone who knows these cars.