Really Annoying Person I Need Advice?

Ok there's this guy who I'm friends with who is really annoying. If he IMs me he just babbles about things I don't care about and during school he also just bothers me with stupid things. I don't really want to be his friend anymore, but I don't want to be mean. I also don't want rumors spread about me if I do break up our friendship, because someone did that to him once and he IM'd me afterwards saying, "Oh, I hate so-and-so they're so mean and they were really rude they said I was annoying..." and on and on. What should I do?