Does it sound like he used me for sex?

Well he didn't really use you, because you both got something out of the relationship you had, and it wasn't all about sex. But even though you're both in your later 20's, it still sounds like he is teenager. I think your right, you should let him go. He's not right for you, and will only cause you more pain in the future.