Was it a joke?

It was delivered like it was a joke but the implicit message says otherwise.

There was one guy in college who used to make nasty comments but dress them up as jokes.

So then if you did take offence, which was only natural as he could be quite cutting, he would then make you feel bad for taking it seriously because of course he was only joking.

I think he got his kicks from being so manipulative.