Why don't you just wait and see if anything else is said in the future. As of right now it has only been one incident. Could have just been casual banter and he probably didn't give it a second thought. If you make and issue out of it, he might start looking at you a little closer. If other comments are made in the future then you know for sure there is an issue.