Just back out, if you think you can't take him. Is it really worth getting your ass kicked? Just tell him, his girl kissed you and its not your problem. The girl came to you.
Now if he's your size, then you could go for it. But if he's way bigger than you, then no.