Men project. Racially different or "exotic" looking women are attractive to many men, but rather than admit they are intrigued, they tell themselves the attraction is coming from the other direction.

And, as several commenters have mentioned, there is the more general projection of men assuming any woman they find attractive is trying to be attractive specifically to them, because she must be interested.

Finally, men of low self-esteem, doubting they are attractive to women of their own race, figure women of a "lower status" race will want to be with them.

In every iteration, it stinks.