I sympathize with your predicament. As a straight cis male--and one conscious of all the privilege that goes with that identity--I am painfully aware of the oppression of nearly everyone who doesn't look like me. And the people of in-between genders--trans, bi, intersex--must be among the loneliest of all. So my heart goes out to you.
That being said, there seems to be an inconsistency in your expectations of the rest of us. Other-gendered people have just as much right as everybody else to be who they are and to feel drawn to whatever kind of person they feel drawn to. But don't the rest of us also deserve to have our feelings respected?
If people were rejecting your friendship based on your genitalia, I would be offended on your behalf. But a penis is one thing most people do have feelings about, and how a person feels about a penis is nearly always going to have an impact on whether they want to be intimate with the body that penis is attached to. It's not transphobia; it's nature.