I was 22-years-old and it was only my third time visiting New York City. The plan was to meet a friend to catch an artsy independent flick. When I emerged from the subway, I had no idea where I was. Rather, I knew where I was based on the directions I used, but I had no idea the dynamics of the neighborhood I was in. I purchased my ticket for the movie, stood outside people-watching while waiting for my friend to arrive.

A man walked past me, moving quickly in the throng of people, yet still managed to utter a barely audible “hello.” I acknowledged his presence with a downward head-nod that I use to greet every Black man and went on about my business of waiting. But he circled back, making his way through the crowd and approached me. He asked how I was doing, introduced himself and asked me what I was doing standing there. I figured him for a “crazy New Yorker” type that I’d seen depicted in any number of movies, who might babble on indecipherably for a few minutes before cursing and storming off to pee on the stairs of a subway station.

I told him I was there to see a movie and he asked me which one. I turned and pointed to the poster for Gomorrah, and upon seeing Martin Scorcese’s name as a producer he alerted me to the fact that Scorcese made a lot of gangster movies. I wanted to hit him with a “no sh*t, Sherlock,” but I just nodded in agreement, hoping my disinterest in the conversation would show on my face and he would walk off. No such luck.

He asked who I was going to the movies with, I told him a friend. Then I finally figured out what was going on. He shyly said, “Maybe you and me could go to the movies sometime.” I smiled nervously while shaking my head to decline his offer as I realized I was being hit on. He asked me why not, was he not my type? “No, you’re not,” I replied, and he quickly shot back with “What type of men do you like?” I told him: “Women.”

I’m 5’8 and 140 pounds on my best day. Back then, I was smaller. No one would ever accuse me of being a physically imposing figure. He was at least 6’1” and had to outweigh me by around 30-40 pounds. Yet, there this man stood, with fear-frosted eyes, in a defensive stance as if he were preparing himself for an attack. As he apologized unnecessarily and asked if I was offended, I did all I could to assure him I wasn’t and there was no threat. We shook hands and he walked away.

I’d tell that story more jovially if I never saw his eyes. This isn’t me bragging about this one time I was in New York and some guy stopped all he was doing to hit on me. What’s important here is the life-or-death fear that settled into his face, a fear I had never seen up to that point and hope that no one ever has to experience again.

Fact is, as a heterosexual man, nearly anything I do that is seen as an extension of my sexual desires will be justified by someone, somewhere. No matter how reprehensible, immoral, violent, illegal, or disgusting, any action that follows from my attraction to women will be seen as legitimate and find defenders. Yet, here was this man afraid I would punch him on a crowded street because he asked me to the movies.

When he left, I noticed all the men walking up and down the street holding hands. They kissed each other, they groped their partners other public. Gay men were literally everywhere, open, loving, and free. So, even in a space that was presumably safe to express same sex attraction, the man who approached me was still afraid his identity could cause him harm.

And we’re OK with that.

When I say “we,” I mean those who identify as heterosexual, and I say we’re OK with that because we do so little to alleviate those fears. Even those of us who claim to be progressives or allies fail at times to seriously account for the experiences of those members of the LGBT community we claim to care for. We spend time speculating people’s sexual orientation, only to co-opt, or dismiss as unimportant “coming out” stories. Some of us even make vicious jokes, suggesting it doesn’t matter because we don’t hate gays. And then we have the unmitigated nerve to wonder why more people are not open about their sexual identity.

This isn’t progress. The applauding of the bravery involved in coming out isn’t what we should be aspiring to, rather, we should be actively rooting out the homophobia in our communities and our own thinking that makes coming out such a brave act. Our goal should not be making coming out safe, but to make it unnecessary.

We can hide behind “who you love is your business,” but that doesn’t cover it. The people I love are straight, they are gay, they are bisexual, they are transgender, they are refusing to label themselves. This world should be safe for all of them. Their eyes should never know fear.

Mychal Denzel Smith is a writer, social commentator and mental health advocate. Visit his official website or follow him on Twitter.