Recently, Ebony magazine released the cover of its November “Family” issue—a portrait of the smiling Huxtable family, marred by shattered glass, the face of “The Cosby Show” patriarch Cliff Huxtable obliterated. Cracks crawl like tentacles over the entire family. Bill Cosby—the man who gave his name and his family stories to “The Cosby Show”—stands accused of sexual assault by more than 50 women. The Ebony cover is an illustration of how those accusations have fractured the comedian’s legacy and everything associated with it.

But the unveiling of the cover unleashed hell on black social media. To many vocal African-Americans, it is an affront—a venerable black publication colluding with racist oppressors and rabid feminists, publicly piling on a beleaguered (and possibly innocent) man. Bill Cosby, for some, has become a symbol of every black man brought down by a racist society with the help of mercenary and vengeful women. To them, the Ebony cover is the latest lynching of a black idol. Not just that, but Ebony’s social media feeds are also rife with folks fretting over trivialities like the financial well-being of other “Cosby Show” cast members and the tarnished legacy of the 80s sitcom. We are left with the impression that nearly everything under the sun is more important than potentially victimized women.

Protective thinking in response to a black man accused of sexual crimes is understandable, given America’s history of stereotyping black men as predators. But reflexive and unthinking circling of wagons is a damaging reaction that protects the black community from exactly nothing. In fact, while Bill Cosby’s alleged victims are mostly white, the idea that the respectable public face of black men must be protected at all costs is ultimately most damaging to black women.

In the late 80s, when I was a college student on a predominantly white campus in the predominantly white state of Iowa, I was sitting in a dormitory commons, the only black face in an otherwise white room, when one of my floormates asked for someone to accompany her to the basement to get her laundry. “I’m so afraid some big, black guy will rape me,” she added. I know that in America, black man and bogey man are too often the same thing. And like many African-Americans, I cannot forget the history of false accusations used as a tool to destroy black men. The image of Emmett Till, murdered for allegedly whistling at a white woman, his bloated and mangled body in an open coffin, is seared in black America’s collective memory. But this sordid history cannot be used as a veil to blind us from present day facts, nor an excuse to ignore the reality that black men do rape–as other men do.

The determination by some to force Cosby into the historic narrative of black man wrongly accused by whites erases his black women accusers. It supposes that all of his alleged victims are liars, but that the black women are something worse–race traitors. And the impact of this thinking extends beyond the tragic case of America’s once-favorite father.

African-American women are more likely to face sexual abuse than their white counterparts. According to Black Women’s Blueprint, 60 percent of black girls have experienced sexual abuse by age 18–most at the hands of black men. (Over 90% of sexual assaults occur between people of the same ethnic or racial background.) What space does the black community give those women to heal and receive justice if its primary concern is always burnishing the public face of always-victimized black manhood? Can’t talk about the deacon, because what about the flock? Can’t talk about the basketball star, because what about the season? Can’t talk about the community activist, because what about the community? Can’t talk about Bill Cosby, because what about Malcolm Jamal Warner’s residuals? Just shut up and let those black men be admired–whether they deserve it or not.

Silencing discussion about sexual violence in order to prove the decency of black men at the expense of women is indecent. And a culture that cannot even brook interrogation of accusations of sexual assault is broken in a way that ensures predators will go unpunished and victims will suffer. No legacy of a sitcom character–even one that humanized the black family for America–is worth that destruction.

Tamara Winfrey-Harris is the author of The Sisters Are Alright: Changing the Broken Narrative of Black Women in America. Learn more at TamaraWinfreyHarris.com.