Lets hope we have seen the limits of U.S. Dept. of Housing and Urban Development Secretary Ben Carson’s historical ignorance since facing backlash by equating slaves with immigrants. Now that he has the keys to the agency, those needing affordable shelter need him to get this right.

Millions of struggling Americans are not considered poor or poor enough for help with shelter, with many full-time workers paying nearly half of their income to rent. Last year’s “Ghost Ship” artist loft fire in Oakland that killed 36 people exposed a nationwide underground culture of makeshift housing, and in its wake, many warehouses in urban areas, such as Detroit, Austin, Nashville and Dubuque, Iowa, have been served noncompliance orders sending residents scrambling.

HUD, which has long stood in the gap for American families seeking affordable housing, is a medium-sized agency with a $47 billion budget and a mission to create strong, sustainable, inclusive communities and quality affordable homes for all, in direct conflict with Carson’s ideology as expressed through his public statements. A child of a single-mother, a striver in her own right, its possible the new HUD chief could take the agency’s charge to heart. As Carson has no governing track record, let’s take a look at his statements about government and the nature of poverty to surmise what he could do.

Carson said: “Poverty is really more of a choice than anything else.” His mindset on the issue of poverty was made starkly clear when he announced the need for a flat tax rate for all Americans. A flat tax means an individual who makes $50,000 would be taxed at the same rate as one who makes $250,000, effectively paying more of a share of income than more well-heeled taxpayers. The upshot: Less money for housing costs.

What He Could Do: The fact Carson feels all income levels needs to be taxed at the same rate suggests he is less likely to ask for more money for HUD, less likely to increase vouchers for housing, even though a study found that offering permanent housing subsidies through Housing Choice Voucher programs are the most effective interventions for homeless families.

Carson said: Calls new HUD rule, requiring local communities to track and assess their own patterns of racial and income segregation social engineering and a failed social experiment and proposes to eliminate them.

What He Could Do: Carson apparently feels racial equity or desegregation is not what housing policy should focus on. But lets push back these so-called social-engineering claims. Such engineering is the ability to influence certain attitudes and social behaviors on a large scale either by governments, media or other private groups to make a certain group of people behave in a particular way. While certain policies, indeed, fit this definition, the Fair Housing Act is certainly not one of them.

What does fit are policies such as Jim Crow, where state and local laws enforced segregation and created a different set of rules for blacks and whites, legally denying blacks rights and freedoms. This socially constructed policy let Whites know it was OK to treat Blacks (and other ethnic groups) as inferior, and U.S. history is rife with policies socially whose sole purpose is to keep certain groups of people out of power. Or consider Richard Nixon failed War on Drugs, which single-handedly fueled an incarceration boom where the sentencing disparity for possession or intent to sell drugs was 100:1 for black men to white men. Nixon’s chief of staff, H.R. Haldeman, quoted him as saying, “You have to face the fact that the whole problem is really the Blacks. The key is to devise a system that recognizes this while not appearing to.”

While Carson clearly doesn’t believe in efforts to learn how and where segregation occurs, research shows the Fair Housing Act benefits low-income families. Deconcentrating poverty is based on promoting and designing mixed-income neighborhoods, which improves health and well-being of all families.

Carson said: He opposes same sex marriage and has compared same sex couples to child molesters or people who practice bestiality.

What He Could Do: Its unlawful to discriminate on the basis of race, gender, sexual orientation, familial status, religion, national origin, etc. But Carson could work to rescind rules protecting LGBTQ Americans from housing discrimination. He might very well eliminate all of or parts of the Equal Access Rule, which makes it illegal to discriminate against LGBTQ people in housing that receives funding from HUD or is in insured by Federal Housing Administration.

Carson said: Federal money has not increased the values of American families and individuals and insists the government has robbed citizens of their dignity by reckless spending.

What He Could Do: Defund or make less important essential programs and policies such as the permanent housing subsidies program (SUB), community-based rapid rehousing (CBRR), affirmatively flouting fair housing rules, and other essential programs that enable low-income families to have affordable housing. Furthermore, deregulation allows room for private investing within HUD programs. Carson’s boss has a track record in this area: President Trump’s father, Fred, was subpoenaed to testify on allegations he had ripped off the government by overestimating construction costs to get larger mortgages from the Federal Housing Administration. The elder Trump was accused of violating fair housing laws by denying Black families opportunities to rent apartments in his buildings; this occurred while the president worked for his father. All this leads one to believe by allowing deregulation, more business owners can take advantage of loopholes giving greater advantages to the rich versus those who need shelter.

While taxpayers should always been concerned about federal overspending, currently this country is short 7.2 million units of affordable housing. Carson needs to remember from whence he came and understand what it means to be a family on the brink in the need of a government that helps people help themselves.

Shetal Vohra-Gupta, Ph.D., is associate director of The Institute for Urban Policy Research & Analysis at the University of Texas at Austin and a Public Voices Fellow.