It’s hard to imagine where exactly U.S. Supreme Court Justice Antonin Scalia was going while hearing yesterday’s Fisher v. University of Texas arguments. Citing points from an anonymous amicus brief, Scalia suggested placing Black students in an environment like the University of Texas could hurt them because they might not be able to keep up. Perhaps, he suggested, Black students might do better if they attended schools that weren’t as “advanced” or where classes didn’t move as quickly.

We should find Scalia’s perceptions of Black students, their aptitudes, and intellectual capabilities, troubling. It is views like this that reinforce the persistent and alarming institutional biases that harm people of color from birth to adulthood. Scalia’s failure to properly contextualize his point within the larger conversation on education in America is yet another example of conservative myopia about race.

We can’t discuss what goes on with Black students at the collegiate or post-graduate levels without acknowledging everything that leads up to those points. By every measure America fails to provide our children access to quality education in a K-12 system that prepares them for college or a career. Often our children sit in classrooms with the least prepared teachers, incoherent curriculum, low expectations, and a lack of resources. Yet, in spite of this many of our students have seized the opportunity to overcome their struggle and graduate from some of the most demanding universities. We will do better by providing strong supports for their success rather than attempting to assign them, once again, to lower rungs of education.

If Scalia is sincerely interested in the real challenge to Black students attending college, he would need a primer about college costs. In fact, the number one reason why Black students do not graduate is that many are unable to pay the rising costs of college education in America. In fact, over a quarter of both Black men and women who dropped out of college cited financial difficulty as the reason they did not continue in school. It wasn’t an inability to learn, classes that move too fast, or a penchant for excess partying—those are all tired myths based in very lazy and offensive stereotype, and perhaps more importantly, markers that are hardly exclusive to Black students. After you deal with college readiness, the biggest challenge to success that those of our children who make it to college have is not in the classroom, but usually in the bursar’s office.

The rising costs of higher education have sorely outpaced any increases in average salaries, and that income gap has created very real problems for black students ability to stay in school.

If Scalia has any question as to what can become of a Black person who is given the opportunity to succeed, he needn’t look any further than to his bench buddy, Clarence Thomas. Justice Thomas was appointed to the bench at just 43 years of age, having had an otherwise unremarkable career as a jurist. It would be easy to argue that there were a host of other judicial appointees with far greater experience than Thomas and who were arguably more qualified. Still, after the loss of Thurgood Marshall on the court, then-President George H.W. Bush needed another African American on the bench, if only for the optics. In steps Thomas and the rest is (a somewhat sordid) history. How ironic that Justice Thomas himself, someone Scalia believes erudite and whose opinion(s) he often agrees with, is on the bench is essentially the result of affirmative action.

The real conversation about Black college students should acknowledge affirmative action and programs like it remain necessary because of a woefully uneven educational playing field in grades K-12 creates similarly unequal points of access to the best colleges and universities.  One needn’t look any further than what has happened at University of California, Berkeley and the state institutions there to understand that eliminating affirmative action would have dire consequences for blacks and other minority students pursuing higher education. According to the UC fact sheet, in 1995 approximately 4.3 percent of entering freshmen were Blacks. Since the elimination of affirmative action in California that same year, the numbers of first year students has gone up by 11,000 but the numbers of Blacks in the class has dipped to around 4 percent. This is an example of what higher education would continue to look like all around the country without affirmative action in place.

If the goal is to achieve a post-affirmative action America, then there will need to be a serious reckoning of the system policies and practices that reproduce inequity from kindergarten through college. Unfortunately, shortsighted comments like those made by Scalia aren’t even a down payment on that discussion.

Chris Stewart is Director of Outreach and External Affairs Education Post, a non-partisan communications organization dedicated to building support for student-focused improvements in public education from preschool to high school graduation.