During the 1960s—while the nation was in the midst of the Civil Rights Movement—many African-Americans were part of a class of firsts: Ruby Bridges integrated an all-White school in New Orleans; Jackie Robinson was inducted into the Baseball Hall of Fame; Morrie Turner (creator of Wee Pals) became a nationally syndicated cartoonist; Thurgood Marshall was appointed to the Supreme Court of the United States; and Shirley Chisholm was elected to the U.S. Congress.

Taking his place alongside these and other barrier breakers was Abraham Bolden.

Bolden was tapped in 1961 to be the first African-American to protect a president of the United States.

But, the stint came with a price: his freedom.

Bolden graduated from Lincoln University in Jefferson City, Mo., with a degree in music before joining the Illinois State Police and, eventually, the U.S. Secret Service—a division of the Treasury Department—for which he sometimes went undercover to bust counterfeiting rings.

In April 1961, President John F. Kennedy visited Chicago and met Bolden, to whom he extended a personal invitation to become the “Jackie Robinson of the Secret Service,” Bolden says.

He jumped at the chance because he saw it as a stepping stone to his ultimate dream of becoming a diplomat, an ambassador to Ghana or another African nation. But his time at Kennedy’s side would be short-lived; in fact, he guarded the president for less than two months.

In his 2008 memoir, The Echo From Dealey Plaza, whose title is a reference to the site of JFK’s November 1963 assassination, Bolden writes about both the racism he encountered from his fellow agents, including his supervisors, and their recklessness while protecting the commander-in-chief.

On a trip to the Kennedy compound in Hyannis Port, Mass., Bolden says, the head of the detail purchased three cases each of beer and whiskey that agents drank before and after their shifts. He was subjected to racist insults, and agents sometimes whispered references to the president as a “n****r lover,” he says. Bolden then decided against staying on and returned to Chicago. There, he continued to monitor counterfeiters and spoke freely about JFK’s lax security.

After Kennedy’s murder, Bolden tried to testify before the Warren Commission, which had been established by President Lyndon B. Johnson to investigate the assassination. Looking back, Bolden says he wishes he’d gone straight to the press instead. “There I was thinking I was doing a patriotic thing for the government, when actually I was going against the government,” he says.

Bolden felt betrayed and set up.

In May 1964, the Illinois native found himself facing prison time for allegedly attempting to sell an investigative file to the head of a counterfeiting operation for $50,000.

Days after his arrest, Bolden called a press conference at his home in Chicago to publicly accuse the government of trying to frame him. He also wanted the charges overturned.

No luck.

In July, his first trial ended in a hung jury because the only African-American juror refused to find him guilty. A second trial the following month ended with a conviction. At his own trial in January 1965, the man from whom he was accused of soliciting bribes admitted that federal prosecutors had asked him to lie about case. Bolden quickly appealed and requested a new trial. He was denied and began serving a six-year sentence in 1966.

Bolden, now 81, talks about a memorable night during his incarceration.

On a steamy night in August 1967, he was jarred awake by a snapping sound, Bolden says. His eyes struggled to find the source of the familiar noise, but he realized he could not move his arms, legs or head. In a corner of the ceiling, a small fleck of light grew larger and brighter into a kind of spotlight. In the middle of the light, Bolden saw the unmistakable shape of a hangman’s rope and a man’s head swinging back and forth.

“I was so afraid I was on the verge of insanity,” Bolden recalls. In fact, he had been put in the psychiatric ward of a federal prison in Springfield, Mo. He believed then, and has maintained since, that he was not a nut case.

He was paroled in 1969.

While on probation, he sought a pardon from the Justice Department’s Office of the Pardon Attorney. A few months went by with no response, which he thought was promising; he believed his application would have been rejected outright if his case lacked merit. More than four decades later, Bolden is still trying to clear his name.

He has twice appealed to the U.S. Supreme Court and asked Presidents Richard Nixon and Bill Clinton to grant him clemency. In March 2016, Bolden applied for a pardon from President Barack Obama. Typically, after a pardon application is filed, the Department of Justice reviews it. If it believes the claim has merit, the FBI is asked to examine the case. Former U.S. Pardon Attorney Deborah Leff estimated a backlog of about 10,000 pardon applications when she left the department in January 2015.

To date, Bolden has still not received a pardon.

With the exception of a fistful of news clippings, his memoir and a smattering of mentions on conspiracy-theory message boards, little has been written about Bolden and his place in history.

George C. Howard, Bolden’s attorney in the 1960s, believes few people know about Bolden because of the turbulence of the era.

“We had any number of people who were fighting for equal rights. We had Fred Hampton [in Chicago], Medgar Evers down in Mississippi, and Dr. King was moving around the country,” Howard explains. “He [Bolden] was not a public official. He was not marching in the streets.”

Bolden believes events since the turn of the century vindicate his life and bolster his claims of injustice endured. In 2000, several Black Secret Service agents sued the government, alleging a culture of discrimination that prohibited African-American agents from earning promotions or choice assignments.

A 2015 investigation of a U.S. congressional committee echoed many of the alarms Bolden sounded about the Secret Service 50 years earlier. The report looked at several security breaches, including an Idaho man who fired several shots from a semiautomatic rifle at the White House in November 2011 and a prostitution scandal in Cartagena, Colombia, five months later. In September 2014 an armed contractor got on an elevator in Atlanta with Obama, a fence jumper made it to the main floor of the White House before being stopped; in March 2015, two allegedly intoxicated Secret Service officials, including a member of the president’s security detail, interfered with a bomb threat scene near the White House.

The fact that allegations of racial discrimination and slipshod presidential security have persisted long after the government’s charges against him, Bolden believes, is cause enough for the FBI to reopen his case.

In the meantime, his health has been in decline. He’s had bouts with cancer and several heart attacks. To keep his mind sharp, he spends his days doing trigonometry problems, posting articles on social media sites, attending worship services and participating in events hosted by his fraternity, Alpha Phi Alpha.

Bolden, who still lives in the house where he and his late wife, Barbara, raised three children, says he’s not looking for a payout for his wrongful conviction.

“I’m interested in due process of the law, pushing this thing so that the government, or whoever is behind these things, will be very careful because they know that someone like me might come along and never give up.”