Four Muslim men who have accused FBI agents of putting them on a no-fly list because they refused to become informants want to pursue damages against the agents even though the travel ban has been lifted, the men’s lawyers told a federal judge Friday.

Earlier this week the men received letters notifying them their names had been removed from the list of tens of thousands of people prohibited from flying to, from or within the United States. Three of the four — Jameel Albighah, Naveed Shinwari and Awais Sajjad — were in federal court in Manhattan to hear the arguments over a government motion to have their lawsuit against 25 FBI agents dismissed.

Plaintiff lawyer Robert Shwartz told U.S. District Judge Ronnie Abrams that “various FBI agents punished (the men) and put them on because they refused to become informants at their mosques.” As a result, they were unable to travel to see loved ones, lost job opportunities and suffered the “stigma of being treated as threats to aviation security,” he said.

He added: “Money relief is really the only relief.”

In a statement, Albighah, a U.S. citizen from Yemen with a wife and three daughters who still live there, said being on the list proved agonizing.

“They messed up my life,” he said. “My youngest daughter doesn’t even know me.”

Assistant U.S. Attorney Ellen Blain argued the case shouldn’t go forward because neither the law nor the evidence supported finding the agents personally liable for violating the plaintiffs’ constitutional rights. She also warned further litigation could unnecessarily delve into national security issues.

Abrams is expected to rule later in a written decision.

In April, the Obama administration announced a new policy allows for an American traveler who has been denied boarding a commercial airliner to petition the U.S. Transportation Security Administration once to find out whether he or she is on the no-fly list, and for an unclassified explanation of why he or she is on the list.

The new policy partially lifted a veil of secrecy enshrouding a strategy that has been a centerpiece of the government’s counterterrorism program since the September 2001 terror attacks.

But the American Civil Liberties Union, which has been challenging the constitutionality of the no-fly list in a separate federal lawsuit, have said the changes don’t go far enough in giving travelers the legal due process they are entitled to, including seeing the evidence held against them and getting an opportunity to challenge it.

Photo Credit: TSA agents man the checkpoint for pre-cleared passengers at Hartsfield-Jackson Atlanta International Airport. Kent D. Johnson/Atlanta Journal-Constitution / Associated Press