In an Aug. 21, 2013 file photo, outgoing FBI director Robert Mueller speaks during an interview at FBI headquarters, in Washington. High-profile supporters of President Donald Trump are turning on Mueller, the man charged with investigating Russian interference in the U.S. election and possible collusion with Trump’s campaign.

WASHINGTON (AP) — High-profile supporters of President Donald Trump are turning on special counsel Robert Mueller, the man charged with investigating Russian interference in the U.S. election and possible collusion with Trump’s campaign.

As Mueller builds his legal team, Trump’s allies have begun raising questions about the former FBI director’s impartiality, suggesting he cannot be trusted to lead the probe. The comments come amid increasing frustration at the White House and among Trump supporters that the investigation will overshadow the president’s agenda for months to come — a prospect that has Democrats salivating.

Trump friend Chris Ruddy, the CEO of Newsmax, went so far as to suggest the president was already thinking about “terminating” Mueller.

“I think he’s considering perhaps terminating the special counsel,” Ruddy said in an interview with Judy Woodruff of “PBS NewsHour.” ”I think he’s weighing that option.”

Former House Speaker Newt Gingrich, an informal Trump adviser, tweeted Monday, “Republicans are delusional if they think the special counsel is going to be fair. Look who he is hiring.”

Just weeks ago, Gingrich had heaped praise on Mueller, hailing him as a “superb choice” for special counsel whose reputation was “impeccable for honesty and integrity.”

But after the testimony of former FBI Director James Comey last week, Gingrich said he’d changed his mind.

“Time to rethink,” he tweeted Monday, citing Mueller’s hiring decisions and Comey’s admission that he’d instructed a friend to share with reporters notes he’d taken of his private conversations with Trump in order to force the appointment of special counsel.

Conservative commentator Ann Coulter offered a similar message, tweeting, “Now that we know TRUMP IS NOT UNDER INVESTIGATION, Sessions should take it back & fire Mueller.”

The talk about dismissing Mueller appeared to be coming from Trump allies — including some close to White House strategist Steve Bannon — who are increasingly frustrated with the prospect of a long and winding probe.

They say Trump did not collude with Russia and see the investigation as a politically motivated sham that handicaps Trump’s ability to execute his agenda, according to one person who advises the White House on how to handle the probe. The person demanded anonymity to discuss strategy on the sensitive matter.

Ruddy appeared to be basing his remarks, at least in part, on comments from Jay Sekulow, a member of Trump’s legal team, who told ABC in an interview Sunday that he was “not going to speculate” on whether Trump might at some point order deputy attorney general Rod Rosenstein to fire Mueller.

“Look, the president of the United States, as we all know, is a unitary executive. But the president is going to seek the advice of his counsel and inside the government as well as outside. And I’m not going to speculate on what he will or will not do,” Sekulow said. Still, he added, “I can’t imagine that that issue is going to arise.”

It wasn’t clear whether Ruddy, who speaks with the president often, was basing his remarks on a specific conversation with the president or entirely on Sekulow’s comments. Ruddy did not immediately respond to questions seeking clarification.

Ruddy was at the White House Monday to meet with White House aides, but did not speak with the president, Press Secretary Sean Spicer said. “Mr. Ruddy never spoke to the president regarding this issue,” Spicer said. “With respect to this subject, only the president or his attorneys are authorized to comment.”

White House spokeswoman Sarah Huckabee Sanders said via email, “Chris speaks for himself.”

Peter Carr, a spokesman for Mueller, declined to comment on Ruddy’s remarks.

Under current Justice Department regulations, firing Mueller would have to be done by Attorney General Jeff Sessions’ deputy, Rosenstein, not the president— though those regulations could theoretically be set aside.

Sessions recused himself from all matters having to do with the Trump-Russia investigation because of his own conversations with Russian officials during the Trump transition.