The president of IRS employees’ union sent a letter dated Tuesday to the Republican governor calling the comments “scurrilous and absurd.”

LePage made the comment during his weekend radio address as he assailed the U.S. Supreme Court ruling that upheld the federal health care law, saying the decision gave Americans no choice but to buy health insurance or “pay the new Gestapo — the IRS.”

Maine Today Media reports that Colleen Kelley of the National Treasury Employees Union also wrote that comparing the agency to the murderous Nazi secret police could “lead to violent attacks” on federal workers.

LePage has said he never intended to insult anyone.