WASHINGTON (AP) — The nation's largest veterans organization and House Speaker Nancy Pelosi on Saturday joined the growing calls for the ouster of President Donald Trump's Veterans Affairs chief, under fire after a government audit found he acted unprofessionally, if not unethically, in the handling of a congressional aide’s allegation of sexual assault at a VA hospital.

“It is unfair to expect accountability from the nearly 400,000 VA employees and not demand the same from its top executive. It is clear that Secretary Robert Wilkie failed to meet the standard that the veteran who came forward with the complaint deserved,” the American Legion's national commander, James W. “Bill” Oxford, said in a statement. He urged Wilkie and several other top VA officials cited in the report to resign because of their “violation of trust” of the agency's commitment to not “tolerate harassment of any kind.”

Pelosi, D-Calif., said Wilkie “has lost the trust and confidence to serve, and he must immediately resign." She said Wilkie “has not only been derelict in his duty to combat sexual harassment, but has been complicit in the continuation of a VA culture that tolerates this epidemic.”

On Saturday, the VA said Wilkie, who has denied wrongdoing, doesn't intend to resign. “He will continue to lead the department,” said spokeswoman Christina Noel.

The demands for Wilkie's resignation came a day after numerous veterans groups expressed similar outrage and sought Wilkie's dismissal in the final weeks of the Trump administration. Those organizations include Veterans of Foreign Wars, Iraq and Afghanistan Veterans of America, Disabled American Veterans, AMVETS, Paralyzed Veterans of America and the Modern Military Association of America, and they said they had lost confidence that Wilkie could effectively lead the department, which is responsible for the care of 9 million veterans.

An investigation by the Veterans Affairs’ inspector general on Thursday concluded that Wilkie repeatedly sought to discredit Andrea Goldstein, a senior policy adviser to Democratic Rep. Mark Takano, chairman of the House Veterans' Affairs Committee, after she alleged in September 2019 that a man at the VA medical center in Washington, D.C., had physically assaulted her.

The inspector general found that Wilkie’s disparaging comments about Goldstein, a Navy veteran, as a repeat complainer as well as the overall “tone” he set influenced his staff to spread negative information about her while ignoring known problems of harassment at the facility.

Wilkie and other senior officials had declined to fully cooperate with the investigation by VA Inspector General Michael Missal. For that reason, Missal said he could not conclude whether Wilkie had violated government policies or laws, allegedly by personally digging into the woman’s past. Wilkie has denied that he improperly investigated Goldstein.

“We’ve had our concerns about Wilkie’s leadership throughout the pandemic and this IG report really cements the fact that the VA is not being led with integrity,” said Jeremy Butler, chief executive of Iraq and Afghanistan Veterans of America. “That calls for an immediate change.”

The report on Thursday drew widespread concern from lawmakers from both parties about VA’s leadership, with Takano the first to call for Wilkie’s resignation. Concerned Veterans for America, a conservative group who supported Wilkie when he became VA secretary in 2018, chided Wilkie and his team, stressing that “VA leaders should always put the veteran and the integrity of the institution ahead of themselves.”

AMVETS national commander Jan Brown said she found it unacceptable that VA would dismiss known problems facing women who receive care at its facilities.

“Women veterans already hesitate to use VA services for a number of reasons and we need a Secretary who will make our community feel welcomed,” she said. “We strongly disapprove of any VA official that took part in the scheme to wreck the credibility of a victim.”

The case of Goldstein, who agreed to be publicly identified, was ultimately closed by the inspector general’s office and Justice Department earlier this year due to a lack of enough evidence to bring charges.

Wilkie is Trump’s second VA secretary after David Shulkin was fired in 2018. A former Pentagon undersecretary, he presided over the nation’s largest hospital system that has seen continuing improvement and veterans’ satisfaction since a 2014 scandal involving lengthy waiting times for medical appointments.

President-elect Joe Biden has pledged to rebuild trust in the VA when he takes office on Jan. 20. He has selected Denis McDonough, who served as President Barack Obama’s White House chief of staff, to be VA secretary.