He said the RHA had presided over a steady souring of relations with the Government since the 2016 Brexit referendum.
The group officially took no position on the vote, but it has blamed Britain’s exit from the EU for exacerbating the lorry driver shortage and piling extra paperwork on haulage firms.
Mr Baxter said: “We have a haulage industry body that is not effectively representing the industry.
“The RHA has been hijacked by people who passionately believe in Remain and at every turn want to make some Brexit-related point.
“I don’t think that’s constructive. All they are doing is alienating people in Government instead of putting forward realistic policy proposals.
“I know lots of other people in our sector who are also unhappy about what is happening in the RHA but they do not want to put their heads above the parapet.”
He has called for Richard Burnett, the RHA’s chief executive, to be replaced with someone who can repair relations with ministers.
The RHA said it had never expressed a view on Brexit and respects the UK’s democratic decision to leave the European Union.
A spokesman added: “However, we have at times reflected our own members’ concerns and their worries were communicated to government and officials as part of our normal activity on behalf of them.”