Bryce Lawrence says his performance in last year's World Cup quarter-final was influenced by complaints from ARU chief executive John O'Neill.
Retired ref Lawrence claims he was hounded out of rugby by South Africa and Australia following his showing in the Springboks and Wallabies match.
"I had four really good games at the World Cup and then I had that," the 41-year-old told the Bay Of Plenty Times.
"There was some pretty nasty political stuff going on about that appointment. I refereed Australia versus Ireland and Ireland had won but behind the scenes guys like John O'Neill were kicking up a massive stink.
"I knew a bit about that and it was enough to affect me, and it probably made me freeze on the biggest stage."
Australia won the game 11-9 which has now led to Lawrence calling it quits and taking up a role as the New Zealand Rugby Union high-performance referee reviewer. He was stood down from the International Rugby Board's elite panel after the World Cup and didn't referee any Super Rugby matches in South Africa.
"It (the reaction) got pretty bad," Lawrence said.
"On Facebook they launched a 'get rid of Bryce Lawrence' site and it was pretty nasty.
"That was absolutely the reason for my career change.
"I got told at the end of the World Cup that I would have a break from Test rugby for the Six Nations and I could totally accept that as there has to be a consequence for poor performance.
"I was told I would be brought back in the middle of this year, as I was ranked in the top three or four referees in the world. But because of the political reaction from rugby unions like Australia and South Africa behind the scenes, they dropped me."
Last month, British & Irish Lions boss Warren Gatland highlighted the pressure Lawrence had been placed under by O'Neill as he outlined the challenges that await his squad.
"They are masters at it and possibly the best one was John O'Neill, as a master of influence in certain things," said 2013 touring coach Gatland.
"I'm not 100% sure, but I think that after Ireland beat Australia in that pool game, certain complaints were made about the referee (Lawrence), subtly and tactfully, and I think that had an impact on the quarter-final.
"We've got to be aware about what sort of things are going to be done on and off the field."
O'Neill is on his way out of his post at the ARU to focus on business interests outside rugby.