Dominic Raab dismissed repeated warnings about his behaviour, a retired senior civil servant who worked with him in the Foreign Office has said.
Lord McDonald described his former boss as a “tough taskmaster” whose methods did not help him achieve his aims.
He told the BBC he raised this with Mr Raab more than once but he “disputed” it and was unwilling to listen.
The former deputy PM, who resigned on Friday after a bullying inquiry, apologised if he upset anyone.
The inquiry found he was “intimidating” and “aggressive” towards officials, but Mr Raab said his behaviour was not bullying, and that almost all of the complaints against him were dismissed.
It looked at eight formal complaints about Mr Raab’s behaviour during his previous stints as justice secretary, foreign secretary and Brexit secretary.
Senior lawyer Adam Tolley KC, who led the inquiry, concluded Mr Raab’s conduct involved “an abuse or misuse of power”, and that he “acted in a manner which was intimidating” and “persistently aggressive” towards officials.
Mr Tolley said he found a description of bullying had been met when Mr Raab was foreign secretary and justice secretary.
But, in relation to complaints from his time as justice secretary, Mr Tolley concluded Mr Raab “did not intend by the conduct described to upset or humiliate”, nor did he “target anyone for a specific type of treatment”.
Lord McDonald, who worked as Mr Raab’s permanent secretary in the Foreign Office, told BBC Radio 4’s Today Programme: “I witnessed a tough taskmaster. I witnessed a minister who knew what he wanted to do.
“Frankly, I witnessed somebody whose methods did not help him achieve what he wanted to do and that I raised with him more than once.”
Asked if Mr Raab had listened, he replied: “No, he disputed it, he disputed the characterisation.”
Lord McDonald said he had not used the word ‘bullying’ to Mr Raab, partly because things had not reached that stage at the time.
“I was trying to get my boss to see how his behaviour was making his professional life more difficult. I was trying to help him get the best out of his team and I felt saying bullying would have been too aggressive,” he said.
Lord McDonald was not at the Foreign Office at the time of the incidents which were complained about and upheld by the inquiry.
Mr Raab, a close ally of the prime minister, stood down on Friday after previously vowing to resign if the inquiry found he had bullied people.
In his resignation letter to the prime minister, he accepted the inquiry’s findings but said they were “flawed”.
Writing back, Rishi Sunak said his former deputy had kept his word, but he thought there had been “shortcomings” in the process and had asked civil servants to look at how complaints are handled.
In his first interview since stepping down, Mr Raab told the BBC the findings set “a very dangerous precedent”, with many ministers now fearful that they may be treated the same if they “fairly” bring “direct challenge” in government.
“If the bar, the threshold for bullying is lowered that low, it’s almost impossible for ministers to deliver for the British people and I think it’ll have a chilling effect on effective government, and the British people will pay a price,” he said.
Asked if the blunt truth was that he was a nightmare to work for, the former justice secretary said: “Well actually, almost all of the complaints against me were dismissed.”
And asked if he wanted to apologise, he said: “If someone had hurt feelings, because of something I did, of course, I want an empowered team.
“The vast majority of the civil servants who worked for me were brilliant, fantastic and actually relished the energy, the challenge, the drive that I believe I brought.
“But of course, I don’t want to upset anyone and I made clear that I’m sorry for that. But that’s not bullying, and we can’t deliver for the British people if the bar is that low.”
Mr Raab also said a “very small minority of very activist civil servants” were effectively trying to block reforms they did not like, related to areas including Brexit, prisoner parole and human rights.
“That’s not on. That’s not democratic,” the MP for Esher and Walton said.
But Lord McDonald denied there was any civil service activism, passive aggression or a separate civil service agenda.
He added: “I saw no evidence of a small group of activists trying to undermine a minister. The issue is a minister’s behaviour.”
The FDA union, which represents civil servants, has accused Mr Raab of peddling “dangerous conspiracy theories that undermine the impartiality and integrity of the civil service” to “deflect from an independent investigation’s criticism of his conduct”.
But Conservative peer Lord Marland said Mr Raab’s resignation was “almost a conspiracy by the civil service”.
Speaking to the BBC’s Newsnight, he said it was “a very black day for all employers” because a “dangerous precedent has been set” [on the issue of workplace bullying] that would “send shudders through all employers in the country”.
Hannah White, director of the Institute for Government think tank, said “no civil servant would feel encouraged to speak out in future” after the responses of Mr Sunak and Mr Raab to the Tolley report.
She said Mr Sunak had missed an opportunity to reinforce standards and “the mutual suspicion which has been growing between ministers and civil servants remains and nothing has been done to reduce the risk of future problems.”