"I was witness to the full force of human grief, and I was six years old and there was nothing I could do about it. And I think that probably marked me for life," he said in a 2007 interview.
In the first volume of his memoirs, James wrote about his lack of paternal guidance and his subsequent amusing but hair-raising childhood escapades.
He not only detailed the antics of his fatherless youth but the subsequent stress inflicted on his widowed mother, and admitted to a self-destructive streak in a 2012 interview.
An ardent smoker, he once filled a hub cap with cigarette butts in one day.
After leaving Sydney Technical High School, James studied psychology at Sydney University, where he edited the university's student newspaper.
He soon became associated with the Sydney Push, a group of liberal-thinking intellectuals and, aged 22 and uncomfortably aware of his mother's proximity, he fled to London.
The lure of swinging London
London in the 60s was a far cry from Sydney, and James lived a bohemian existence alongside fellow Australian émigrés Robert Hughes and Germaine Greer.
While studying at Cambridge University, he began contributing to various undergraduate periodicals and his writing soon came to the attention of London's literary editors.
He also found himself president of Footlights, the university's amateur theatrical club.
VIDEO: Clive James reflects on interviewing Roman Polanski, Katherine Hepburn (An excerpt from ABC TV's Clive James: The Kid From Kogarah) (ABC News)
In 1972, The Observer newspaper hired James to write a weekly column of humorous and scathing television reviews, which ran for 10 years.
It was during this time that James first appeared before the cameras, gradually becoming a renowned television presenter while also writing and hosting numerous TV series and specials.
These included Clive James on Television, Fame in the 20th Century and the pioneering travel program series, Postcards From ... .
Monty Python star Eric Idle called James his "pal at Cambridge" and said it was "savage news" to hear of James's death, coupled with the death of theatre director Jonathan Miller, announced on the same day.
"To lose one friend is bad, but two reeks of carelessness," he wrote on Twitter.
"It's a f***ing rainy day in LA appropriate for tears."
Another giant of UK comedy, Stephen Fry, paid tribute to James (as well as Miller), describing them as "heroes" of his.
James retired from television in 2001 to focus on his writing, and began presenting a weekly BBC Radio 4 broadcast, A Point of View.
It gave him the opportunity to deliver pithy reflections on issues ranging from politics to pop culture in a series of vocal "essays".
"The secret of criticism is to know what your real feelings are before you try to express them," he once said.
Declining health did not slow him down
Despite his declining health, James did not abandon his career — just redefined it on his terms, from the comfort of his home.
Most recently, he had focused his creative abilities on his personal website, a platform for his cultural critique of art, music, poetry and literature.
It was here also that James showcased his series Talking in the Library, a collection of interviews with charismatic individuals held in his home.
While maintaining his website and attending multiple medical appointments, he also continued to write a weekly television column for The Telegraph in London.
But it was not all literature for the multi-lingual James who was a fan of Game of Thrones, Formula One racing and art.
Tango dancing, another great love, led him to Buenos Aires to learn the technique before installing a dance floor in his flat in London.
An unlikely Lothario, James had a deep admiration for women, an interest which resulted in his being evicted from the family home in 2012 following revelations of an eight-year affair with a former model.
"I realised that being a married man was the centre of my existence and the anchor," he told Kerry O'Brien in a 2013 interview.
"I'm not built for it. I'm built to be Ulysses — not physically perhaps!"

James addressed the affair in a poem titled Lecons des Tenebres (Lessons of Darkness) that was included in his 2015 book of poetry, Sentenced To Life.
"Far too casually I broke faith when it suited me, and here I am alone and now the end is near," he wrote.
After the publication, he admitted to the BBC he "was a bad husband" and apologised for his mistakes.
"I mustn't be too facile about it or actually talk too much because I have a pact with my family that they'll execute me if I do," he said.
"But yes, I could've behaved a lot better and [I'm] sorry I didn't."
James made his last stage appearance at London's inaugural Australia & New Zealand Festival of Literature & Arts in June 2014, and shared his effusive wit and humour — and the true poet within — with his audience.
"The poetry I write now, I think, is quite a lot more penetrating and sensitive than my earlier work — because it needs to be," he said.
"Inevitably you start saying goodbye. I like to think that I hit a sort of plangent tone of threnody, a sort of Last Post, a recessional tone."