After nine years in the acting industry, the then 24-year-old felt he was no longer the person to tell any of the stories that were coming to him.
"I'd get a piece through and I'd be like, 'so and so would be fantastic at doing that. I don't know why they're giving it to me'," the Australian-born actor tells AAP.Â
"Even for the jobs that I got."
So he decided to retire.
For many actors, self-doubt is a familiar feeling throughout their careers.
Take Tom Hanks, who admitted his belief people were going to discover he "was a fraud", or Kate Winslet, who was afraid of people thinking they had "cast the wrong person".
But a last-minute call from his agent pleading for a final audition for the prequel to hit historical fantasy series Outlander marked a "sliding doors" moment for the young actor.
"This came through; I remember reading that and going, 'I can f***ing tell that story'," he says.
Fast-forward two years, Retford is starring in a leading role as Dougal MacKenzie in Blood of my Blood, chronicling the romantic saga surrounding the parents of the main series protagonists.
It's a day off for the 26-year-old, who's based in Glasgow shooting the already greenlit second season, but Retford plans on spending it practising stunts in the garden.
Several weeks earlier, the cast had been in Hollywood for the series premiere.
"We got off the plane in Los Angeles at the airport and there's all these people with posters. You've got billboards, and everyone's screaming. We're on some trams in Sydney," he says.
Retford's on-screen persona, Dougal MacKenzie, whom he views as a "revolutionary", is a brutish warrior and, at times, reckless when it comes to the family name.
But don't ask about comparisons between him and the character he has lived and breathed for two years.
"You have to be completely on board with all of your character's decisions," he says.
While many who have watched the series have heaped praise on the cast, Retford admits he steers clear of his own performances.
"Either you watch it and you're like, 'oh, that was really good', which I never think, or you watch it and you just slate yourself, and that also isn't good," he says.
"At the end of the day, it's creativity. I'm not in this to be the best actor on the planet, because I know I'm not, and I've no interest in being one of the greats."
In Retford's eyes, there are two sides to the industry: the "athletes", who study the craft and push acting as far as it can go, and there's the other side.
"We are entertainers and jesters, in a way still storytellers, and we're here to give people an hour a week of disconnect from whatever's going on in their life," he says.
Born in Gosford, on NSW's Central Coast, he grew up along a dirt track speckled with a couple of houses surrounded by dense eucalyptus.
As a teen, the family moved to England, and his need to become a "chameleon" to fit in was when he got his first taste of acting.
Retford credits his diving instructor mum for his love of nature and travel and also his most unconditional supporter.
It's something Retford has taken with him through life as he travels the world, riding motorbikes, diving, caving and camping in the wild.
But don't mistake him for an adrenaline junkie.
"All of the sports I pursue are all ultra-mindful. You can only think about the thing you're doing whilst you're doing it."
He admits Australia is a place he'd like to do a pilgrimage back to, but that will have to wait as he faces a second season in the Scottish highlands.
5°C