The 54-year-old actor-and-director's new movie centres around the kidnapping of a Los Angeles mayoral candidate's son, and he initially had friend Matt Damon on board to play the politician and was keen to sign up his former wife opposite his pal.
Affleck noted bringing Garner - who he split from in 2015 - on board would have had an "added benefit" for them both in the form of more time with their kids, Violet, 20, Finn, 17, and 14-year-old Samuel.
He told Entertainment Weekly: "I thought, God, Jen Garner would be amazing for this role.
"It also has the added benefit of, like, both of us will see our kids more because we do a week and a week and back and forth. I thought, well, we can coordinate our schedules around it."
However, the project was then scrapped and when Affleck was able to revive it, Damon was no longer available due to his commitment to the Odyssey, prompting the Argo filmmaker to cast himself instead.
However, Garner wasn't keen to bring their "baggage" to the screen, particularly as the couple in Animals have a rocky marriage.
He said: "I talked to Jen, and she was like, 'We don't wanna do that. You just get baggage.'"
However, Affleck is still hopeful his ex-wife will work with him in the future because he thinks the Alias star is "fabulous".
He said: "I would love to work with her, frankly.
"She's fabulous, and we're really good friends, and I'm profoundly grateful to her, and we have a great thing. I still would wanna work with her in the future, but what you don't wanna do is take the audience's baggage and have it work against you, you know?"
After Garner, 54, declined the role in Animals, Ben brought in Kerry Washington instead after getting to know the Scandal star during the Hollywood writers' strikes.
He said: "She's very polished, but she's also very warm. I thought, 'Oh my God, she's amazing for this.'"
And bringing Washington on board saw Affleck make some changes because he didn't want to "ignore" the fact their characters were in an interracial marriage.
He recalled telling his co-star: "What I don't wanna do is ignore it.
"This is a movie about real life and real people, and our ethnicity is as much a part of how people see us as anything else.
"So, then we built this whole backstory around that, which opened the door to really unexpected moments in scenes that were just honest and real."
11°C