Joaquin Phoenix and Lady Gaga play two of DC Comics’ most psychologically twisted characters in Joker: Folie à Deux. But they remained “really funny and normal” on set, according to writer-director Todd Phillips.

“I know that people always think actors are method actors,” the filmmaker, 53, tells PEOPLE. “I personally don’t even know what it means when people say that because I’ve never seen that on any movie.”

The sequel to 2019’s Oscar-winning origin story Joker (in theaters Oct. 4) follows Phoenix’s Arthur Fleck as he is on trial for murder. At Arkham State Hospital where he’s incarcerated, he “not only stumbles upon true love, but also finds the music that’s always been inside him,” per an official synopsis.

Phillips acknowledges that Batman villains make for disturbing material. Yet Phoenix, 49, he says, “could be as dark as he is in some of these scenes. And I say, ‘Cut,’ and he comes over and tells me a joke about something totally nothing to do with it, and we’re just laughing on the floor.”

He adds, “Now, that’s my experience with Joaquin. He might be different on another movie, but my experience is that’s how he’s been.”

Gaga, 38, plays Folie à Deux’s similarly dark Arkham patient Harleen “Lee” Quinzel a.k.a. Harley Quinn. As the actress-musician tells PEOPLE, she “didn’t really” leave the character behind after filming. “I mean, I didn’t bring her home with me,” she says. But “you fall in love with the characters that you play. I always have, anyway.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories. 

Phillips says casting the A Star Is Born Oscar winner came out of developing the sequel with Phoenix and realizing Arthur’s inner psyche could be expressed through song. “We just found ourselves writing more and more music into it. And I, at some point just thought, ‘I want to find an actor who brings music with them.’”

Of course, Gaga’s star power was also a factor. “One of the benefits of the first Joker was we really flew under the radar,” recalls Phillips. “And then, you become a big success, and now you’re under a microscope. And then… you take a global superstar and bring her into our little world here, and all of a sudden, the spotlight gets even that much brighter.”

The writer-director’s biggest hurdle, he says, was not focusing on fans’ expectations or response to the sequel. “In today’s day and age — in general, not just this movie — there’s always noise surrounding projects,” says Phillips. “I would say the biggest challenge is all the outside noise and just blocking it out. But I’ve become very adept at it.”

Joker: Folie à Deux, costarring Brendan Gleeson, Catherine Keener, Leigh Gill, Zazie Beetz, Sharon Washington and Steve Coogan, is in theaters Oct. 4.