Nicole Richie and Joel Madden are not happy with each other as rumors say. The couple was apparently fighting over the idea of having more kids.

As reported by Dish Nation, the Good Charlotte frontman wants more kids but the fashion designer mom reportedly won't let it happen.

"Joel and his twin brother Benji come from a large family with four kids," an insider told the source above. "And growing up Joel always wanted to have a big family too."

"Joel is adamant about adding to their family and insists they can well afford nannies and housekeepers, but Nic isn't buying his sales pitch," the insider added.

Although the couple has not addressed the recent Nicole Richie Joel Madden rumors, the supermom has expressed recently that she's done with kids for now.

"I am not planning on any more, but you never know," she told E! Online in July. "I don't wanna say I don't want them and then they come and I feel bad."

Richie and Madden currently have two kids -- Harlow Winter Kate and Sparrow James Midnight. The couple seem to be enjoying parenthood at the moment.

"Right now, I know how to be a mother to [a 7-year-old]," Richie has told The Daily Telegraph in March. "When they were born, I knew how to be a mom to a baby."

"I will have to see how I'm going to be when they're teenagers, but I think it also depends on who they end up being," she added. "I don't believe in black-and-white parenting ... I'm going to have to figure it out as it happens."

Just days ago, the family celebrated Sparrow's sixth birthday. Richie took to Instagram to post a picture of her and baby Sparrow years ago. Madden, meanwhile, posted a picture of both their kids and simply captioned it "fatherhood."

While Nicole Richie and Joel Madden seem to be done with kids for now, their siblings Cameron Diaz and Benji Madden are reportedly awaiting their first kids. According to a report on Hollywood Life, the newlyweds are purportedly expecting twins.