Khloe Kardashian's baby rumors is apparently stressing out estranged husband Lamar Odom as rumored beau French Montana's neighbors are reportedly not very happy about the "Keeping Up with the Kardashians" star being in their "hood."

Khloe Kardashian is reportedly shacking up with the rapper's new home and they are reportedly planning to have a baby together, HollywoodLife.com reported.

The news doesn't bode well from Lamar Odom's plans to win back his wife and his friends consider the situation serious enough that they fear for his safety as Khloe Kardashian baby rumors could push him into using drugs again.

"Lamar will be devastated and betrayed if she and French have a baby," the May 5 issue of In Touch Weekly, citing as source the friend of the basketball star who signed on with the New York Knicks.

While the Khloe Kardashian's baby plans should be welcomed as good news, her presence in French Montana's neighbourhood is not as welcomed, according toRadar Online.  

According to the gossip site, French Montana's neighbors are worried that the circus surrounding the Kardashians-the paparazzi and the partying-might spill over to their quiet community.

"This is a quiet bedroom community. No one wants Khloe around and all of the ruckus she brings," an insider told Radar, who claimed that her presence in the neighborhood is going to be a "complete nuisance."

The source also confirmed that Khloe Kardashian spent numerous nights holed up in French Montana's home, the most recent of which happened last weekend.

The rumors about Khloe Kardashian dating French Montana have been splashed in tabloids in recent weeks, after they were seen several times together.

Although the pair hasn't issued any statements to confirm their relationship, it seemed serious enough that the rapper's girl, Trina, was already going the rounds complaining about being left out.

TMZ had reported that both Trina and the rapper were not officially on but they were already living together in New Jersey.

The gossip news site added that Trina and French Montana were communicating regularly "until a few weeks ago" when he was started recording his next album in L.A. However, "now it's radio silence."