LOS ANGELES - Simon Cowell is playing it coy about rumors that Fergie, Britney Spears and Janet Jackson are being considered for "The X Factor."

Two judges' spots on the Fox TV singing contest opened up when Paula Abdul and Nicole Scherzinger left after the first season. Cowell told a teleconference Wednesday that two women will replace them, but he declined to comment on who's being considered.

"There's been a load of speculation, some true, some not true," Cowell said. He added that he waited to see who was interested enough to ask about joining the show before focusing on the search.

Judge Antonio "L.A." Reid confirmed there had been discussions about adding Whitney Houston but no contact was made with her. Houston died Feb. 11.

Meetings will be held over the next few weeks with "a number of people," Cowell said, primarily to make it clear to contenders how much of a time commitment "X Factor" represents.

He didn't shed much light on the departures of Abdul and Scherzinger, saying only that shows evolve and "this time it was decided to do it sooner or later." He said the two have been "incredibly gracious and respectful" and he's looking forward to working with them again.

With host Steve Jones' exit, Cowell said he wants to switch to a two-host format that would better suit the show and the amount of information that needs to be explained, entertainingly, to viewers. He didn't say who might be hired but said they could be people who don't have "host" on their resume.

He's not looking far for new judges and emcees, Cowell indicated, saying, "it's important we have more Americans on the show."

Cowell, who's a judge on the Fox show he based on his hit British series, is English. Jones is Welsh.

He was guarded about guessing the ratings for "X Factor" when it returns this fall for its second season. After Cowell predicted it would be a hit in its freshman outing with 20 million viewers, the show averaged around 12 million weekly viewers, which is respectable but not a blockbuster.

"I should have kept my mouth shut," Cowell said. It took time for "American Idol," the home he and Abdul once shared, and NBC's "The Voice" to grow in the ratings, he said, expressing confidence the same will happen for "X Factor."

"I absolutely expect the second season to do better than the first season," he said. "We're not doing this to be second or third. ... We've learned how to make the show better."