And now the big question of who will join Simon Cowell (who's also returning), Louis Walsh and Cheryl may have been answered - Fergie.
Cheryl told press, "There are people I would love to join on the panel. Especially a strong woman, preferably.
I'm kind of hoping I don't have to deal with three guys otherwise I may be appearing on a Saturday night with no hair because I've pulled it out."
A source for The Mirror has said, "'It's not a done deal just yet but things have been positive.
Both sides are keen to move things forward and people at the show and ITV are excited at the prospect of having Fergie.
'She brings a lot to the show and her success globally with The Black Eyed Peas and her solo albums obviously speaks for itself.
On top of that, she looks fantastic and fans love the glamour of the live shows."
Since Rita Ora officially turned down an offer and Nicole Scherzinger's out, we've been thinking of other possible alternatives, if the Fergie option falls through.
With the return of two old faces and ever present Louis Walsh, Simon may want to go for the full retro effect.
Also, seeing her sister Kylie Minogue on The Voice could well have inspired Dannii Minogue.
Rumours have circulated that Cheryl wants her LA friend Khloe Kardashian to get a spot on The X Factor.
However, with her presenting gig on the US version going rather badly, we can't see her taking over from the amazing Dermot O'Leary.
We love Derms.
Sharon did say she would only do one series, but needs must. We also think she certainly counts as a strong woman.
Plus the series without that laugh would be a bit sad.
We reckon a Spice Girl could fit well into the set up and with Mel B offending a few too many people on her stint on America's Got Talent, we're not sure she'll make it.
We also caught a glimpse of Emma on The Big Reunion the other week and remembered why we loved her.
They may not want to sweet girls on the panel though...but it could be a good balance to Simon?
Just because Cheryl wants a girl, doesn't mean she's going to get it.
Olly Murs is at home at The X Factor and will have finished his tour with Robbie Williams and general world domination by then.
Just as long as it's not Robbie. Dear god.
Who do you want to join them? Send us your ideas @handbagcom
CAN CHERYL COLE SAVE X FACTOR?
HOW TO MAKE A COMEBACK LIKE CHERYL COLE
MORE TV NEWS AND GOSSIP