Well, perhaps not at the exact moment he was sat in the stalls, but he's certainly got some plans for the new series of the show.
Simon is desperately trying to claw the viewers back from Strictly Come Dancing and find the new One Direction style mega-stars.
So, how's he going to make the show more exciting - other than his and Cheryl Cole's return?
One option being considered is ditching Boot Camp all together.
"Simon thinks the Boot Camp process can drag on a bit and isn't very exciting," a source told The Sun.
"The plan is to axe it and just focus more on the six-chair challenge. Simon loves it - the drama and controversy is right up his street."
The idea was backed up at The Mirror Online, where a source claimed, "It is still to early to say what the show is going to look like but Simon has confirmed the six-chair challenge will stay and Boot Camp and the rest of the show is being looked at."
Would you miss bootcamp? Tweet us @handbagcom
IS FERGIE TAKING THAT FOURTH X FACTOR SPOT?
CAN CHERYL COLE SAVE X FACTOR?
MORE TV NEWS AND GOSSIP