Despite cast members like Michelle Dockery's eagerness to continue the show for as long as possible, it seems the party may well be over.
Speaking about his similar US project, The Guilded Age he said, "I just couldn't do both at once."
Although a fifth season has been confirmed, he added, "I don't know yet if there is a season six."
ITV has not confirmed one way or another but there is speculation that even if Julian leaves, they will hire new writers.
We reckon this could spell disaster for an already floundering show.
Who remembers Mary and the Turkish diplomat? Those were the days.
SHERLOCK TO FACE SHOCK NEW VILLAIN IN SEASON 4
THE TV SHOWS RETURNING IN 2014
MORE TV NEWS AND GOSSIP