And it sounds like we may have the Americans to blame.
The period drama found serious popularity across the pond and since then the writer and creator Julian Fellowes has been working on a US version.
So if you'd been wondering why none of the story lines seemed that developed or had just noticed Downton just isn't as good as it used to be, you now have an explanation.
A source told The Sunday Express, "Some of the plots during the [current] series haven't run that coherently because Fellowes has been writing his new American thing at the same time, jumping from one to the other.
It hasn't had his full attention."
Well thank you very much America.
There is also talk that the next, the fifth series, will be its last.
The actors are apparently eager to go out on a high and pursue other paths, like the now-skinny, success story, Dan Stevens.
The project Fellowes is working on for NBC is about the Vanderbilt family in New York, set in the 1880s.
So perhaps fans will have to look there for the next big costume drama.
IS GOGGLEBOX THE "BEST SHOW ON TV"?
MADE IN CHELSEA SERIES 6: WILL JAMIE EVER LEARN?
MORE TV GOSSIP AND NEWS