It goes back to the 1970 merger.
Patriots had some extremely difficult days in the early '70s as well as most of the 80s and the early 90s.
I might not be in total agreement with the rankings but I sorta understand why the Patriots fall where they fall. It's not because of the past decade or so, its the history before that.
EDIT: Also, this poll takes into account, MNF football appearances. For quite a while, I believe '81 or so until '95, MNF wasn't even allowed in Foxboro. That's a severe hinderance alone.