I can't remember which one it was but there was an ex-EPL ref (from the early 00's, someone else might know) talking at half time on one of the American broadcasters about why the FA protect their refs so much and take away all accountability. Basically, there's only a handful of refs in the country that are at the level required for the EPL. You just don't appreciate how difficult a job it is, how fast it is, how hard it is to cut through the play acting, to see everything through the crowds at 100mph and then be 60 metres down the pitch doing it there as well.
So if they start hanging refs out to dry for their poor decisions or start demoting them on every bad call, not only does it undermine their already fragile authority, it will make their profession downright miserable and unmanageable and there will be no queue to do an already unpopular job and eventually the quality will go down and down.
He also said that the majority of referees know they aren't perfect and are all in full support of video refs making their life easier and having a more informed decision on the big calls.