I don't think I am and I don't think you can generalise every neutral fan in that way. It could depend on the clubs involved as well. For example, I consider United, Liverpool, Spurs, and Arsenal all as our rivals on the pitch these days. If I was invited to a United v Liverpool game I probably wouldn't have much positive to say about either club in fairness and would want you both to get beat if it was possible
However, if I was invited to an Arsenal v Spurs game, I'd be able to offer up a far more unbiased and less critical opinion.
Speaking of Liverpool, one of our customers at work is a Liverpool fan but far from being bitter towards rival clubs, he's totally magnanimous - he often phones us up at work if City or United have had a good result to congratulate me and the United fan. He's way too polite to be a football fan and a little bit out of touch as evidenced when my boss invited him to a United v Liverpool game a couple of seasons ago, stating that he was shocked to hear how vitriolic some United fans were towards the Liverpool team and fans