His time here is judged harshly because it coincided with a period of transition. Ronaldo and Rooney were our most talented players at that time (at least, from 04-06). Yet, neither was consistent enough to spearhead a title winning campaign, despite both being evidently far too talented to relegate to the margins. Keane was also coming to an end, Arsenal were in their "Invincibles" period, and Mourinho's first Chelsea (one of the best teams in the history of the Premier League), was completely dominant.
That we didn't win enough during that period was in spite of Ruud, not because of him. He'd easily score 50+ goals a season in the team we have now.