Surely it should be the goalkeepers responsibility to not get distracted by players moving away from the goal, as long as they're not actively blocking his line of sight? Obviously, if they're blocking the goalkeeper's view of the ball, they're offside, but if not, the goalkeeper has to be responsible for what he looks at, no?
If I'm the last defender and I'm picking my nose and looking at an injured player near the corner flag while ignoring where the ball is, does that make the injured player offside?