Millwall is my local club, im 18, black african and went to school in Bermondsey. What i've seemed to notice is that away from football, even some of the most die-hard fans are 'normal' however when you group them together it's like they lose all sense of...sense. The area Bermondsey in itself is pretty normal, to call it wealthy would be a stretch but considering it's close to areas such as Peckham, Walworth and New Cross, its hard for me, as a local, to see it as deprived, although i think that says more about South London as a whole.
The weirdest thing is despite all the racist abuse i've heard from Millwall fans, i don't think many ARE racist. It's hard to believe i know, but from what i've seen many of them see it as a joke or harmless banter. Another thing i've noticed is that it's ingrained in them from young, Millwall fans i went secondary school with were pretty much just wannabe thugs with high pitched voices.
EDIT: I will say this, i can only speak of what i've seen in the past 3-4 years, i know historically Millwall are a vile club.