Brizzy, it's like a lot of things it is context that may or may not make the use of a word offensive. Millwall as used to describe an area of London/ Middlesex is quite innocuous but used as a term in football it is highly offensive and inappropriate.
Ooh, that Millwall! I don't think you'll find M'sex went south of the river. The New Den is, in A London borough. That's the Millwall I know and love, darling!