In fairness I don't know if anyone really knows what's ok to say these days. There are certain words which are clearly derogatory like half-caste and I think everyone knows are completely unacceptable. My father in law who is black, for example, has no issue with 'coloured' and pointed out he'd rather be that than 'colourless'. Also people say 'person of colour' all the time so it's counter intuitive. I have seen people get into trouble for saying 'brown' and also seen people argue online about why you should never say 'Indian' for someone from India or how African-American is not acceptable anymore. If someone makes a one off error which isn't clearly meant in a racist way, I hope people give them the benefit of the doubt the first time.
I'd say in this specific case if he has no history of racism, I hope common sense prevails.