Could the same argument be made with the words "black" or "brown"?
Yes. Its an outdated classification that never had scientific basis and is loaded with racial bias.
"According to Mukhopadhyay, Blumenbach went on to name four other “races,” each considered “physically and morally ‘degenerate’ forms of ‘God’s original creation.’” He categorized Africans, excluding light-skinned North Africans, as “Ethiopians” or “black.” He divided non-Caucasian Asians into two separate races: the “Mongolian” or “yellow” race of Japan and China, and the “Malayan” or “brown” race, which included Aboriginal Australians and Pacific Islanders. And he called Native Americans the “red” race.
Blumenbach’s system of racial classification was adopted in the United States to justify racial discrimination—particularly slavery. Popular race science and evolutionary theories generally posited that there were separate races, that differences in behavior were tied to skin color, and that there were scientific ways to measure race. One way racial differences were defined was through craniometrics, which measured skull size to determine the intelligence of each racial group. As you can imagine, this flawed application of the scientific method resulted in race scientists developing a flawed system of racial classification that ranked the five races from most primitive (black and brown races), to more advanced (the Asian races), to the most advanced (the white, or Caucasian, races). Even though the five-race topology was later disproven, “Caucasian” still has currency in the U.S.
One reason we keep using the term “Caucasian” is that the U.S. legal system made use of Blumenbach’s taxonomy. As early as 1790 the first naturalization law was passed, preventing foreigners who were not white from becoming citizens. But according to Mukhopadhyay, Blumenbach’s category of “Caucasian” posed a problem because his classification of white also included some North Africans, Armenians, Persians, Arabs, and North Indians. The definition of Caucasian had to be reinvented to focus the ideological category of whiteness on northern and western Europe. The term, even though its exact definition changed over time, was used to shape legal policy and the nature of our society...
The bottom line is that it is time for a modern—and accurate—terminology. The use of an outdated and disproven term that falsely purports to describe a separate race of people has no place in the U.S."
https://www.sapiens.org/column/race/caucasian-terminology-origin/