As far as I know, Joseph is white. He's described as identical to the modern inhabitants of the Basque area, who are white (as white is defined in Europe, which isn't always identical in looks to WASPS). He also passes without notice in California in the 1920s.
I think he may be very well be a little darker, but it's a long way from "olive skin" to "non-white."
Budu and the enforcers are all not-of-any existing race, and he IS a major character.
Nan d'Araignee is certainly a major character, as are:
Suleyman - African
Latif - African
Sarah (Sarai) - African
Nefer - African
Nicholas/Edward/Alec, while 'somewhat' Caucasian are Recombinants made with DNA from 'Homo crewkernensis'
I think more than most science fiction authors, Baker went out of her way to portray many characters of many different types, nationalities, backgrounds, etc.