In a racist system like Jim Crow or Apartheid, people of different races but the same class are treated differently.
Do you think Jewish and Asian Americans are more privileged than white Christian Americans because they're disproportionately wealthy?
And do you have a source for "black people themselves were explicitly targeted for exclusion by the federal government"? One of the things that surprised me when reading about public housing during the depression was that even though housing projects were segregated in the south, the contractors used the same plans and materials for white and black housing. It was one of the few cases where things actually were separate but equal. Which does not excuse it, of course.