In a racist system, some people in the same class have privileges--see Jim Crow. Where there's no significant privilege for any poor person, the issue is class. Remember that there are two times more poor white people than black. A 2-to-1 ratio is one hell of a lot of collateral damage.