Your analysis omits class. On the racial hierarchy of the old South, poor whites were at the bottom:

"In 1833, Fanny Kemble, an English actress visiting Georgia, noted in her journal: "The slaves themselves entertain the very highest contempt for white servants, whom they designate as 'poor white trash'"."

We see that today when people like the Comedian Godfrey mock homeless white people for failing their white privilege.