Wikipedia is good on this: https://en.wikipedia.org/wiki/White_trash

In particular: "The first use of "white trash" in print to describe this population occurred in 1821.[34] It came into common use in the 1830s as a pejorative used by house slaves against poor whites. In 1833, Fanny Kemble, an English actress visiting Georgia, noted in her journal: "The slaves themselves entertain the very highest contempt for white servants, whom they designate as 'poor white trash'".[35][36]"