I agree all racist terms are classist, but not all classist terms are racist. I have yet to find out whether poor whites were ever hired to pick cotton during the age of slavery, but we know they did afterward. For example:

"my own grandmother was only barely literate; she was forced to drop out of school by the seventh grade to pick cotton."

https://www.aaihs.org/masterless-men-a-new-book-on-poor-whites-and-slavery/