They prospered under the Old South's political structure. They provided money to it. They had infinitely more privileges than the people who were dismissed as "white trash". Saying rich black slave owners were of little consequence seems, well, unkind to the accomplishments of prosperous black people in the antebellum South.