What's tricky is slavery had nothing to do with race, but race was the product of slavery, and with race comes racism.

While the Poltifact article is good, it overstates the case that black slaveowners were benign. Rich black slaveowners like Wiliam Ellison were not known for treating their slaves better than other owners did.