And in many cases, they were not. People of all races owned slaves the way farmers own tractors. The richest black slave owner, William Ellison, was known for treating his slaves badly.