I'm not saying race was irrelevant. Once race was created and enforced by law, race became relevant to everything.
I'm saying that when "white" (meaning rich) people agreed with Anthony Johnson that he could own John Casor, they were making a decision based on class, on the rights of property owners to own people as property. As Barbara Fields observed, the Atlantic Slave Trade was not created because white people hated black people. It was created for profit. As Eric Williams observed, race was created by slavery, not the other way around as some race reductionists argue.
We don't know if there were black masters with white slaves, but it would not have been illegal. A slave in the US was someone whose mother was a slave. If that slave was 15/16 white, that slave would be white in any state of the US before the Civil War and still be a slave.
The black people who owned slaves were not poor. Slaves were more expensive in the US than in many and maybe all other countries.
We agree that the true threat is the owning class, not the working class.