I did a little googling, and it looks like Adam Smith was anti-slavery too. He tended to argue against it with economic points on the theory defenders of slavery didn't give a damn about morality.