The problem is we don’t know why the woman didn’t recognize your friend. You said they hadn’t met. Perhaps the woman had seen a picture and didn’t recognize your friend.

But if the two had never met, I see no reason to assume racism is the reason the woman didn’t recognize your friend. If all I had to go on was a voice, I certainly would’ve recognize someone I was talking to. At least, not unless the other person had an extremely unusual voice, either in terms of region or class, and I believe you said your friend has a conventional middle or upper-class accent.