Because we aren't losing our biggest trading partner, we will continue to trade with them, like any other country outside the EU. We had the same alarmist scare mongering when Britain decided against signing the schengen agreement, the same when Britain opted out of the EU charter of fundamental rights. Business will not be hurriedly leaving our borders because we voted Leave, there are ample reasons why multinationals base their headquarters in London, it's why London is regularly voted as the best city in the world for business, culture and innovation, with the vast majority of its closest competitors coming from outside of the EU. There will be a period of resettlement, but the EU aren't going to cut off their nose just to spite their face just to make an example out of Britain. If that is their attitude then why would we want to be associated with an organisation that holds personal pride over the interests of its members?
It's not racist to want tighter border controls, in fact, lumping all the Leavers together as racists is no different than the generalisation you're claiming they hold about immigrants. For many, like me, this was a secondary issue when deciding my vote.