First of all, nobody said three challenges each. If you're worried about time, give them two each and have it so that it's one per half so you only get one in the second half. That's you'd usually want to waste time anyway.
If it's a baseless appeal, ie something that's clear cut, it'll be over in five seconds (two minutes if Moss has to get his arse to the monitor, but he'll be gone soon) and no significant time will have been wasted. If it's an outrageous call, the time is lost to protests anyway. How long did it take for Gylfi to take that free kick after Fred's "handball"? It would've taken the same time for the ref to sprint to the monitor, see that it hit his knee and restart play with a drop ball, plus we would've had the right decision.
If teams challenge throw-ins and they get it right every time and get to keep their challenges, well that's not a criticism of the system, is it? It would shine a spotlight on refereeing incompetence if they call them wrongly that many times, right?
Three challenges is usually what's suggested when this is brought up. Fewer than that would mean fewer delays but also far, far more chance of a team being screwed over by bad decisions. Which will then lead to a different variation of the "what's the point of VAR?" argument. It also means that decisions the ref gets "wrong" are even more contentious. Subjective calls will still exist, with a 50/50 referee interpretation now not only seeing a team on the wrong side of a decision but also potentially future decisions too. Presumably the "clear and obvious" stuff would be jettisoned under a challenge system, so marginal calls could then have an exaggerated impact on the course of a game.
Also, very few decisions would be over as quickly as you're suggesting. First a team would have to decide if they want to appeal, then communicate that to the referee. If the appealing team get to look at a replay before deciding whether to appeal, that's an extra delay. If they don't then there will be yet more bad calls and complaints. Plus few referees are going to take just five seconds to look at even a relatively straightforward decision, even more so when there's the extra pressure of a limited challenge being used. Also, there would inevitably be delays over arguments as to whether a team
can appeal. If a clear offence happens in a different phase of play to a goal being scored (which can itself be a subjective call) is the team refused a challenge? Or is the challenge just rejected, to the now increased ire of the team who have just conceded a goal after a clear offence and become more vulnerable to further bad calls? Whatever way you set up the system, there will be protests and delays accompanying each challenge.
Plus major incidents (goals, red cards, penalties, etc.) are already being checked as is. So you'd effectively be increasing the risk of major incidents being missed (because there are no challenges left) for the sake of being able to check minor incidents instead, which hardly seems a sound trade-off. Especially when there's no real evidence that referees would suddenly be capable of making better decisions for fear of being embarrassed. The challenge system would also have inevitably introduced greater variance into the decision making process, which in turn would lead to yet more mistrust and accusations of bias.
Basically, I don't see enough of an upside.