I doubt anyone's against high speed rail. It's just that the project itself requires massive spending, and has massive environmental and social impact - without actually dealing with the real infrastructure problems that stop a lot of people switching to public transport.
A lot of the journey time improvements (20 minutes off the 2 hour journey from London to Manchester for example) come from fewer stops, and the use of transport hubs, rather than higher track speeds. The trouble is, having arrived in Manchester, you're then entering an area with patchy metro (tram) and local train coverage, and multiple private uncoordinated bus companies.
The Manchester Metro is so successful it's packed solid for long periods of the day, and most of its park and rides are full before 8am. Yet, it still doesn't cover the whole of greater Manchester, and almost every journey involves getting a tram into town and another one out again. That 20 minutes for a relatively tiny number of passengers who do that London run occasionally is more or less irrelevant for people trying to get from one part of Manchester to another - every day, with potentially massive environmental impact if they switch from cars.
The same goes for Leeds, HS2 will cut its journey time to London but will reduce services on the traditional route through Doncaster - for a big chunk of Yorkshire the journey time will go up, and will start with people doing a trip on the slow overcrowded local service to Leeds. Leeds-Manchester-Liverpool is a more critical route from a commuting and freight perspective but that work hasn't been funded in any meaningful way. HS2 gives the illusion of government investment in northern infrastructure projects without the reality.
It does reduce journey times from Birmingham to London - potentially to a point where if you live in central Birmingham, you could work in London. It's arguable that it'll bring higher house (apartment block) costs to Birmingham and more coffee shops, but it won't bring the good jobs to Birmingham, and it might even do the opposite.
So yeah, nothing wrong with it in principle, but given it's sucked a lot of funding away from other projects and is likely to keep sucking - it's not greeted with a lot of enthusiasm even by people like me who want to see more, better, cheaper public transport options.