Having thought about it, I think Starmer is right to be cautious on Brexit and I say that as a remainer.
He can't say he'd rejoin, because (a) it's not up to us, the EU has no reason to trust us or want us back (b) it's not the right time to open that can of worms, can you imagine the arguments all over again (c) Britain needs to learn to deal with the consequences of Brexit s so if we do return to Europe, we will properly value it and be prepared to live with the tradeoffs it would involve (d) there is no such thing as a 'Swiss style deal' anyway, so don't engage with it.
You have to accept the Brexit settlement at some level, while seeking to improve the deal we have where we can, if we can, but we have a long way to go before we can even get into that discussion with the EU. There's a lot of groundwork to be done. We need to EU to trust us again, for starters. All he can realistically say is he'd seek better relations with the EU at this point and look for some easy wins.