Why not?
How old were the likes of Xavi, Modric, Pirlo before people were discussing them among the best midfielders in the world? We hadn't even signed Michael Carrick yet, our best midfielder of the last decade, when he was 24.
It's a well known fact that midfielders are among the last to realize their full potential due to the discipline and maturity it requires to play there. There are only a handful of examples out there of top, top midfielders who were already brilliant aged 24.
EDIT: To clarify, I'm not saying Pogba will become anywhere as good as those three, just using them as examples
Firstly Pogba is a completely different mould of a midfielder to who you've named. But to entertain it:
Modric was 24 in 2010, 2 seasons before he went to Real Madrid. He was well established with Spurs and attracting a lot of interest from top clubs and far more consistent to what Pogba has shown for us in England so far.
Pirlo at 24 (2003) just came off the back of leading the entire league in 4 categories: successful balls, successful passes, passes played and most passes per game. He was widely regarded as one of the best midfielders in the world and firmly Ancelottis pet favourite. He also registered 9 goals from a deep lying playmaker position (career best, not bad for a guy playing so deep).
Xavi was 24 in 2004/2005 season where he was named vice captain and played a pivotal role in helping Barcelona win La Liga and the Supercup before being named the leagues player of the season the following year.
So yeah, all of your examples were pretty phenomenal by 24, let alone have buffers for wildly inconsistent performances. Pogba is a long way off that.
Infact from a consistency angle and comparing to present day players who aren't even legends, he's not as good as Thiago when he was 24, or Vidal when he was 24, or Kevin De Bruyne and so on and so on.