No. Neither have to have a set numerical value or range. There is a difference between these words and, for example, majority/minority.
Let’s say we find out 100 million Americans eat glue-that may be a pastime you know well. “A huge number of Americans eat glue” or “a huge portion of the country eat glue” would make sense as sentences there. By your insistence (because it’s certainly not logic), as that is less than half of the US population we would have to say “a tiny portion of the country eat glue.”