I'd agree with that. Although, it seemed to me that the White Walkers were a threat prior to the Pilot. It's just the threat wasn't really known to man at that point, only to the Wildlings. We don't know how fast the baby white walkers grow either, we know that they had that deal for a long time, so perhaps it was also a case that the Night King needed to wait for more White Walkers to become mature before starting to build forces and this was the first time he was able to get a steady flow of human babies.Maybe he was waiting on the dragons coming back into the world.
Ultimately we'll never know. Unless they write it in with the prequel.