You cannot simply say it is up to the Prime Minister because the mechanism for enacting Article 50 has not been explained.
"It is a matter for a member state’s “own constitutional requirements” as to how it decides to withdraw. The manner is not prescribed: so it can be a referendum, or a parliamentary vote, or some other means.
In the UK, it would seem that some form of parliamentary approval would be required — perhaps a motion or resolution rather than a statute. The position, however, is not clear and the UK government has so far been coy about being specific."
http://blogs.ft.com/david-allen-gre...vernment-legally-disregard-a-vote-for-brexit/
The part in bold goes along with what MP David Lammy said today.