Geert Wilders said the controversial contest would not go ahead following death threats and concerns other people could be put at risk.
“To avoid the risk of victims of Islamic violence, I have decided not to let the cartoon contest go ahead,” the far-right opposition politician said in a written statement.
Mr Wilders, who lived for years under round-the-clock protection because of death threats sparked by his fierce anti-Islam rhetoric, said he did not want others to be endangered by the contest he had planned for November.
The contest sparked angry protests in Pakistan, where thousands of hardline Islamists marched towards the capital Islamabad this week.
Around 10,000 supporters of the Tehreek-e-Labbaik group, which helped Imran Khanbecome prime minister following last month’s national elections, set out on the march on Wednesday, calling on Mr Khan to cut diplomatic ties with the Netherlands.
The competition also allegedly attracted a death threat from a 26-year-old man, reportedly a Pakistani, who was arrested in The Hague on Tuesday.
It was cancelled shortly after the Taliban urged Afghan soldiers to attack Dutch troops serving in the Nato-led Resolute Support mission in retaliation.
Members of the Afghan security forces “if they truly believe themselves to be Muslims or have any covenant towards Islam should turn their weapons on Dutch troops” or help Taliban fighters attack them, the Taliban’s main spokesman said in a statement.
Physical depictions of God or the Prophet Muhammad are forbidden in Islam and are considered blasphemous by Muslims. Previous cartoons depicted him have provoked violence.
In 2005, a Danish newspaper published cartoons of the prophet which sparked a wave of protests across the world.
Ten years later, Islamist gunmen killed 12 people in an attack on the offices of French satirical magazine Charlie Hebdo, which had published similar caricatures.
“It’s not just about me,” Mr Wilders said in the statement. He said strong opponents of the event “see not only me, but the entire Netherlands as a target”.
The contest would have been held at the tightly guarded offices of his Party for Freedom in the Dutch parliament building.
The government had been at pains to distance itself from the contest.
The country’s prime minister, Mark Rutte, questioned Mr Wilders’ motive for organising the contest and said he did not support the planned contest, but he would defend Mr Wilders right to hold it.
“His aim is not to have a debate about Islam. His aim is to be provocative,” Mr Rutte said.
However, the prime minister added that people in the Netherlands have far-reaching freedom of speech rights and the government did not intend to seek the contest’s cancellation.
Earlier on Thursday, a Dutch judge extended the detention of the man who allegedly threatened to attack Mr Wilders by two weeks.
Prosecutors said in a statement that an investigating judge ordered the suspect held while he is investigated on charges of making a terrorist threat, making preparations for a terrorist murder and incitement.