The 27 nation bloc banned the export of raw ivory last year, but the trade between member states continues.
Under current rules, antique ivory can still be exported and campaigners say dealers are using the exception to smuggle newer ivory to Asian markets.
Up to 30,000 African elephants are killed by poachers every year, with the practice particularly rife in the centre of the continent.
At the Giants Club summit - a forum of leaders devoted to tackling elephant poaching - the presidents of Botswana, Uganda and Gabon, signed a petition organised by the Avaaz activist group which demands that the EU shut down the domestic market, end all ivory exports, and support efforts to ban the global ivory trade.
They joined 29 other African nations, that were among 1,000,000 signatories from across the world.
“Europe has led the fight against illegal poaching, but these efforts are being undermined by the continued legal trade in ivory in EU countries,” the petition says. “In the context of China announcing its ban on ivory, Europe must now do all it can to prevent elephants being driven to extinction. “
Bert Wander, campaigns director at Avaaz said: “European officials told us they couldn’t ban ivory because not enough African leaders wanted them to. Now we’re going back to them with signatures from more than 30 countries where most of Africa’s elephants live to ask if they have any other excuses.
“The truth is there are none -- the rest of the world is turning its back on the ivory trade. Why not Europe?”
Max Graham, chief executive of Space for Giants said: “We’ve long argued that all ivory markets should be closed because any vagueness in can you buy it or can’t you buy it drives confusion, and criminal networks make billions of euros a year exploiting that confusion.
“It’s simple: the EU must follow the US and especially China, and say no ivory is for sale, full stop.”