Father Jacques Lacroix, 89, said: “I am finishing my ministry now...there is an end to everything."
Troubling video footage emerged earlier this week of him hitting a baby across the face with an open palm as he attempted to conduct his baptism.
In a 40-second clip posted online, which has since went viral, the clergyman lashes out during the ceremony because the boy would not stop crying.
In an interview with French Info radio on Friday, Mr Lacroix denied that he had been too severe with the child.
“It was somewhere between a caress and a slap, I hoped to calm him down, I didn’t know what to do,” he said.
“The child was screaming a lot and I had to turn his head to pour water over it. I told him to ‘calm down, calm down’ but he was not calming down. I tried to hold him close. I just wanted him to calm down."
He added: “I apologised for my clumsiness to the family. I am finishing my ministry now, it was my last baptism, there is an end to everything."
Jean-Yves Nahmias, the bishop of Meaux, said he had "taken measures so that the priest is suspended from all baptism and marriage celebrations."
The bishop said Father Jacques would only be allowed to celebrate mass in the future if he had secured prior permission, adding that the priest had been guilty of a "loss of coolness that can be explained but not excused".
The bishop added: "This gesture is all the more regrettable because the baptism is meant to be a happy moment, but tiredness and great age clearly played a part."
The baby's parents accepted the priest’s apology, he claimed.
The exact location of the church, as well as the date of the incident, is unclear; although the video surfaced on the internet on Wednesday.
“I'm going to put water on the forehead, and after I'll embrace the little man, because he will have become a little Christian,” the elderly priest says in the footage.
However, he swiftly became annoyed that the boy will not stop crying and grabbed him by the face, and said sternly: “Stop crying, calm down. Calm down.”
He then unleashes a hard slap against the baby’s cheek, provoking a chorus of gasps among those sat in the church.
A man thought to be the child’s father is seen attempting to take him away – only for the priest to grip the boy by his neck.
Those gathered in the church can be heard to tell the cleric: “Don’t hit the child,” and: “Sir, we do not hit him,” but he is seemingly unapologetic.
“You have to calm down, you have to calm down. There you go,” the priest tells the child as the parents of the infant appear increasingly agitated.
The mother pleads with the priest to let go, but he continues to grip the boy by the head.
The father is then able to wrestle the baby away and into his arms, as another woman attending the service also intervenes in a bid to ease tensions when the video cuts out.
Additional reporting by agencies