Tottenham's victory at Burnley saw Chelsea's advantage cut to seven points on a day which had started so promisingly for Antonio Conte's side.
The hosts needed just five minutes to take the lead in the London derby clash when Cesc Fabregas tapped in from close range Eden Hazard's cross from the left, but Palace replied in superb fashion with two goals in three minutes.
Fresh off the back of a superb solo goal for the Ivory Coast this week, Wilfried Zaha restored parity for the visitors before Benteke's fine goal in the 11th minute helped the Eagles turn the game around in a breathtaking way.
Wayne Hennessey then pulled off a string of fine saves to help ensure the win, which takes Palace six points clear of the drop zone while Chelsea's lead over second-placed Tottenham was cut to seven points following Spurs' 2-0 victory at Burnley.
Antonio Conte was handed a major boost ahead of the match at Stamford Bridge after star man Eden Hazard was cleared to play against Palace.
The winger, who has scored 11 league goals this season, missed Chelsea's last game - a 2-1 win away to Stoke City - with a calf injury, which also resulted in Hazard being ruled out of Belgium's international fixtures against Greece and Russia.
But after recovering fully from the problem, the 26-year-old started in his usual position on the left flank while Fabregas started on the right as Pedro took on Victor Moses' wing-back role with the Nigerian ruled out through injury.
And the move to start Fabregas - and start him in an unfamiliar position on the flank - paid off when the Spaniard sent a long ball into the path of Hazard in the fifth minute.
The winger bamboozled Joel Ward to cross the ball back to Fabregas, who coolly tapped in from close range to hand the hosts an early lead.
Despite the ease of the opener - the first goal Palace had conceded in four games - the visitors remained positive and they were duly rewarded just four minutes later when Zaha netted in stunning fashion.
Zaha - who scored a fabulous strike for the Ivory Coast against Russia in midweek - received the ball with his back to the goal before finding a pocket of space in which to turn and drive in a low effort that Thibaut Courtois could not keep out.
But if Palace fans at the Bridge thought that would be the end of their team's impressive attacking play, they were wrong.
Just two minutes later, Benteke combined with Zaha to bulldoze his way through the Chelsea defence before the Ivorian's pass back to him gave him the chance to lift the ball over Courtois and into the back of the net.
The goal turned the game on its head, although Chelsea should have been handed the opportunity to restore parity shortly afterwards when Pedro's goal-bound shot down the other end of the pitch hit Andros Townsend's arm.
Despite frantic calls from the Chelsea players, no penalty was given and it led to the home side upping their tempo and piling on the pressure on Palace's defence.
Hazard and Fabregas combined in the 22nd minute to tee up Costa, but Hennessey denied the striker before the Palace goalkeeper made a number of stunning saves to send Allardyce's side into the break still leading 2-1.
Allardyce was forced to make a change for the second period after replacing the injured James Tomkins with Scott Dann. But the defender lasted just 14 minutes after being caught by Costa as he blocked the Chelsea striker's shot which resulted in him having to be stretchered off.
With the Palace boss being forced to use up all of his three substitutes so early in the game, Conte introduced Willian for Nemanja Matic but Chelsea found themselves having to defend shortly afterwards.
Zaha found himself in space and able to get a shot at goal, but Courtois' last-ditch stop with his leg kept it out.
Chelsea appeared more and more frustrated as the second half progressed, and they were let off the hook when David Luiz's foul handed the visitors a free kick which ended in Benteke firing just high of the bar.
Costa could have scored down the other end but he headed Hazard's cross over before Mamadou Sakho made two superb late clearances to help Palace secure a famous 2-1 win over their London rivals Chelsea and inflict the fourth league defeat of the season on the Blues.