Lazio's Ciro Immobile celebrates after scoring during the Serie A match against Atalanta
Atalanta, who travel to Manchester City in the Champions League on Tuesday, had led 3-0 with 21 minutes to go at the Stadio Olimpico.
But Immobile scored from the spot on 69 minutes and less than 60 seconds later set up Joaquin Correa for the second, before capping the comeback two minutes into injury time with his ninth goal in eight league games this term.
Atalanta missed the chance to usurp Juventus at the top of the table ahead of the defending champions' game against Bologna later on Saturday. The Bergamo side remain third, two points behind Juve, with Lazio in sixth place.
"There was a loss of focus on our part, but it's hard to neutralise a team like Lazio at home," said Atalanta coach Gian Piero Gasperini, who was furious with both penalties awarded.
"With penalties, nobody can understand a thing nowadays.
"Immobile went for a walk, felt someone touch him slightly and dived. I've seen it again and it's obvious to me.
"Even more, on the second penalty, Immobile put his foot in front of De Roon. That's not a penalty either.
"We lost the Italian Cup final here (last season) on a similar incident."
Star Atalanta striker Duvan Zapata was sidelined with an injury suffered on international duty with fellow Colombian Luis Muriel taking his place.
The forward bagged a first-half double as Lazio struggled initially in a rematch of last season's Italian Cup final which the Romans won.
Muriel put the visitors ahead on 23 minutes when Robin Gosens played the Colombian in on goal.
He scored his second five minutes later from a free-kick, with Alejandro Gomez netting the third eight minutes before the break.
But the visitors lost momentum after the break.
Immobile got one back from the spot after being brought down by Jose Luis Palomino and then combined with Correa for their second.
Immobile earned a late penalty after a Marten de Roon foul, celebrating wildly after converting to grab a dramatic point before Lazio's Europa League trip to Celtic next week.
"I'm losing years of my life kicking penalties in the 90th minute," said 29-year-old Immobile, after scoring his third league goal from the spot this season.
"This result gives us confidence because coming back from three goals down gives an extra boost."
Polish striker Milik scored in either half as Napoli got back to winning ways at the Stadio San Paolo with a 2-0 victory, after a goalless draw at Torino before the international break, to stay fourth, one point behind Atalanta, before their Champions League trip to Salzburg.
The Serie A runners-up could thank goalkeeper Alex Meret early on for a triple save to deny Verona's Darko Lazovic, Matteo Pessina and Mattia Zaccagni after quarter of an hour.
The breakthrough came eight minutes before the break with Milik finishing off a Fabian Ruiz cross for his first goal of the season.
And the Polish international got the second when he got his foot to a Lorenzo Insigne free-kick on 67 minutes.
Verona drop to 13th place after their third loss of the season.