Chelsea consolidated fourth place in the Premier League with a 2-1 win at Brighton & Hove Albion that became harder work than had seemed likely in the first half.
Having won 4-0 at the Amex Stadium last season, they were on course for a repeat after taking a 2-0 lead at the interval.
Eden Hazard set up the first goal for Pedro after 17 minutes and scored the second himself before halftime following a defensive error.
Brighton came back into contention through Solly March's goal midway through the second half and Lewis Dunk went close to an equaliser.
Austin settled a thrilling Premier League clash at St Mary's when he headed home with just five minutes left after Leno misjudged Shane Long's cross.
Southampton had twice taken the lead through Danny Ings, only for Henrikh Mkhitaryan's brace to draw Arsenal level on each occasion.
Leno's mistake and Austin's predatory finish handed Arsenal their first defeat in 15 league games and brought a stunning halt to their longest unbeaten run in all competitions since 2007.
Unai Emery's fifth placed team hadn't lost since a 3-2 defeat at Chelsea in August and the north Londoners' first setback since the summer leaves them two points behind the fourth placed Blues, who won at Brighton on Sunday.
While it was a bitter afternoon for Arsenal, Southampton's first win in 15 matches in all competitions was cause for much celebration.
Southampton manager Ralph Hasenhuttl, taking charge of his first home game, danced a jubilant jig on the pitch at full-time after securing the maiden victory of his reign.
Hasenhuttl watched Southampton lose 1-0 at Cardiff in his first match and, in a PR stunt this week, he sent free drinks vouchers to Saints season ticket holders.
The Austrian, who left his role as Leipzig boss in May, was the toast of St Mary's after Southampton won at home in the league for the first time since April to climb out of the relegation zone on goal difference above Burnley.
Laurent Koscielny was making his first Premier League start since April after recovering from an Achilles injury, but Arsenal's French centre-back looked rusty when Southampton took a surprise lead in the 20th minute.
Matt Targett whipped over a superb cross and Koscielny was caught by Ings' perfectly-timed run, completely missing his attempted clearance as the on-loan Liverpool striker powered his header past Leno.
Ings' sixth goal of the season reflected well on Hasenhuttl, who had recalled him in place of Austin on his return from a hamstring injury.
It took Arsenal just eight minutes to equalise with a flowing move that saw Alex Iwobi pick out Nacho Monreal's run before the Spaniard chipped a cross for the unmarked Mkhitaryan to head home.
Ings' movement was causing Arsenal problems and he struck again to restore Southampton's lead in the 44th minute.
Nathan Redmond's cross unhinged Emery's creaky back-three and this time Ings got in between Koscielny and Stephan Lichtsteiner to loop a fine header over the stranded Leno.
Ings hadn't scored with a header in the league for three years, but now he had two in just one half.
For the 17th time this season, Arsenal had failed to secure half-time lead in the league and Emery responded by sending on Alexandre Lacazette.
The Gunners have been far more dangerous after the interval this season and once again they hit back to equalise in the 53rd minute.
Allowed too much space on the edge of the area, Mkhitaryan collected Lacazette's pass and drilled a low strike that took a crucial deflection off Jannik Vestergaard as it flashed past the wrong-footed McCarthy.
Southampton thought they had taken the lead in the closing stages when Long stabbed in after Maya Yoshida's header was clawed out by Leno, but the goal was ruled out for offside.
Hasenhuttl cursed in frustration on the touchline, but the Austrian was celebrating soon after as Southampton punished Leno's blunder to snatch the points in the 85th minute.
Long's cross floated across the penalty area, tempting Leno off his line, but the German missed with his attempt to punch clear and Austin was on hand to head into the empty net.
The Pope is said to be recovering from a lung inflammation
The all-rounder hammered an unbeaten 104 off 48 balls to secure a five-wicket victory for Australia
Motorists have been advised to use alternative roads such as Jumeirah Road, Sheikh Mohammed bin Zayed Road, and Emirates Road