Dusan Tadic's opener, a placed effort following Ryan Bertrand's marauding run, and Polish defender Jan Bednarek's left-footed shot looked to be lifting Saints out of the bottom three.
It was Conte's changes that turned the game in Chelsea's favour, with Giroud grabbing his first and second for the club, but the manager was more concerned with the difference in his team as a whole for Chelsea's first league comeback from two goals down since 2002.
Conte was happy that his players showed "great heart" to comeback, but gave a frosty response about Rudiger's omission from the matchday squad.
Southampton had got off to a flawless start thanks to a former Chelsea player as Ryan Bertrand burst past Cesar Azpilicueta before squaring for Tadic to apply a simple finish.
Who will come out on top in the battle of the managers Mark Hughes or Antonio Conte?
Speaking to Sky Sports after the game, Giroud said: "I said when I came on to Eden Hazard we need to fight to the end keep believing".
"But I saw a great reaction".
Chelsea's frustration overspilled as tempers flared late in the first period when Marcos Alonso was lucky to escape unpunished for raking his studs down Shane Long's calf. I think between the referee, linesman or fourth official that one of them should have seen it.
The home side doubled their advantage on the hour when debutant Bednarek side-footed a wonderful Ward-Prowse free kick into the far corner of the net.
And Giroud showed what he is capable of with a courageous near post header from Alonso's pinpoint cross to get Chelsea back in the game and spark an incredible fightback 20 minutes from time.