Andy Clarke gave The Posh the lead on 22 minutes but his effort was quickly cancelled out by Phil Jevons.
Steve Thompson and Adam Newton gave them a two-goal cushion, but Marcel Cas's goal five minutes from time set up a tense finish.
In the end though, it is Barry Fry's men who will wake up with their name in the hat for Sunday's draw.
Telford caused the upset of the round as they knocked out Second Division side Brentford with a superb hat-trick from Christian Moore.
Moore put Telford on their way with his first on 26 minutes and went on to bag a further two goals in the second half.
Ibrahima Sonko was dismissed for Wally Downes's men as Telford claimed a notable cup scalp.
Yeovil look set for another run in the competition where they have made their name for giant killings after a 5-1 win over Barnet.
Colin Pluck put Yeovil in front with a goal after nine minutes but his effort was quickly ruled out by Peter Beadle just 60 seconds later.
Gavin Williams restored the home side's advantage with a penalty after 18 minutes and made it 3-1 with a second strike nine minutes later. Nick Crittenden and Jake Edwards completed the rout.
Sheffield Wednesday left it late to earn a replay against Scunthorpe United.
Steve Torpey netted on 12 and 77 minutes which looked to be enough to send the Third Division outfit through.
Guylain Ndumbu-Nsungu clawed one back for The Owls five minutes from time before Grant Holt got the all-important equaliser at the death.
Southend comfortably saw off Lincoln 3-0 in this all-Division Three game at Roots Hall.
Tesfaye Bramble took The Shrimpers into a first half lead with a goal six minutes before half time and Mark Gower made it two with a goal on the hour mark.
Lincoln rarely threatened the Essex-based side and Jimmy Corbett completed the win with a goal a minute before full time.
Swansea City overcame Stevenage Borough 2-1 at Vetch Field.
Kevin Nugent put The Swans ahead on 23 minutes and Lee Tundle added a second on 51 minutes. Anthony Elding scored the consolation for Stevenage.
Steve Robinson's penalty after 20 minutes and Lee Mansell's effort gave Luton the win against Rochdale at Spotalnd as Mike Newell's men advanced.
Dean Bennet's brace and Steve Burton goals were enough to claim a 3-0 victory for Kidderminster Harriers over Woking.
FA Cup results:
Peterborough 3-2 Grimsby
Rochdale 0-2 Luton
Scunthorpe 2-2 Sheff Wed
Southend 3-0 Lincoln
Swansea 2-1 Stevenage
Telford 3-0 Brentford
Woking 0-3 Kidderminster
Yeovil 5-1 Barnet
