After a goalless first half, Port took the lead 13 minutes into the second half when they split the Bankies defence to score.
Stevie Dalls fired the home side level with a fantastic finish with around 20 minutes left.
The winner came from the penalty spot. Nikki Cairney was pushed and Gary Lynn fired home with not long to go avoiding a replay next weekend.
Read Tim Lovejoy's weekly blog at Football.co.uk and join the debate.