Elgin took the lead in the 27th minute with a little bit of help from the pitch.
Martin Johnstone danced his way into the area before falling under a challenge just inside the box, but when the ball rolled loose to Adam Nelson he hit a left-footed effort that the keeper seemed to have covered until a wicked bounce lifted it over his arm.
Berwick levelled nine minutes later. Kevin McLeish's deflected 20-yard free kick forced Kieron Renton into a fine diving save as he turned the ball away for a corner.
However, from Sean Palickza's delivery and Gordon Connelly's knock down, Gareth Hutchison was on hand to sweep the ball home from two yards out.
Berwick were beginning to take the upper hand and had a penalty shout waved away when Stuart Easton appeared to have handled before clearing for a corner and then Renton had to look lively to make saves from Palickza and Kevin Haynes.
Despite Renton's heroics Berwick were not to be denied and just a minute before the break they got their noses in front.
Right winger Palickza had been switched to the left and he wasted no time in springing the offside trap and advancing on goal through the inside left channel.
Renton got down well to save the initial shot with his knees but the ball came back out to Little who rattled the ball into the centre of the unattended net from 10 yards out to give the Gers a half-time lead.
Elgin started the second half on the front foot but were hit with a sucker punch in the 57th minute when Little grabbed his second of the game.
A free-kick 40 yards out was simply passed along the ground by Hutchison to Little who was clean in on goal and he made no mistake with a well placed low shot ending up in the left-hand corner of the net.