The Hoops found the winning goal in the eleventh hour and avoided a goalless stalemate.
After a successful buildup of play, Odsonne Edouard netted the ball benefiting from the pass of Callum McGregor in injury time.
Following the goal, the Hoops boss ran across the field, overwhelmed with joy.
The win was their second closest one in space of less than a month.
Lennon was appointed as the new makeshift manager following the departure of Brendan Rodgers to Leicester City.
The former Celtic captain has led his side to three big wins since taking the charge at Park Head.
Lennon has previously managed the Hoops and holds a success rate around 70 per cent, higher than Rodgers.
The Hoops are currently topping the Premiership table ten points clear of Old Firm rivals Rangers. They are eyeing for the eighth straight league title in a row.
Watch more Celtic FC video highlights, click here- Latest Celtic FC video highlights
The Hoops are set to face Aberdeen in the Scottish Cup semi-final.