Built at a cost of £1250, on land purchased from William Lloyd for £65, this sandstone building replaced an earlier public reading room of timber construction which stood on the corner of Main (previously called Church Street) and Long Streets.
Built in 1858, during the tenure of the first magistrate Mr James Fichat, Mayor C.W. Thesen opened the present library building, with its west wing only. Built by Cooper & Rogers of Oudtshoorn with roofing erected by local carpenter, Andrew Hepburn, of The Knysna Forest Company, on 28 November 1893.
The east wing was added in 1936, using matching stone, most of which had been quarried in 1893 and stored for the purpose.
More recent extensions were completed in 1987, on the north side of the building, and in 2014, creating a second floor..