The bishop's palace of Kronoberg was built during the 14th century. The fort gradually evolved into a castle made of wood and stone. In 1542 Nils Dacke and his rebels seized the castle, after the rebellion was squashed, the castle was seized by the king Gustav Vasa. Gustav Vasa subsequently extended the castle into a fortification with a large number of cannons. The main reason for the extension was to give the king a firm hold in Småland in an attempt to prevent further uprisings. The fortification eventually became part of the national defence system, and was conquered several times by the Danes. After the treaty of Roskilde, however, the fortification lost its importance and fell into disrepair.

Adjacent to the castle ruins is a café called Ryttmästargården and the home port of the steamboat S/S Thor.