Forum layouts are stored in <web project>\CMSModules\Forums\Controls\Layouts. The folder contains three sub-folders by default.
The Flat and Tree folders contain controls for the identically named layouts. The Custom folder can be used for defining custom layouts.
To create a custom layout, you need to create a sub-folder inside the Custom folder. The name of this sub-folder will be used as the name of the layout. The sub-folder has to contain controls with the same names as included in the Flat or Tree folders. You can see a screenshot of these files below. When writing the controls' code, make sure that they inherit from ForumViewer.