You should add the gif file you like to use as an icon of custom document into the directory below:
/App_Themes/Default/Images/CMSDesk/Icons/
Then rename the file according the code name of the document type. Follow the example below how to rename the file:
namespace_documentType.gif
So the name consists of namespace and document type of that document separated by underscore.
UPDATE for v5.5: the location of the icon files was changed to
\App_Themes\Default\Images\DocumentTypeIcons</span>