You need to define the set of documents from which you want to choose the random document by setting the Path property to e.g. /Products/% and setting the Document type property e.g. to cms.product. The document will be displayed using the transformation specified in the Transformation name value - e.g. cms.product.preview.
Help URL: http://devnet.kentico.com/docs/webparts/index.html?randomDocument_overview.htm