You can use following sample expression in WhereCondition parameter of SelectNodes or SelectDocuments method to get documents from category specified by its ID:
"DocumentID IN (SELECT DocumentID FROM CMS_DocumentCategory WHERE CategoryID = " + categoryId.ToString() + ")"