What are your settings for the page types being indexed? Are you indexing the content, HTML or both? If you're indexing HTML or BOTH, then I'd highly recommend WHY. When you index the HTML or BOTH, the search index has to physically render the page, read the content, and index the content. Per page that can take anywhere from 1 to 10 seconds or more. We had this issue on a site that had a couple thousand product pages that were set to index the HTML output of the page, the rebuild process took almost 8 hours. When we cut it back to content only, it took minutes.