You don't need to rebuild(See documentation) The system automatically updates search indexes to reflect all changes made to the indexed content. Over time, these updates can make indexes less efficient, particularly in the case of large indexes. To restore optimal search performance for an index, defragment the index by clicking Optimize. You can enable the Optimize local search indexes scheduled task to have the system automatically optimize all smart search indexes once per week.
P.S. I think this task is not enabled by default.