The first time accessing the site after you publish it, is most likely because the site needs to be compiled. The subsequent times most likely is because the site is "shut down" in IIS on the server due to no traffic.
By default the site will be rebuilt if specific files or settings change. It is best to not make any of the changes unless you need to otherwise you'll be back were you were initially. Best thing to do is find a way to keep your site running if you don't have enough traffic and ensure the caching settings are as expected within Kentico.