I keep seeing a flood of errors in the event log related to the scheduler page on our various sites. It's all related to various bots hitting that page. I have a robots.txt page in place but doesn't seem to have any effect.
Message: The underlying connection was closed: An unexpected error occurred on a send.
Exception type: System.Net.WebException
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadData(Uri address)
at CMS.Scheduler.SchedulingHelper.RunSchedulerRequest(String url)
Message: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Not all the robots respect the robots.txt file. Sometimes it make sense to cut them off in the beginning of Request. Especially if you are that you dont about these robots and they dont respect your robot txt settings.
I am not quite sure what is your scheduler page? I would assume that this some sort of page with an online form and bot are trying to submit it? You might want to add captcha to this page.
It's just the standard Kentico scheduler.aspx page it uses for scheduled tasks. When hitting the URL it returns...
This page is used for execution of scheduled tasks.
The problem is something when these bots are hitting the page.
You mean Scheduling tasks page?
Are you saying that bots are hitting the page inside the admin panel?
Not the scheduled tasks page in the admin. The scheduler page under /CMSPages
The scheduler page could also be 'hit' by the bot as it's just time to run. You could look into running your schedulers as a service so neither users or bots have this extra load. Secondly you can more reliably say the tasks run.
May be you can play with scheduler web.config settings and protect the page?
Add CMSSchedulerUserName and CMSSchedulerPassword and see if that changes anything?
We ran into some wierdness when trying to use the scheduler service when we were first rolling out Kentico a couple years ago (v8) and so never installed it when we went to v9 Might take another look at it and see what happens. Just trying to do some end of year cleanup and these scheduler errors are creating a ton of garbage entries in our event log.
Please, sign in to be able to submit a new answer.