Thanks for the suggestion Zach. I was planning to do that as a last resort. I'd rather leave the web farm on automatic so we don't have to add them in manually when new servers are added or the names change.
Also, I monitored the servers and noticed that as long as they exist in the web farm app within Kentico, healthy or not, there will be tasks generated for them. When the servers are "not responding" or "transitioning", tasks are generated and the server is updated once it becomes healthy again.