It's probably because not all pages have that specific field and you just get a SQL error.
A few ideas but not sure what works in your scenario.
Explicitly select only the page type that has that custom field.
- Optionally you could combine multiple ones, having a sitemap for each type and just list them seperatly
Add the custom field to ALL page types that you select in the sitemap and just hide it for the editor on types that don't need it
- might be a bit overkill if it's specific and/or you have a lot of different types
Use your own transformation and Move this "Where" condition to a custom Macro that you call from the transformation. Here you can check the value based on the page type
- Make sure to keep an eye on performance for large sites though
completely customize it, there are a few nice Nuget packages that would easily allow you to create your own XML sitemap with full control.