In case anybody is reading I got an answer from kentico themselves:
This behavior is correct from SEO point of view. Previously we had the behavior you wanted but it was really bad for SEO to do a redirection so it was changed and just the 404 status code is returned and the URL is the same.
If you want to change this behavior, you can create custom event handler and in the request end event check the status code and if it is 404, do a redirection - but this can harm your SEO rankings.
https://docs.kentico.com/k11/custom-development/handling-global-events/reference-global-system-events#Reference-Globalsystemevents-RequestEvents