Kentico supports all 133 ISO world languages. What you are describing can happen only if a culture of the page is different than current culture or default culture and the page doesn't have its own culture-specific url.
It's not very SEO friendly if you have it set up like this (non-culture-specific url would return different than preferred/default url which is not good and could potentially lead to non-deterministic behavior -> but you can implement some custom logic for this using URLRewritingEvents), however, also in this case you can simply append a lang query string (?lang=jp-JP) to links that lead to those pages to enforce the culture.
It's always about generating a correct url of the given page, so please consider one of the following options that would solve your issue in a more proper way:
1) Turning language prefixes on: in Settings -> URLs and SEO you can turn the Use language prefix for URLs setting on which adds a language prefix to all of your pages to make their URLs culture-specific
2) Generating URLs from page names -> in the case your page names are translated to respective language as well you can tick the Settings -> URLs and SEO -> Use name path for URL path setting and then the URL will be also culture-specific, because it would be generated from its name
3) If you want to change this behavior just for those exatra pages or just to some subset of pages, you can switch to Properties -> URLs section of such a page and fill out the Path or pattern property witch some custom path to that page that also makes the url culture specific
In all described scenarios you get the jp-JP page instead of 404.