Hi David,
Thank you for your message.
If you want to have different favicons for each site you will need to make some custom logic the same way as you would did on non azure site. There is unfortunately not yet easier way, but I believe the better support for favicon will be in future versions.
Please checkout this and although he is quite old the logic is the same.
Kind regards,
Richard Sustek