There are a few things you can do:
Create a custom membership provider which would involve a lot of work in my opinion
Another option would be to create a custom webpart that performs your authentication as needed. When the user authenticates log in a dummy user that you created in Kentico. So this approach would be ok if you didn't need personalization information or complex access rules. For instance if you had only a few security roles (member and non-member). It would be pretty easy to say all users that are in a "members" role in your custom db belong to "CustomMember" role in Kentico (which really consists of one user and anytime one of those "members" from custom db log in you also log in the "CustomMember" user. I've seen (and done) it this way before but the security was very very simple.
Use out of box Kentico membership provider and create a custom event handler to perform the authentication (in both areas) and create that same user in Kentico with similar properties and roles.
Yes the same user resides in both places but your authentication method could perform a one way sync (custom db -> Kentico) each time they authenticate. If you have sections which need to perform updates to the members information in the custom db, then handle that in code and create methods to update that other database whenever the user information is updated using an additional custom event handler.
The last way is what I'd recommend as you can handle everything you need much easier and still use all th eout of the box functionality.