It is possible to create a filter for custom table records by creating an alternative form named Filter. However, in some cases the filter will not be immediately visible due to the amount of records in the custom table. By default, there needs to be at least 25 records of data in the custom table in order for the filter to become visibile. It is also possible to reduce that minimum requirement by adding the following key to your Web.config:
<add key="CMSDefaultListingFilterLimit" value="10" />