you can use s where in the parameter of your api call to the rest service, check out Kentico documentation
https://docs.kentico.com/k10/integrating-3rd-party-systems/kentico-rest-service/getting-data-using-rest
if you have the category id you can do a where of (documentid in (select documentid from cms_category where categoryid = 1))
if the query is too complex for the rest service, you can create your own api using a repeater with custom query (using macros to pass a category id or category name into the where condition), format the data into json using the transformation, and then attach the webpart to api converter that's found on the marketplace.