First off, Thanks Froggeye and Richard for your ideas... I sort of combined several of them and came up with a solution that did not require customizing a web part. For anyone that is trying to do something similar, here is the solution. TO summarize the problem:
I have an ecommerce site with a tree menu web part that was displaying some sub categories that were not empty but all products in there were not published so the sub category was showing in the menu even though when you clicked it, no products showed up.
So Froggeye's idea of adding a sub query to the where clause worked, I just had to join to the View_CMS_Tree_Joined instead of the View_CONTENT_MenuItem_Joined view because I was concerned about products in the sub category, not pages/menu items. This worked but then in the CMSDesk it was throwing an error because it was joining to a different view (View_CMS_Tree_Joined_Versions) then it was on the live site (View_CONTENT_MenuItem_Joined). So looking at Richard's idea of customizing the web part to check a querystring value, I realized instead, I could just use a query string macro, and the following where clause seems to have finally worked. Oh and I had already switched to a hierarchical viewer so technically I only tested this with that web part, I had abandoned the tree menu web part at Froggeye's suggestion but I think this solution would work there as well, you might not need the additional nodelevel part of the query with the tree menu web part, I think it handles that for you. My category structure never goes beyond NodeLevel 3 and I only wanted the sub categories to display if the user selected one of the sub categories under the same top level category or if they selected that level itself. (Just clarifying for someone that finds this thread in the future so they know what I was trying to do).
The final where clause:
(SELECT COUNT(1) FROM [dbo].[View_CMS_Tree_Joined] X WHERE X.Published = 1 AND X.NodeParentID = {% z = QueryString.GetValue("viewmode").ToInt(1); if (z==3 || z==6) {"View_CMS_Tree_Joined_Versions"} else {"View_CONTENT_MenuItem_Joined"} %})))