Lucene Query work in Luke, but not Kentico

Michael Legacy asked on August 23, 2019 17:31

Hey all, bit of an odd one.

I have an index of jobs on a site I'm working on and have trouble getting results when dealing with multi-word field searches. An example of the raw lucene query is:

+(nurse)  +employmenttype:"full - time" +pf_site:external +fld_jp_position_category:nursing

If I paste this exact query (which i am getting straight from Visual Studio debugger) into Luke using the same exact indexes that the site is using I get this:

Image Text

As you can see, we get 56 results. However, using the same lucene query on Kentico, returns nothing. If I removed the field search "employmenttype", which is the only field search using a multi-word string, I get results again. So the issue is the multi-word terms for sure. This is true on any other field that has a multi-word term to be searched as well.

Here is the controller I built for the search API.

public class SearchAPIController : ApiController
{
    // GET api/<controller>
    [HttpGet, Route("webapi/careers/searchapi/")]
    public IHttpActionResult SearchApi(string q, int page = 1, string type = "", string category = "", string nursecategory = "", string shift = "", string site = "", string org = "")
    {
        SearchResult results = null;

        int skip = 0;
        int take = 25;
        if (page > 1)
        {
            skip = ((page - 1) * take);
        }

        q = HttpUtility.UrlDecode(q);

        // Gets the search index
        string indexes = "JobsIndex";

        string extracondition = null;

        if(!String.IsNullOrEmpty(type))
        {
            extracondition += " +employmentType:" + "\"" + type.ToLower() + "\"";
        }

        if (!String.IsNullOrEmpty(site))
        {
            extracondition += " +pf_site:" + site.ToLower();
        }

        if (!String.IsNullOrEmpty(category))
        {
            extracondition += " +FLD_JP_POSITION_CATEGORY:" + category.ToLower();
        }

        if (!String.IsNullOrEmpty(nursecategory))
        {
            extracondition += " +FLD_JP_NURSE_SPECIALTY_AREA:" + nursecategory.ToLower();
        }

        if (!String.IsNullOrEmpty(shift))
        {
            extracondition += " +jpm_shift:" + shift.ToLower();
        }

        if (!String.IsNullOrEmpty(org))
        {
            extracondition += $@" +hiringorganization:""{org.ToLower()}""";
        }

        var condition = new SearchCondition(extracondition, SearchModeEnum.AnyWordOrSynonyms, SearchOptionsEnum.FullSearch, null, true);
        string searchCondition = SearchSyntaxHelper.CombineSearchCondition(q, condition);

        if (indexes != null)
        {
            // Prepares the search parameters
            SearchParameters parameters = new SearchParameters()
            {
                SearchFor = searchCondition,
                SearchSort = "##SCORE##",
                Path = "/%",
                CurrentCulture = "EN-US",
                DefaultCulture = CultureHelper.EnglishCulture.IetfLanguageTag,
                CombineWithDefaultCulture = false,
                CheckPermissions = false,
                SearchInAttachments = false,
                User = (UserInfo)MembershipContext.AuthenticatedUser,
                SearchIndexes = indexes,
                StartingPosition = skip,
                DisplayResults = 25,
                NumberOfProcessedResults = 1000,
                NumberOfResults = 0,
                AttachmentWhere = String.Empty,
                AttachmentOrderBy = String.Empty,
                ClassNames = ""
            };

            results = SearchHelper.Search(parameters);

        }

        List<Job> jobsList = new List<Job>();

        foreach (var job in results.Items)
        {
            var viewModel = new Job();

            viewModel.title = job.Title;
            viewModel.hiringOrganization = job.ResultData.ItemArray[7].ToString();
            viewModel.site = job.ResultData.ItemArray[29].ToString();
            viewModel.employmentType = job.ResultData.ItemArray[12].ToString();
            viewModel.FLD_JP_DEPARTMENT = job.ResultData.ItemArray[26].ToString();
            viewModel.jobLocationName = job.ResultData.ItemArray[15].ToString();

            jobsList.Add(viewModel);
        }

        string final = new JavaScriptSerializer().Serialize(jobsList);

        return Json(jobsList);
    }
}

Any ideas would be appreciated. I've been fighting with this for a bit longer than I would like to be. I'm guessing it's something small that I just don't understand with Kentico's implementation of Lucene.

Recent Answers


Dmitry Bastron answered on August 23, 2019 17:46 (last edited on August 23, 2019 17:48)

Hi Michael,

In kentico Smart Search index details, what analyzer do you have selected? I'd suggest trying a few other variants from that dropdown (do not forget to rebuild the index after selecting a new one).

Check out analyzer types here.

0 votesVote for this answer Mark as a Correct answer

Michael Legacy answered on August 23, 2019 17:51

I will try switching it from Standard to Keyword like I'm using in Luke. I will report back. Thanks Dmitry!

0 votesVote for this answer Mark as a Correct answer

Michael Legacy answered on August 23, 2019 19:14

Dmitry, unfortunately switching the Analyzer type in Kentico does nothing.

I tried Simple (default), Standard, Keyword and White Space.

Standard and Keyword gave me no results for any searches that had fields related. White space and Simple do the same thing as stated above (no results when multi-term field search is included in the query). Any other possibilities?

As I said, I'm not sure that the INDEX is the issue, cause the results are coming up fine in Luke.

0 votesVote for this answer Mark as a Correct answer

Dragoljub Ilic answered on August 26, 2019 10:42

Hi Michael,

Problem with your multi-word term is that contains lucene special character hyphen (-), which could mess up your results in Kentico. When you try to search with different value for employmenttype, do you still get empty results?

Please reference on this answer, it looks like you will need to create custom analyzer or to escape special characters in search.

Best regards, Dragoljub

0 votesVote for this answer Mark as a Correct answer

Michael Legacy answered on August 26, 2019 15:46

Dragoljub,

Yes, unfortunately a phrase like "Hospital Name" still gives zero results. It's any value with a space in it. The hyphen could possibly be another issue I run into down the line, but for now, no values with spaces in them are returning. Even if I use the white-space analyzer (or the others I've tried).

It's just odd because the indexes work fine in Luke with the same exact queries.

0 votesVote for this answer Mark as a Correct answer

Brenden Kehren answered on August 26, 2019 18:57

In your Lucene screenshot, you have a "parsed" tab and a "rewritten" tab, click the "rewritten" tab and see what the result of your query was rewritten to. I wrote a post about a similar issue with date searches and Kentico finding different results than Lucene did.

0 votesVote for this answer Mark as a Correct answer

   Please, sign in to be able to submit a new answer.