Using Azure Functions to Integrate Azure Search with Kentico Cloud
For the past few years, I’ve presented on Azure Search at several conferences and events. Microsoft’s Software-As-A-Service search solution is a great way to add powerful capabilities to a site with a few lines of code. Combine that functionality with Azure Functions and a Headless CMS and you get a very powerful search experience for your applications. In this article, I’ll show you how I integrated Azure Search into my Kentico Cloud site, using Azure Functions.
As part of my series on Building Applications using Microservices and Azure, I mentioned leveraging Azure Functions and Azure Search within my site. Well, here’s the details! In this article, I’ll show you how I leveraged a cloud-hosted function to run daily to keep my Azure Search index up to date. I then used the Azure Search API to search the index for real-time results, complete with highlighting, faceting, and other awesome capabilities. So let’s get to the code!
Warning: There is a lot of code in this post! I wanted to include as much as I could to show you how to do all parts of the integration.
Create the search service
The first step of the process was to create my Azure Search service. In the Azure portal under the New menu, I selected Web + Mobile / Azure Search. I entered my service URL, resource group, location, and pricing tier.
With the service in place, I was ready to create my function. I decided to put all my index creation logic within my function. You may want to create to your index directly within your service, if you know your data structure ahead of time.
Creating the function
To create the function, I selected Function App under the Compute category in the New menu. I entered my account and service information, as well as connected my storage account to the new function.
This function would contain all the logic needed to pull my data from Kentico Cloud and push it to my Azure Search index. I selected a Timer function, as I wanted to run the sync once a day. There are lots of different types of functions, so be sure to choose the type that fits your needs the best.
If you want to know more about Azure Functions, check out my previous blog posts here.
Adding Application Settings
Because Azure Functions are an App Service, they have all the functionality of a Web or Logic app, including Application Settings. Using these values, you can store sensitive information as part of your App Service configuration and reference them from within your code. For my purposes, I added the following settings in my App Service. I will reference these settings within my Function code.
- AzureSearchServiceName – This is name of search service.
- AzureSearchIndexName – This is the name of my search index.
- AzureSearchAPIKey – This is my Azure Search API Key.
- DeliverProjectID – This is my Kentico Cloud Project ID.
- BlogRSSURL – This the URL for my blog RSS feed.
- DeleteAndCreateIndex – This is a flag to determine if the index should be deleted / created.
- Debug – This is a debug flag.
Adding the Nuget packages
Because I would be using the function to integrate with both Kentico Cloud and Azure Search, I needed to include the required Nuget packages. Because Azure Functions still utilize a project.json file for loading dependencies, I created a new project.json file within my function. In the file, I added the Kentico Cloud and Azure Search Nuget packages.
Here is the full file contents, in case you’re interested.
{
"frameworks": {
"net46":{
"dependencies": {
"KenticoCloud.Delivery":"4.6.0",
"Microsoft.Azure.Search": "3.0.4",
"System.Text.RegularExpressions": "4.3.0",
"System.Xml.Linq": "3.5.21022.801",
}
}
}
}
Note that I included the System.Text.RegularExpressions and System.Xml.Linq libraries, as I would need these for formatting my data for the index.
Saving the project.json file causes the function to download the new packages, as needed. This also attempts to recompile the function, to ensure your code is still valid.
2017-04-17T18:32:10 No new trace in the past 3 min(s).
2017-04-17T18:32:12.851 Restoring packages.
2017-04-17T18:32:12.851 Starting NuGet restore
2017-04-17T18:32:14.603 Restoring packages for D:\home\site\wwwroot\SoltisWebSearchTriggerFunction\project.json...
2017-04-17T18:32:16.635 Committing restore...
2017-04-17T18:32:16.682 Lock file has not changed. Skipping lock file write. Path: D:\home\site\wwwroot\SoltisWebSearchTriggerFunction\project.lock.json
2017-04-17T18:32:16.682 D:\home\site\wwwroot\SoltisWebSearchTriggerFunction\project.json
2017-04-17T18:32:16.682 Restore completed in 2190ms.
2017-04-17T18:32:16.713
2017-04-17T18:32:16.713 NuGet Config files used:
2017-04-17T18:32:16.713 C:\DWASFiles\Sites\soltiswebfunctions\AppData\NuGet\NuGet.Config
2017-04-17T18:32:16.713
2017-04-17T18:32:16.713 Feeds used:
2017-04-17T18:32:16.713 https://api.nuget.org/v3/index.json
2017-04-17T18:32:16.760
2017-04-17T18:32:16.760
2017-04-17T18:32:16.815 Packages restored.
2017-04-17T18:32:17.119 Script for function 'SoltisWebSearchTriggerFunction' changed. Reloading.
2017-04-17T18:32:17.322 Compilation succeeded.
Creating the Search client
The next step was to create my function logic. Every Azure Function contains a single Run method, which is where my custom logic would go. In my method, I added an Azure Search Client, as well as calls to helper functions I created.
public static async void Run(TimerInfo myTimer, TraceWriter log)
{
try
{
// Get the search client
serviceClient = new SearchServiceClient(ConfigurationManager.AppSettings["AzureSearchServiceName"], new SearchCredentials(ConfigurationManager.AppSettings["AzureSearchAPIKey"]));
if (Convert.ToBoolean(ConfigurationManager.AppSettings["DeleteAndCreateIndex"]))
{
DeleteIndex(log);
CreateIndex(log);
}
await LoadIndex(log);
}
catch (Exception ex)
{
log.Info(ex.Message);
}
}
Note that I am utilizing the ConfigurationManager.AppSettings functionality to pull in my Application Settings.
Creating the index
The first step of the process was to create the index, if needed. Using the Azure Search SDK, I created the index and defined the different types of data I would be searching. Because I intended on combining speaking engagements and blog posts, I made a generic data structure to hold both types of data.
// Create the index definition
var definition = new Index()
{
Name = ConfigurationManager.AppSettings["AzureSearchIndexName"],
Fields = new[]
{
new Field ( "CodeName", DataType.String) { IsKey = true, IsSearchable = false, IsFilterable = false, IsSortable = false, IsFacetable = false, IsRetrievable = true},
new Field ( "Type", DataType.String) { IsKey = false, IsSearchable = false, IsFilterable = true, IsSortable = false, IsFacetable = true, IsRetrievable = true},
new Field ( "Name", DataType.String) { IsKey = false, IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = true, IsRetrievable = true},
new Field ( "PageAlias", DataType.String) { IsKey = false, IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = true, IsRetrievable = true},
new Field ( "Location", DataType.String) { IsKey = false, IsSearchable = true, IsFilterable = true, IsSortable = true, IsFacetable = true, IsRetrievable = true},
new Field ( "Date", DataType.DateTimeOffset) { IsKey = false, IsSearchable = false, IsFilterable = true, IsSortable = true, IsFacetable = true, IsRetrievable = true}
},
};
// Create the index
serviceClient.Indexes.Create(definition);
You can find out more about creating Azure Search indexes in my previous blog here.
Populating the index (Kentico Cloud)
To populate the index, I leveraged the Kentico Cloud API to retrieve my data. Note that I created a List<IndexAction> object to hold all my search index actions.
List<IndexAction> lstActions = new List<IndexAction>();
// Get the Kentico Cloud content
DeliveryClient client = new DeliveryClient(ConfigurationManager.AppSettings["SoltiswebProjectID"]);
// Get the content from Kentico Cloud
var response = await client.GetItemsAsync(
new EqualsFilter("system.type", "speakingengagement"),
new LimitParameter(5),
new OrderParameter("elements.date", SortOrder.Descending)
);
foreach (var item in response.Items)
{
var doc = new Document();
switch(item.System.Type)
{
case "speakingengagement":
if (Convert.ToBoolean(ConfigurationManager.AppSettings["Debug"]))
{
log.Info(item.System.Id + " | " + item.GetString("name") + " | " + item.GetString("pagealias") + " | " + item.GetString("eventlocation") + " | " + item.GetDateTime("date").ToString());
}
doc.Add("CodeName", item.System.Id);
doc.Add("Type", item.System.Type);
doc.Add("Name", item.GetString("name"));
doc.Add("PageAlias", item.GetString("pagealias"));
doc.Add("Location", item.GetString("eventlocation"));
doc.Add("Date", item.GetDateTime("date"));
break;
}
lstActions.Add(IndexAction.MergeOrUpload(doc));
Populating the index (Blog RSS feed)
In addition to my Kentico Cloud content, I also wanted to include my blogs as part of the index. To accomplish this, I used an HttpClient to retrieve my blog posts and add them to my List<IndexAction> collection.
// Get the list of blogs to add to the index
var feedUrl = ConfigurationManager.AppSettings["BlogRSSURL"];
using (var webclient = new HttpClient())
{
webclient.BaseAddress = new Uri(feedUrl);
var responseMessage = await webclient.GetAsync(feedUrl);
var responseString = await responseMessage.Content.ReadAsStringAsync();
//extract feed items
XDocument blogs = XDocument.Parse(responseString);
var blogsOut = from item in blogs.Descendants("item")
select new BlogPost
{
PubDate = Convert.ToDateTime(item.Element("pubDate").Value),
Title = item.Element("title").Value,
Link = item.Element("link").Value,
};
foreach(BlogPost post in blogsOut)
{
if (Convert.ToBoolean(ConfigurationManager.AppSettings["Debug"]))
{
log.Info(post.Title + " | " + post.Link + " | " + post.PubDate.ToString());
}
var doc = new Document();
doc.Add("CodeName", Convert.ToBase64String(Encoding.UTF8.GetBytes(post.Title)).Replace('/', '_'));
doc.Add("Type", "blog");
doc.Add("Name", post.Title);
doc.Add("PageAlias", post.Link);
doc.Add("Location", "");
doc.Add("Date", post.PubDate);
lstActions.Add(IndexAction.MergeOrUpload(doc));
}
}
Adding the collection to the index
With the collection of IndexActions populated, I was ready to add the content to the index. Using the Azure Search SDK, I passed the entire collection IndexActions to the search service.
indexClient = serviceClient.Indexes.GetClient(ConfigurationManager.AppSettings["AzureSearchIndexName"]);
indexClient.Documents.Index(new IndexBatch(lstActions));
With the function configured to create and populate the index, I executed it to confirm my records were added.
2017-04-17T18:49:07.292 Function started (Id=74d4bc9e-2f51-4757-8c31-0ef7b37dd7a1)
2017-04-17T18:49:07.557 Function completed (Success, Id=74d4bc9e-2f51-4757-8c31-0ef7b37dd7a1)
Searching the index
With the index created and populated, I was ready to update my site to search the index. In my project, I created new AzureSearchHelper class to hold the code to connect to Azure Search. Using the Azure Search SDK, I added the logic to access my index and return the results.
_searchClient = new SearchServiceClient(_projectoptions.AzureSearchService, new SearchCredentials(_projectoptions.AzureSearchAPIKey));
_indexClient = _searchClient.Indexes.GetClient(_projectoptions.AzureSearchIndexName);
//Build the SearchParameter object
SearchParameters sp = new SearchParameters();
sp.SearchMode = data.SearchMode;
sp.SearchFields = new[] { "Name", "Location" };
//Check if highlights should be returned
if (data.ShowHighlights)
{
sp.HighlightFields = new List<string> { "Name", "Location" };
sp.HighlightPreTag = "<span class='highlight'>";
sp.HighlightPostTag = "</span>";
}
sp.Facets = new List<string> { "Type" };
if(data.FilterBy != null)
{
sp.Filter = "Type eq '" + data.FilterBy + "'";
}
if (data.OrderBy != null)
{
if (data.OrderBy != "")
{
sp.OrderBy = data.OrderBy.Split(',');
}
}
return _indexClient.Documents.Search(data.SearchValue, sp);
I updated my Search controller to the helper function to return results. I added some UI elements to allow specifying the search options, as well.
SearchData searchdata = new SearchData();
StringBuilder sb = new StringBuilder();
// Get content page data
var response = await _client.GetItemAsync<ContentPage>("search");
searchdata.Content = response.Item;
// Set the page title
ViewData["Title"] = searchdata.Content.Title;
//Get the search form values
SearchFormData searchformdata = new SearchFormData();
searchformdata.SearchValue = model.SearchFormData.SearchValue;
searchformdata.ShowHighlights = model.SearchFormData.ShowHighlights;
searchformdata.ShowScores = model.SearchFormData.ShowScores;
searchformdata.OrderBy = model.SearchFormData.OrderBy;
DocumentSearchResult searchresult = AzureSearchHelper.Search(_projectoptions, searchformdata);
if (searchresult.Results != null)
{
searchdata.SearchResults = searchresult.Results;
if(searchresult.Facets != null)
{
searchdata.FacetResults = searchresult.Facets;
}
}
searchdata.SearchFormData = searchformdata;
return View(searchdata);
I used a lot of Azure Search-specific code in this class, so that I could get the exact results I wanted. Be sure to read up on Azure Search so you that you know all the options available when using the service. You can learn everything you ever wanted to know about Azure Search here.
Testing
To test the functionality, I accessed my Search page and entered a value. I then confirmed the results were accurate.
Note how my results include both speaking engagements and blogs. I also enabled highlighting to show where the records were matched with the value.
You can test out the search functionality for yourself on my site here.
Moving forward
By using an Azure Timer Function to sync my Kentico Cloud content to Azure Search, I eliminated a lot of custom programming. Because the function runs automatically every night, I can keep my search index up to date as content gets updated on my site. By leveraging the Kentico Cloud and Azure Search APIs, I easily integrated the entire solution into my application to achieve a great search experience for my users.
Hopefully, this blog showed you how easily you can combine these services to provide a unique search experience for your applications. Because search is so unique to every project, the possibilities are endless when it comes to building out your specific functionality. Good luck!
Get the Azure Function Code
Get the Azure Search Helper Code