From developers to developers - K#

Does K# sound familiar to you? ... C#? Kentico? ... You are absolutely right. It pretty much explains itself very well. In Kentico CMS version 6, we have introduced a whole new level of writing macros. We have our own parser for C# syntax built into our solution, which allows you to achieve incredible results by just writing text expressions anywhere you used to use our "old" macros.

Imagine macros like:

{% if (CurrentUser.IsMale) { return "Male" } else { return "Female" } %}

... your dream has just become true. Yes, even stuff like that is now possible with Kentico macros.

Backward compatibility

When it comes to something new, the first question on the table is always "How will it change what we have now?". The good news is that the new macro engine, even that it is built from the scratch, is designed to fully support all expressions that were available before, so whatever you had before, it will still work.

Technically, even expression like {%CurrentUser.UserName %} is still a C# (or K#) syntax, so we process it like that. Also, we have kept the support for the "old" parameters, so if you write {%CurrentUser.UserName|(tolower) %}, the old parameters take priority and get processed the old way.

The reason for the backward compatibility is that we didn't want to introduce a new character to differentiate between new macros and old macros to keep things simple. Whatever you need to access is available through {% ... %} macros, including querystring or cookie values, but let's talk about that later.

If you don't need anything more than just output single values, you can stop right here, if you do, continue and learn.

The origin of K#

I don't wan't to start with just writing expressions because this post is a pure developer stuff. I want to start with you being able to see what is behind all this.

Most of the coolest features we have, now including K#, aren't officially planned. They rise from a continuous trial-error process or some very basic need. In fact, the whole new macro engine started when I was once bored in the evening, there was nothing going on on the TV, and I was thinking how to make our CMS even more flexible. When it comes to flexibility, whatever is compiled isn't very good, especially on cloud solutions, because each compilation means a re-deployment of the whole project. Also, if you have a UI where you actually don't write code, but text only, how do you tell the system what to do? We already had macros, but I felt like they are getting too far from being user friendly if we throw any more parameters to them and try to explain you how to make logical operations on them. But I still wanted them to be more flexible to be able to cover more in future. Need to say, at that time, I didn't plan this big change so soon. I was thinking more about version 7 or 8.

My goal at that time was just some kind of prototype. I had about 3 hours before my usual sleep time and I was too lazy to just write something from scratch. I spent first 20 minutes by searching through my CDs of archived university projects until I found one project where we implemented a very simple language parser. Looking back at it, it was quite lousy code. I took it's lexical analyzer, did some refactoring, and after about one hour I had a working lexical parser that was able to process C# language. I wasn't able to remember how LL, LR or whatewer language grammars we studied work anymore, so I just spent another two hours to write an object oriented syntax parser that was actually able to do the work. By that time, my goal narrowed down to be able to support logical operators and function calls, and my prototype engine was able to properly parse the syntax tree from simple expressions (including brackets and priority of operators). I was so excited about it that I spent another two extra hours that night to finalize it to some state worth of presenting to others, and after that several more evenings to support even more. Still, it wasn't on our official plan, so I had to just keep it aside for a while, until we finally had the time to somehow integrate it. At that time, I pretty much left it to my coleague Stepan Kozak, who is also very much into the low-level stuff such as myself. As the time went, he finalized it as our ultimate macro language. By adding support for code blocks, conditions, loops and other stuff, it got on much higher level than I expected.

So technically, you have K# now thanks to me being bored, having that university project assigned in my student's days, and also thanks to Stepan being on my team, because otherwise I would never get enough time to finish it till version 6.

And where the name K# comes from? It was simply too complicated to talk about it as "The new macro engine" ... "You mean the 5.5 with parameters or 6?" ... "The NEWest one!" ... so K# was a code name that was cool enough to make it public ;-)

As you may see, I like to tell stories. The reason behind it is that I see how these improbable coicidences have a potential result on a lot of lifes. And I love it! The life would be boring knowing exactly what will come. Where our solution is now is mostly based on some of you being stubborn enough to make us see your point of view, or a random talk with somebody in the hallway during our events. So in the end, it is also your story behind this whole great product ...

How does it work

As I mentioned, there is an object oriented parser behind all this. If you type a macro expression, it gets parsed into a tree of simple operations. It is quite similar to what LINQ does, but we didn't want to use those so we can do anything with it in future to extend it. This expression tree is cached, so the engine doesn't need to parse it over and over, and the resulting performance can actually get even better than with old macros.

When an expression is executed, it actually walks that tree, processing information that is needed. Technically, it works the exact same way as most of today's scripting languages (at least I suspect that). Also, similar to those, there is no strong typing, everything is processed as general object, and the resulting expression is then converted to a string.

What is great about it, having this structure and parsing, we can do whatever we wish to improve it. In fact, we have been very nice to your end users, and made some things less strict than in C#, helping them out against their small mistakes as you will see later in this article.

Accessing data

The whole point of macros was and still is the ability o populate dynamic data in a text or a general property (in case of web parts). Macros without the proper access to the data would be useless. Our other feature ... revised bussiness layer ... actually played a very important role in this. With it, you can hiearchically access any data in the CMS database through object oriented model by accessing properties or indexers of various objects. K# is actually connected to this model, and is able to get the same data with macros.

If you browse to the page ~/CMSAdminControls/UI/Macros/ObjectBrowser.aspx, you can see this whole model available to macros. This UI serves for both educational and testing purposes, and requires the global administrator access.



As you can see, it will allow you to browse anywhere through the system, to prevent unwanted access, we have a security model set up on macros, that prevents that. I will describe it in a few moments.

Anyway, there is a bussiness layer behind this that provides various collections of system objects, and you can access those with macros, for instance:

{% GlobalObjects.Users %} gives you the list of the system users. You can do the same with {% GlobalObjects["Users"] %} or {% GlobalObjects["cms.user"] %}.

This is where K# actually gives you more flexibility than C#. Since it is data-driven, it is able to dynamically provide indexers as regular properties to allow you for a cleaner and nicer code. If you want to do the same in C#, you need to use indexers.

You might also noticed that you can use constant values in your macro code, whether it is a string or number, you can just use it as you are used to in C#. 

The best way how to learn how to use it is to use one of our macro selectors for inserting macros. You can find them in various locations, e.g. if you edit an e-mail template. That macro selector allows you to select from this exact tree (based on the template type with some additional data related to context), and you can easily see the resulting macro.



I mentioned that the whole engine is data driven. Another great feature it therefore allows is transforming foreign keys directly to properties as target objects, unfold the CurrentUser to see that:



It works automatically, each foreign key in an object (in this case UserBadgeID) is automatically presented as the target object through property with the same name, just without the ID suffix. It works this way with all the objects.

Intellisense

Having a data-driven macro engine with access everywhere and C# syntax, we were able to built Intellisense functionality around it. Wherever you type a macro, you get the hints on what is available for you with autocomplete functionality ...



Also, notice here how the K# code gets it's own syntax highlighting, no matter in which context it is placed ...

There is only one minor issue with the Intellisense of K# ... the first version of K# in Kentico CMS version 6 only uses the current context data to populate everything, so you may not reach the items in enough depth. See that CurrentDocument looking empty in the tree above? That is because in that context, there was none available. When a context value returns null, the macro Intellisense doesn't know where to continue. But don't worry, we are covering that in version 7 and you will be able to browse items to any depth regardless of whether they are available from context or not. We have already prepared a "virtual mode" for our macros to be able to handle that.

Security

I promised I will tell you something about security of macros. There are several key points you need to know:

• For simple expressions like {% CurrentUser.UserName %}, {% CurrentDateTime %} (in general mostly those that were provided earlier), the security is not checked at all, one of the reasons is backward compatibility, and another is that those are values directly related to currently displayed context so there is no reason to restrict user from those.
• If an expression goes through a collection of objects or through a nested object, it checks the security. E.g. {% GlobalObjects.Users %} or {% CurrentUser.UserBadge.BadgeName %}

The security is always checked against the user that INSERTED or EDITED the macro, not the one who is seeing the results. The reason is to be able to evaluate values even for anonymous users. Basically, the person who inserted the macro permits the people who will see it's result to see the data that he or she is able to see. If you write a macro as global admin, you can write any macro. If you write it as a very limited user, you and nobody who sees the result is able to see the data that you can't see.

Macros are signed automatically with the user signature, you may notice that at the end of any macro after you submit it, where you can see the hash sign (#).



If a user changes the macro or deletes the hash sign, the macro gets re-signed by that user, so technically even users with not enough permissions can prepare macros, and then admin can sign them for them by removing the hashes and re-submitting the value. In fact, the hash is just a simple visual representation of much more complex signature, that is kept in background, just adding a hash sign won't obviously sign the macro properly if you thought about that ;-)

One important note: The whole K# is a READ-ONLY language. Everything it does, including assigning a variables as you will see in a few moments, is done in a sandbox, there is no chance how to change some data in the system through macro engine (unless you do it from your own registered method written in C#).

Logical (and other) operators

As I mentioned above, one of the main reasons for the new macro engine was the ability to use operators. You can use them as you are used to in C# with the same meaning, as well as you can use brackets to define priorities of the operations. Let me just give you one small example of something that you can put to a visibility field of some web part:

{% (CurrentUser.UserName == "Andy") && CurrentUser.IsMale  %}

Here you can see the logical AND (&&), and also a comparison to a string. Comparison is one of the things that is by default different from C# to make typical scenarios easier, here are some rules that you should know:

• Comparison (and all string operations) are by default case insensitive, e.g. {% CurrentUser.UserName.Contains("Admin") %}.
• If you compare an object to a string, it matches if either code name or display name of the object matches the string, e.g. {% CurrentUser == "andy" %}.
• All operators have the same priorities as in C# e.g. {% A && B || C && D %} will first evaluate "and" and then "or".

We also have an assignment operator, so you can store results of macros for later use (only within the same resolver object):

{% A = C + D; return A*A; %}

But ... you may also use single equal sign as a comparison, which is used in case the left-hand side is a read-only object (e.g. property of some object or an object) so even if the end user writes {% CurrentUser = "andy" %}, it gives the results as he or she expects. This is another difference from pure C#.  

Keywords and statements

You could already notice the keyword return. Similar to C#, in K# you can control the flow of the macro. By default, the macro returns the value of the last expression, but you can finish the execution by using the keyword return as you can see above. You can also use:

IF conditions:

{% if (CurrentUser.IsMale) { return "Male" } else return { "Female" } %}


WHILE loops:

{% i = 0; while (i < 5) { ... i++; }   %}


FOR loops:

{% for (i = 0; i < 5; i++) { ... }  %}


And even FOREACH for collections:

{% foreach (user in GlobalObject.Users) { print(user.UserName); } %}


In the last example, you may see a print function, which is collecting the result in a similar way as console does, and when the macro finishes, it outputs that instead of the default result.

I don't want to go into to much details, because the full reference is available in our documentation. Just know that you can use a lot of things that you are used to use in C#, including break and continue for loops.

Calling methods

Just as in C#, in K# you can call either standalone methods or methods on objects. You can do it exactly the same way as you are used to in C#, here are few examples:

{% CurrentUser.UserName.ToLower() %}

{% ToLower(CurrentUser.UserName) %}

As you can see, you can either call the method from an object, or pass the object as a first parameter of that method if it is used as standalone one.

{% currentUser.userName.contains("abc") %}

As I mentioned before, the string comparison is case insensitive. So is the processing of property names, and method names, so you don't need to worry that much about case sensitivity typo's.

{% EProductsTable.ApplyTransformation("Ecommerce.Transformations.Order_EProductsTable") %}

Did I mention that calling method on a collection is the same as doing foreach through that collection and calling the method on each of it's members? Take it as a fact, it behaves like that by default. (what else to do, right?) The example above shows you how you can render a list of items, in this case table of E-products using a new type of transformation called text transformation, but I will explain after I finish the options of K#.

The methods in intellisense display based on the current type of object that is evaluated, so you will typically get string methods for text properties of objects, etc. We have also hidden some of the methods under "namespaces" so they don't bother you if you don't need them. For instance, we have a Math namespace, from which you can call mathematical methods:

{% Math.Cos(A) %}

The namespace is more an Intellisense thing to organize available items in a better way, you can use those methods even without the namespace and it will work as well:

{% Cos(A) %}

But the very best thing about methods is that you may register your own ...

Custom methods

Registering custom methods is very simple, you just need to provide enough information for Intellisense, because they will be available there as well as our default methods. The best way how to register custom methods is to provide your custom module, that registers the methods within it's initialization code. Here is some more information how to do that: Code customization in Kentico CMS 6, but we will stick with sample that we provide in default installation for now.
 
See ~\Old_App_Code\Samples\Modules\SampleMacroModule.cs file. The Init method (if you uncomment it) calls method from another file, just to keep the code separated: 

CustomMacroMethods.RegisterMethods()

That file is located here: \Old_App_Code\Samples\Classes\CustomMacroMethods.cs

There you may notice the registration of the method by calling line such as:

MacroMethods.RegisterMethod("MyMethod", MyMethod, typeof(string), ...)

There is quite a lot of parameters, the first three are the most important ones:
 

• "MyMethod" (1st parameter) - Is a name of the method as it will be used by macro engine.
• MyMethod (2nd parameter) - Is a function that represents the method in your code and gets called by the macro engine.
• typeof(string) - Is a type on which the method can be called, so the Intellisense knows which methods to offer in particular locations of the code.

The rest of the parameters are also mostly for the Intellisense, and you may also see in that example how to register a code snippet that helps you to faster write your code. I should probably also mention, that we provide snippets for the typical scenarios such as loops and conditions as well:



Let's look at how MyMethod is implemented, there are actually three of them in that file, but the one registered is this:

/// <summary>
/// Wrapper method of MyMethod suitable for MacroResolver.
/// </summary>
/// <param name="parameters">Parameters of the method</param>
public static object MyMethod(params object[] parameters)
{
    switch (parameters.Length)
    {
        case 1:
            // Overload with one parameter
            return MyMethod(ValidationHelper.GetString(parameters[0], ""));

        case 2:
            // Overload with two parameters
            return MyMethod(ValidationHelper.GetString(parameters[0], ""), ValidationHelper.GetString(parameters[1], ""));

        default:
            // No other overload is supported
            throw new NotSupportedException();
    }
}

The methods that get registered in the macro engine always have variable number of object parameters. That is because the engine being general can pass pretty much anything to the method. Having that, you must process the parameters and potentially check their number and types within the method itself. In this case we have variants for 1 and 2 parameters, and no more parameters are supported. On the other hand the method returns an object, so the macro engine can pass it futher or access the properties of it from the result of the method.

Go ahead and play with it a little.

I mentioned in the security topic that you can potentially implement some working (or updating code) here in custom methods, but I do not recommend it, the macro engine is meant to be READ-ONLY from it's nature and typical usage scenarios.

Feeding the macro engine

Knowing about how you can register you custom methods, you should know what allows the macro engine to be so flexible. We have several interfaces that our objects implement, that are able to drive the macro engine (and the business layer in general) and it's Intellisense:

• IDataContainer (or ISimpleDataContainer) - Object that provides simple value fields, such as columns from database, e.g. {% CurrentUser.UserName %}.
• IHiearchicalObject - Object that provides properties that expose nested objects, either related or children of the object, e.g. {% CurrentUser.Children %}.
• INameIndexable - Special interface that is used for collections to provide a string indexer on it's members, e.g. {% GlobalObjects.Users["Andy"] %}.

When the indexer key is in the format of identifier, you can also access that item as a property {% GlobalObjects.Users.Andy %}, however if it doesn't match the identifier format or is somewhat variable, you need to use indexer.

We also support some of the general system classes, such as DataRow, DataTable, DataSet, but I recommend to rather go with the interfaces above, which gives much better transparency. By the way, we also have wrappers for these, such as DataRowContainer to be able to pass that object as IDataContainer object.

If you want to register your own data to a resolver object, you can do it this way:

var resolver = new ContextResolver();

// Register admin object
var admin = UserInfoProvider.GetUserInfo("administrator");
resolver.SetNamedSourceData("Admin", admin);

And then resolve the macro in a regular way:

resolver.ResolveMacros("Hello {% Admin.FirstName %}!");

You should use this only in case you are using the resolver in your own code, because in this case you need to prepare the data in advance, which is not very efficient in case you are not sure whether you will need it or not. For such cases, it is better to register a method, and let the macro engine call it only in case it really needs it.

Lambda expressions

Because sometimes you need to define something more complex operation that you would need to copy over and over, K# supports lambda expressions. It is another way how to write user-defined functions, but this time available only within the scope of resolving, not globally, and only limited to what macro engine can do. Here is an example of that, again, very, very simple:

{% myMul = ((x, y) => x * y); myMul(2,3) %}

We defined a function called myMul that does multiplication of two numbers, and then we simply call it. 

Caching

You are reading it right! When it comes to K#, we have a caching solution for your macros. Since they can access the system data in quite a depth, they may become quite resource-expensive. You can choose from two ways of caching:

Caching of the whole data context:

We have a CMSDataContext object that represents the whole CMS structure that you could see above. This is by default available through CMSContext.Current within your macro, but it's properties are available even if you don't use it specifically to make things simpler. What you can do in macros is following:

{% CMSCachedContext().GlobalObjects... %}

There are some more parameters available (such as cache minutes, item name or dependencies) but these are the basics. In this case you get the whole hierarchy of the CMS cached and the object keeps everything until the whole context object expires (by default in cache minutes set in settings). This way you can access various locations and everything will be cached together.

Caching of particular result of expression:

The other way is to cache the result of a particular expression, like a single-place independent caching. In that case, we have a special method registered called Cache which gets the same parameters as our CMSCachedSection object that you use in the API. Here is again just a simple example, you can find more in our documentation:

{% Cache(GlobalObjects.Users["Andy"]) %}

The engine is so smart, that even that you don't give it any more parameters, it generates itself a default cache key, and cache dependencies on everything it used for the evaluation, so in most cases you don't even need to worry about that.

This is one of the reasons why we didn't wan't to base K# on something existing, because this method is executed more as a functional programming method, while other methods are executed as standard procedural programming ones. This allows our engine to execute the parameter only when the method wants internally (when the cached result is not available). See how it all fits together forming an ultimate language for your web site?

I don't want to go into too much detail, because this article is too long already, so just know that you may write such functions as well, because one of the overrides for registering macro methods has parameter called parametersAsExpressions which says if you get the method parameters as values or expression trees that you let evaluate upon request.

Let the world know via a Tweet or whatever channel if you just said "WOW" ... :-)

Future plans

I think that is about it for today, I hope I didn't forget anything important. You could potentially learn even more, but I don't want you to get confused too much by so many options, so I will keep it for some later time. What I can tell you already that we plan for version 7 is:

• Support for "open" macro conditions and loops, e.g. {% if (a > b) { %}<div>Hello</div>{% } %} similar to what you can do in ASPX or Razor code.
• "Nicer" API for registering methods.
• Ability to register "lazy load" properties in a similar way as methods.
• More advanced Intellisense with "virtual mode" to browse even not available objects.
• Provide a priority properties based on context so that the users have the typical items on the top of the macro selector tree.
• Tweaking it further based on your feedback.
• Based on your current feedback even simpler rule designer for end users similar to what you can find in MS Outlook.
• More support of macros (mostly macro conditions) throughout the whole system.
• More detailed and readable macro debug, because with the new macro engine, the macro debug got a little crowded and is not as friendly as we would wish.

Anyway, here is a full reference in our documentation, and I hope this article together with it will help you to leverage all it's powers you need:

http://devnet.kentico.com/docs/6_0/devguide/index.html?macro_expressions_overview.htm

Enjoy and let me know how you feel about it and what we can do to make it even better

See you next time!