Some template sections or fields are missing after using Update Data Templates commerce command in Sitecore 9.0.1

Recently I have been involved in the project based on Sitecore 9.0.1 + Sitecore SXA + Sitecore Commerce.

This was really a great experience to be and allow to look at some things from a different angle :)

In this post, I would like to share some problem I faced when extending the schema for the Sellable Item in commerce.
As mentioned above we needed to extend the default fields we can see for a sellable item in Commerce interfaces. This was not the easiest task but we managed to do this with the KB article (big thanks to everybody who was publishing it). After we added a new section we had to regenerate Commerce templates in Sitecore backend to see the new section in content items.

Problem
After we updated Sitecore Commerce templates using "Update Data Templates" commands in "Commerce" ribbon tab we figured out that some sections that were present before are missing now. The one that we noticed was section "Images". What is even more interesting was the fact that this problem appeared only in a few developers only.

Investigation and solution
After some investigations, we have found that is it "CatalogTemplateGenerator" who is responsible for template generation. The problem was related to that fact that it generates templates basing on the first available object (e.g. Category templates are generated based on the first found Category commerce entity, Sellable Item templates are generated basing on the first available sellable item commerce entity etc.). In our case, it found the same commerce entity for all developers but some developers did have images assigned to the sellable item entity in their commerce database but others - not.
The problem is happening to be jing of specific since it will not be reproduced for the bost of the sections and fields. But images field is a bit special since, on the commerce side, it is implemented as a list control. In case it has no value the information about images is not added to the entity json returned from commerce server.

To fix this problem we had to override default commerce command with an own one and use our own implementation of template generator. Unfortunately, we were not able to override implementation from the standard implementation since all methods in "CatalogTemplateGenerator" class are private and non-virtual. Thus, we had to copy-paste implementation using reflector and change the implementation of "EnsureTemplateFields" method from (taken from dotPeak):

private void EnsureTemplateFields(TemplateItem templateItem, JToken view, string section = "Content")
    {
      EntityView entityView = view.ToObject<EntityView>();
      if (entityView.Properties.Any<ViewProperty>())
      {
        foreach (ViewProperty property in (Collection<ViewProperty>) entityView.Properties)
        {
          TemplateFieldItem templateFieldItem = templateItem.GetField(property.Name) ?? templateItem.AddField(property.Name, section);
          using (new EditContext(templateFieldItem.InnerItem))
          {
            templateFieldItem.Title = property.DisplayName;
            templateFieldItem.InnerItem.Appearance.ReadOnly = true;
            templateFieldItem.InnerItem.Appearance.Hidden = property.IsHidden;
            templateFieldItem.InnerItem[TemplateFieldIDs.Shared] = this.IsFieldLocalizable(view, property.Name) ? "0" : "1";
            templateFieldItem.Type = !(property.OriginalType == typeof (bool).ToString()) ? (property.OriginalType == typeof (double).ToString() || property.OriginalType == typeof (int).ToString() ? "Number" : "Single-Line Text") : "Checkbox";
          }
        }
      }
      else
      {
        if (((IEnumerable<string>) this._blacklist).Contains<string>(entityView.Name) || !entityView.ChildViews.Any<Model>() || view[(object) "ChildViews"].FirstOrDefault<JToken>() == null)
          return;
        TemplateFieldItem templateFieldItem = templateItem.GetField(entityView.Name) ?? templateItem.AddField(entityView.Name, section);
        using (new EditContext(templateFieldItem.InnerItem))
        {
          templateFieldItem.Title = entityView.DisplayName;
          templateFieldItem.InnerItem.Appearance.ReadOnly = true;
          templateFieldItem.InnerItem.Appearance.Hidden = false;
          templateFieldItem.InnerItem[TemplateFieldIDs.Shared] = this.IsFieldLocalizable(view, entityView.Name) ? "0" : "1";
          templateFieldItem.Type = "Treelist";
        }
      }
    }

to something like this:

protected virtual void EnsureTemplateFields(TemplateItem templateItem, JToken view, string section = "Content")
        {
            EntityView entityView = view.ToObject<EntityView>();
            if (entityView.Properties.Any<ViewProperty>())
            {
                foreach (ViewProperty property in entityView.Properties)
                {
                    TemplateFieldItem templateFieldItem =
                        templateItem.GetField(property.Name) ?? templateItem.AddField(property.Name, section);
                    using (new EditContext(templateFieldItem.InnerItem))
                    {
                        templateFieldItem.Title = property.DisplayName;
                        templateFieldItem.InnerItem.Appearance.ReadOnly = true;
                        templateFieldItem.InnerItem.Appearance.Hidden = property.IsHidden;
                        templateFieldItem.InnerItem[TemplateFieldIDs.Shared] =
                            this.IsFieldLocalizable(view, property.Name) ? "0" : "1";
                        templateFieldItem.Type = GetPropertyType(property);
                    }
                }
            }
            else
            {
                if ((this._blacklist.Contains<string>(entityView.Name) ||
                    !entityView.ChildViews.Any<Model>() || view[(object) "ChildViews"].FirstOrDefault<JToken>() == null) && !entityView.Name.Equals("images", StringComparison.OrdinalIgnoreCase))
                    return;
                TemplateFieldItem templateFieldItem =
                    templateItem.GetField(entityView.Name) ?? templateItem.AddField(entityView.Name, section);
                using (new EditContext(templateFieldItem.InnerItem))
                {
                    templateFieldItem.Title = entityView.DisplayName;
                    templateFieldItem.InnerItem.Appearance.ReadOnly = true;
                    templateFieldItem.InnerItem.Appearance.Hidden = false;
                    templateFieldItem.InnerItem[TemplateFieldIDs.Shared] =
                        this.IsFieldLocalizable(view, entityView.Name) ? "0" : "1";
                    templateFieldItem.Type = "Treelist";
                }
            }
        }

Also, since already had own generator we decided to update the logic that resolves the template fields type. This logic was noticed in the same method and looks like the one below:

templateFieldItem.Type = !(property.OriginalType == typeof (bool).ToString()) ? (property.OriginalType == typeof (double).ToString() || property.OriginalType == typeof (int).ToString() ? "Number" : "Single-Line Text") : "Checkbox";

As you can see only "Number", "Single-Line Text" and "Checkbox" fields types are supported + some special logic for lists. We decided to change the implementation a bit and add support for additional types - "html" and "memo". We have extracted this to a separate method:

protected virtual string GetPropertyType(ViewProperty property)
{
  string type = "Single-Line Text";
  if (property.OriginalType == typeof(bool).ToString())
  {
    type = "Checkbox";
  }
  else if (property.OriginalType == typeof(double).ToString() ||
                     property.OriginalType == typeof(int).ToString())
  {
    type = "Number";
  }
  else if (string.Equals(property.OriginalType, "html", StringComparison.OrdinalIgnoreCase))
  {
    type = "Rich Text";
  }
  else if (string.Equals(property.OriginalType, "memo", StringComparison.OrdinalIgnoreCase))
  {
    type = "Multi-Line Text";
  }
    return type;
}

After these changes, the problem with missing "images"  section has gone and fields started to look better in Content Editor.

Changing service user for Sitecore Commerce

Recently, I have been working on the pre-production preparations for the Sitecore 9.0.1 based solution. One of the things to be done was either disable default Admin user or change its password to be more secure.

Once I changed the password for default Admin user and tried to open Sitecore Commerce interfaces I was not working. Browser Dev tools show a number of Javascript errors in console relater to the communication problems. I was even more surprised when figured out that updated Admin user cannot login into Sitecore backend.

After some investigations, I have figured out that Admin user is locked for some reason. Unfortunately, after unlocking Admin used it was locked again and again which made me think that someone is trying to login using an incorrect password and Sitecore locks the user.

The hardcoded Admin user has been found in Commerce Server configurations, in the file below:

/wwwroot/data/Environments/PlugIn.Content.PolicySet-1.0.0.json

To workaround the problem I decided to register a dedicated Commerce admin user and reconfigure Commerce server to use an own user instead of standard one.

Note: do not forget that all Commerce instances “Authoring”, “Shops”, “Minions” and “Ops” needs to be updated. To do this, one should open “PlugIn.Content.PolicySet-1.0.0.json” located in “/wwwroot/data/Environments” in every site mentioned above and update properties “UserName” and “Password”.

After changing Commerce Server configuration, one needs to bootstrap Sitecore Commerce server again to ensure the configuration is applied. This should be done using Postman requests (use GetToken to get the auth token and then call “Bootstrap Sitecore Commerce”)

Sitecore cache problem when iterating through the content using API

Sitecore uses a number of various caches to keep the most frequently used data in memory instead of making requests to a database every time API needs the data.

The most important thing here to remember is that the cache should keep the necessary data only. However, there are a number of cases when things might get wrong and cache will start keeping data that has been accessed once. In this case, next time, you need the item which is not stored in cache, Sitecore will make a database request and cache the item again. Later, the obsolete data will be removed from the cache (in other words, the cache will be recovered) but this requires some time. During the recovering time, the performance of the processing requests will be decreased.
Things might become even worth in case the database server is located far from the Web servers and latency start playing its role.

Which operations might cause problems with cache?

I would say that any that work with a great number of data items that are not supposed to be cached or requested again during a short period of time.

A few examples:

1. Reindexing data. 
In this case, the indexing API iterates thought the Sitecore data (tree) and perform an indexing operation. As a result, all indexing items will be added to Sitecore caches. Just imagine, what will happen in case you are reindexing the whole content tree with millions of items... The cache will be overfilled soon and Sitecore will start cleaning it up in the middle of the indexing operation. Thus, instead of just indexing data, the processor time is spent on cache operations and further data clean up.

2. Publishing data
Similarly to the indexing, the publishing process is accessing a number of data, thus a number of odd items are cached.

3. Sitecore initialization logic
Sitecore initialization logic performs a number of operations that also require access to the items. For example loading localization data, scanning some items to load application settings, etc.

What can we do to improve the situation?

The easiest solution that came to my mind is using a sort of cache disabling context that would prevent data to be cached.
Sitecore allows disabling data caches by using Sitecore.Data.DatabaseCacheDisabler:

using(new DatabaseCacheDisabler())
{
  // your code here...
}

All the code, executed in DatabaseCacheDisabler context will not get and put data from \ to Sitecore caches. Another good thing is that DatabaseCacheDisabler inherits from Switcher class that is thread static. This means that cache disabling logic will be performed in current context only and will not influence other threads.
The bad thing about this code is that it will not get data from the cache even if it is there! In other words, if part of the items, you need to work with, from your code is already in cache Sitecore will not use them and make a request to a database by introducing a performance penalty.

To improve the performance we would rather want to get data from cache if it is already there but do not add new data to Sotecore caches if we read it from a database.

After some investigation, I have managed to find the switcher that does the trick: Sitecore.Data.CacheWriteDisabler.
The usage of this class is quite the same as the previous one but the code in the scope of this disabler will be using cached data if it is available.
Performance tests confirmed that the code in the scope of this disabler works faster than when I was using just DatabaseCacheDisabler.

At the end, I would like to warn about using of disablers in Sitecore.
One should remember that in case the code inside the disabler not only reads the items but also modifies them, then changes you did with these items will not appear in Sitecore caches if these items have alreay been cached.
Thus, one should use disablers wisely without breaking the cache integrity.

Using PowerShell with Sitecore

Recently, I have been asked whether it is possible to use PowerShell to work with Sitecore services.
I have never tried this before. I knew about existing modules that allow managing Sitecore via PowerShell.
After brief research, I found a few but all of them require a custom package to be installed on the Sitecore instance. This is not what I would like to do without knowing all the details about the package to be installed. It would be interesting to check what we can do with a clean Sitecore instance.

As a starting point, I have installed clean Sitecore 8.2 Update-3 (with the hostname sitecore82u3) and started my experiments.

Requesting not protected page using PowerShell

The task seems trivial but still quite useful.
For example, you might have a page that would return some statistics or perform some actions basing on passed parameters. I have decided to try to request sample layout.aspx page that is located in /layouts folder by default.
The PowerShell command looks simple:

Invoke-WebRequest -uri "http://sitecore82u3/layouts/sample layout.aspx"

I have got a response with status code 200 and page content.
Good, but what if I need to request security protected page that requires login before I continue?

Requesting protected page using PowerShell via login dialog

Once I request protected page e.g. /sitecore/admin/cache.aspx I will receive 200 response code but from the content, I can figure out that my request has been redirected to the login page:

Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx"

Response content fragment:

...
<title>
        Sitecore Login
</title>

...

Thus, we need a way to fill in the login credentials before requesting the protected page. The easiest way of doing this is to get the input controls from the page and set some data there:

# define session variable
$session = $null

# url to the login page in admin
$loginUrl = "http://sitecore82u3/sitecore/admin/login.aspx"
$actionResponse = Invoke-WebRequest -uri $loginUrl -SessionVariable session -UseBasicParsing
$fields = @{}

#search for input fields
$actionResponse.InputFields.ForEach({
if($_.PSobject.Properties.name -match "Value"){
    $fields[$_.Name] = $_.Value
  }
})

# Set login info. Note: this code is specific for admin login page. For standard Sitecore 8.2 Update-3 login page one should use $fields.UserName and  $fields.Password
$fields.LoginTextBox = "sitecore\my_user"
$fields.PasswordTextBox = "my_password"

# Perform POST request with credentials
Invoke-WebRequest -uri $loginUrl -WebSession $session -Method POST -Body $fields -UseBasicParsing

# Using authenticated session make a request to a protected page
(Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx" -WebSession $session).Content

In result, you will see the cache page content.

Current approach works but... we should remember that current approach works with the html markup that might be changed at some point and your code will not work. It would be better to use a better approach.

Requesting protected page using PowerShell via login endpoint

After some investigations, I have noticed that Sitecore includes Sitecore Client Services component by default. This component allows creating own services easily and provides Login endpoint service. This is exactly what I need.
To authenticate the user we just need to update our script to use service instead of working with login page:

# define session variable
$session = $null

# url to the login page in admin
$loginUrl = "https://sitecore82u3/sitecore/api/ssc/auth/login"
$params = @{"domain"="sitecore";
        "username"="my_user";
        "password"="my_password";
    }

# Perform POST request with credentials
$actionResponse = Invoke-WebRequest -uri $loginUrl -SessionVariable session -Method POST -Body $params -UseBasicParsing

# Using authenticated session make a request to a protected page
(Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx" -WebSession $session).Content

After executing this script, I was able to see the content of the protected cache page.

Current approach looks more secure one since the login endpoint requires SSL connection configured and will not allow http connections.
Another important note about this endpoint is related to the fact that, by default, it is configured in a way of allowing local connections only and will reject all remote ones.
In case you need to connect to remote Sitecore solution, I can see at least a few options:

1. Use PoweShell to connect to the remote server and proxy requests so that real requests are done by remote server to a local Sitecore instance.
2. Configure Sitecore Client Services to process all requests by changing value of the setting "Sitecore.Services.SecurityPolicy" in "Sitecore.Services.Client.config" configuration file. 

To me, the first option seems more appropriate. It does not open a potential security problem and does not require updating Sitecore configuration.

In addition, it is important to mention that, using described approach you can make calls to Sitecore services based on Sitecore Client Services component, WebAPI component or any other services protected by Sitecore security.


PS: while searching for Sitecore Client Services login endpoint I have also noticed that Sitecore WebAPI component also provides login endpoint: "authenticate". However, I decided not to describe its usage here. The component has not been updated for a long time and the main development has been shifted to a more powerful SSC component.