Sitecore cache problem when iterating through the content using API

Sitecore uses a number of various caches to keep the most frequently used data in memory instead of making requests to a database every time API needs the data.

The most important thing here to remember is that the cache should keep the necessary data only. However, there are a number of cases when things might get wrong and cache will start keeping data that has been accessed once. In this case, next time, you need the item which is not stored in cache, Sitecore will make a database request and cache the item again. Later, the obsolete data will be removed from the cache (in other words, the cache will be recovered) but this requires some time. During the recovering time, the performance of the processing requests will be decreased.
Things might become even worth in case the database server is located far from the Web servers and latency start playing its role.

Which operations might cause problems with cache?

I would say that any that work with a great number of data items that are not supposed to be cached or requested again during a short period of time.

A few examples:

1. Reindexing data. 
In this case, the indexing API iterates thought the Sitecore data (tree) and perform an indexing operation. As a result, all indexing items will be added to Sitecore caches. Just imagine, what will happen in case you are reindexing the whole content tree with millions of items... The cache will be overfilled soon and Sitecore will start cleaning it up in the middle of the indexing operation. Thus, instead of just indexing data, the processor time is spent on cache operations and further data clean up.

2. Publishing data
Similarly to the indexing, the publishing process is accessing a number of data, thus a number of odd items are cached.

3. Sitecore initialization logic
Sitecore initialization logic performs a number of operations that also require access to the items. For example loading localization data, scanning some items to load application settings, etc.

What can we do to improve the situation?

The easiest solution that came to my mind is using a sort of cache disabling context that would prevent data to be cached.
Sitecore allows disabling data caches by using Sitecore.Data.DatabaseCacheDisabler:

using(new DatabaseCacheDisabler())
{
  // your code here...
}

All the code, executed in DatabaseCacheDisabler context will not get and put data from \ to Sitecore caches. Another good thing is that DatabaseCacheDisabler inherits from Switcher class that is thread static. This means that cache disabling logic will be performed in current context only and will not influence other threads.
The bad thing about this code is that it will not get data from the cache even if it is there! In other words, if part of the items, you need to work with, from your code is already in cache Sitecore will not use them and make a request to a database by introducing a performance penalty.

To improve the performance we would rather want to get data from cache if it is already there but do not add new data to Sotecore caches if we read it from a database.

After some investigation, I have managed to find the switcher that does the trick: Sitecore.Data.CacheWriteDisabler.
The usage of this class is quite the same as the previous one but the code in the scope of this disabler will be using cached data if it is available.
Performance tests confirmed that the code in the scope of this disabler works faster than when I was using just DatabaseCacheDisabler.

At the end, I would like to warn about using of disablers in Sitecore.
One should remember that in case the code inside the disabler not only reads the items but also modifies them, then changes you did with these items will not appear in Sitecore caches if these items have alreay been cached.
Thus, one should use disablers wisely without breaking the cache integrity.

Using PowerShell with Sitecore

Recently, I have been asked whether it is possible to use PowerShell to work with Sitecore services.
I have never tried this before. I knew about existing modules that allow managing Sitecore via PowerShell.
After brief research, I found a few but all of them require a custom package to be installed on the Sitecore instance. This is not what I would like to do without knowing all the details about the package to be installed. It would be interesting to check what we can do with a clean Sitecore instance.

As a starting point, I have installed clean Sitecore 8.2 Update-3 (with the hostname sitecore82u3) and started my experiments.

Requesting not protected page using PowerShell

The task seems trivial but still quite useful.
For example, you might have a page that would return some statistics or perform some actions basing on passed parameters. I have decided to try to request sample layout.aspx page that is located in /layouts folder by default.
The PowerShell command looks simple:

Invoke-WebRequest -uri "http://sitecore82u3/layouts/sample layout.aspx"

I have got a response with status code 200 and page content.
Good, but what if I need to request security protected page that requires login before I continue?

Requesting protected page using PowerShell via login dialog

Once I request protected page e.g. /sitecore/admin/cache.aspx I will receive 200 response code but from the content, I can figure out that my request has been redirected to the login page:

Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx"

Response content fragment:

...
<title>
        Sitecore Login
</title>

...

Thus, we need a way to fill in the login credentials before requesting the protected page. The easiest way of doing this is to get the input controls from the page and set some data there:

# define session variable
$session = $null

# url to the login page in admin
$loginUrl = "http://sitecore82u3/sitecore/admin/login.aspx"
$actionResponse = Invoke-WebRequest -uri $loginUrl -SessionVariable session -UseBasicParsing
$fields = @{}

#search for input fields
$actionResponse.InputFields.ForEach({
if($_.PSobject.Properties.name -match "Value"){
    $fields[$_.Name] = $_.Value
  }
})

# Set login info. Note: this code is specific for admin login page. For standard Sitecore 8.2 Update-3 login page one should use $fields.UserName and  $fields.Password
$fields.LoginTextBox = "sitecore\my_user"
$fields.PasswordTextBox = "my_password"

# Perform POST request with credentials
Invoke-WebRequest -uri $loginUrl -WebSession $session -Method POST -Body $fields -UseBasicParsing

# Using authenticated session make a request to a protected page
(Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx" -WebSession $session).Content

In result, you will see the cache page content.

Current approach works but... we should remember that current approach works with the html markup that might be changed at some point and your code will not work. It would be better to use a better approach.

Requesting protected page using PowerShell via login endpoint

After some investigations, I have noticed that Sitecore includes Sitecore Client Services component by default. This component allows creating own services easily and provides Login endpoint service. This is exactly what I need.
To authenticate the user we just need to update our script to use service instead of working with login page:

# define session variable
$session = $null

# url to the login page in admin
$loginUrl = "https://sitecore82u3/sitecore/api/ssc/auth/login"
$params = @{"domain"="sitecore";
        "username"="my_user";
        "password"="my_password";
    }

# Perform POST request with credentials
$actionResponse = Invoke-WebRequest -uri $loginUrl -SessionVariable session -Method POST -Body $params -UseBasicParsing

# Using authenticated session make a request to a protected page
(Invoke-WebRequest -uri "http://sitecore82u3/sitecore/admin/cache.aspx" -WebSession $session).Content

After executing this script, I was able to see the content of the protected cache page.

Current approach looks more secure one since the login endpoint requires SSL connection configured and will not allow http connections.
Another important note about this endpoint is related to the fact that, by default, it is configured in a way of allowing local connections only and will reject all remote ones.
In case you need to connect to remote Sitecore solution, I can see at least a few options:

1. Use PoweShell to connect to the remote server and proxy requests so that real requests are done by remote server to a local Sitecore instance.
2. Configure Sitecore Client Services to process all requests by changing value of the setting "Sitecore.Services.SecurityPolicy" in "Sitecore.Services.Client.config" configuration file. 

To me, the first option seems more appropriate. It does not open a potential security problem and does not require updating Sitecore configuration.

In addition, it is important to mention that, using described approach you can make calls to Sitecore services based on Sitecore Client Services component, WebAPI component or any other services protected by Sitecore security.


PS: while searching for Sitecore Client Services login endpoint I have also noticed that Sitecore WebAPI component also provides login endpoint: "authenticate". However, I decided not to describe its usage here. The component has not been updated for a long time and the main development has been shifted to a more powerful SSC component.

Sitecore Fast query: performance problem

Sitecore Fast Query is designed for retrieving and filtering items from the Sitecore database. Sitecore
Fast Query uses the database engine to execute queries.

Original article with detailed information about the Sitecore Fast query you can find here.

Looks good, right?

The feature is cool, especially when you need to find all items with a specific name or field value which is not possible to do via Database API.
In Sitecore 7+ version one can use Content Search feature for this purpose but... sometimes the index is not up to date or field data might be stored in a different way than you expected, etc.

Getting data using Fast query could be very useful in some cases and do not introduce additional dependencies.

Let's have a look at the performance, is it good enough?

Sitecore Fast Query converts all the query conditions into SQL statements, and this places certain limitations on the use of queries.

It is more or less easy to generate an SQL statement for getting data for the statement that defines item fields and properties conditions (e.g. find items by some field value, by template ID or name and other combinations). But, what is happening when we need to search for items under the specific location?
For example: fast:/sitecore/content/Home//*[@@templatename='Sample Item']

Since Sitecore database structure does not keep item paths in tables and the Sitecore content tree is build basing on PrentID reference, we should create an SQL statement that should join Items table a number of times. 

To improve the performance of searching in the hierarchy and simplify the SQL statement, Sitecore databases have a special table that keeps item-parent relations.  This table updated on every item save \ move \ delete operations. This slows doewn item operations a little bit but allows to use item hierarchy search in Fast Query.

Let's have a look at the queries, corresponding SQL statements and execution time.
Test Data:
800k items in Items table
6M records in Descendants table
3M records in Versioned fields table

Fast Query
fast://*[@@templateid = '{0}']

This query is translated into the simple SQL that searches for all items with a specified template.
The execution time of such request is 0.47 ms

If we change this query a little bit and try to search items with a specific template in a specific location (under content item), the picture will be a little bit different.

Fast Query
fast:/sitecore/content//*[@@templateid = '{0}']

This request is executed ~1500 ms


I have got a little bit worth result with searching by field.

Fast Query
fast://*[@__lock='%\"" + Context.User.Name + "\"%'] is executing ~820 ms

while the same query but under specified location is executing ~3200ms

Fast Query
fast:/sitecore/content//*[@__lock='%\"" + Context.User.Name + "\"%']

Summary

Sitecore fast query can be very effective and useful for cases when one needs to select items using conditions that do not involve item hierarchies.

Some could say that Fast Query does not use cache and always perform requests to SQL server. Well, yes, this is true but fast query execution is not the common type of the requests to the database and spending resources on caching and invalidating the data in this cache might appear more expecnsive then just request the data from database. Also, the data constructed from the fast query request is taken from the cache. Thus, it might be not so bad.

As for the hierarchical data (queries that use item paths, I would recommend to review the query and search for all data that fits your conditions using fast query and then filter this data by location on the Web server side. For huge solutions with a lot of content, this approach will work much better. To test this I have replaced query
fast:/sitecore/templates/System/Analytics/External/Matchers/Client Matchers/* with just Database.GetItem(<item_path>).GetChildren() call and got 10 times faster execution.

Thanks.

What is Sitecore update package?

As you probably know, Sitecore distributes updates via special packages, called Sitecore Update packages. These packages have ".update" extension and can be installed using special page, called UpdateInstallationWizard that can be found in /sitecore/admin folder.

What is in the package and how is it different from standard Sitecore package?

The standard Sitecore package is designed for moving content from one Sitecore instance (server) to another. Thus, the operations that must be supported for this scenario is just adding Sitecore items, files and security roles. In other words it should support Add operations.

The Sitecore update package is designed for upgrading Sitecore instances between minor Sitecore versions. The main difference of upgrading process in comparison to just copying data is that during upgrade the most common operation not Add (install new item or file) but upgrading already existing content with support for analyzing and resolving conflicts.

The main features of the Sitecore update package:

• The package supports such commands like:
  • Delete items and files;
  • Add items and files;
  • Change specific set of the item fields;
  • Change the file (actually for usual files it is implemented as a replace);
  • Patch Xml file (the feature is disabled by default and not used right now);
• Generate a "real" rollback (uninstall) package during the package installation. The rollback package will contain the only revert operation for actions that have been performed.
• Support for Analyze (Dry Run) mode, when you install the package without real installation. Very useful mode for analyzing and fixing potential problems with the package.
• Support for 2 different installation modes: Install (when we guarantee that Sitecore will contain all features form released Sitecore) and Update (when we guarantee that in case of conflict with customer data, the custom data will not be lost).
• Support for post installation instructions that allow you to execute a custom code after package installation is finished.
• Using API from Sitecore.Update assembly, one can generate a snapshot package - the package that contains information about files and items but not their content. This package has a small size can be used as a source for generating update packages. This is very useful since you don't need to have Sitecore instance up and running.

The package format.

Despite the fact the package has extension ".update" this is just usual Zip archive. In this archive you can see folder structure that keeps installation commands grouped by command type. For example, all add item commands will appear in "addeditems" folder, deleted items - in "deleteditems", etc. Every entry in these folders are either just regular folder or xml file that contains command metadata and instructions.

User scenarios we considered when working on update package framework.

Actually when designing the framework we wanted to cover one scenario - simplify the upgrade procedure between minor versions. However the scenario is currently can be achieved using 2 different approaches.

• Upgrading Sitecore instance.
• Extracting added and customized content with further installation to the new version of Sitecore (in other words - moving the custom data to a new instance).

The first scenario requires generating an update package that contains differences between Sitecore versions.

Props:

• A Single package that can work for a number of customers.
• Does not require access to customer solutions.
• The package is cumulative which means that using the same package you can upgrade from any version in between source and target.

Cons:

• It is too generic and does not reflect customization.

The second scenario requires generating an update package that contains only customized data that can be easily moved to a different Sitecore instance.

Props:

• In some cases it might be much easier to upgrade.
• Having a single package with custome data simplifies upgrade to any Sitecore version.

Cons:

• Package generation logic should be moved to a customer side which means that customers should have very strong knowledge about changes we done in Sitecore. Otherwise the upgraded solution might be broken.

How to generate an update package?

All API for generating packages, working with package data and installing update package is located in Sitecore.Update assembly. We just use a set of batch files that calls necessary API.

Also there are a number of custom applications that use our API and allow to integrate package related commands into explorer context menu or even in Visual Studio.

Is it possible to control of what is going to be added to the package?

Yes, the package generation API supports file filters (to filter scanned items) and custom C# filters (to filter or change commands in already generate package). All filters can be configured in configuration file or added from code.

Is there an easy way to explore the update package?

I know at least several tools that could represent the package data in a readable way.

Personally, I either unpack package files and analyze it's structure or use own tool for analyzing package.

The Package Analyzer tool features

• Has a set of filters to filter package commands.
• Can perform initial package analysis for a number of dangerous commands like changing configuration files, etc.
• Allows the extract filtered commands to a text file.
• Can show the command Xml.
• Can compare 2 update packages and use WinMerge app for showing diff between commands.

Breaking changes: Introduction

Since I am working in Sustained Engineering team, one of my activities in the company is working on Updates.

Update in our company must improve the quality of the specific major version of the product by resolving a number of associated issues.The update release should comply to a number of requirements. Some of them (that I personally consider as very important) are:

• Must improve the overall quality of the product by fixing issues received from our customers and internal teams. At the same time regressions are not allowed and must be fixed before the release.
• Must not contain breaking changes, thus customer should not rebuild (recompile) their solutions and\or workarounds after upgrading to the Update.
• Must be backward compatible, thus all customer applications and features should work without any changes.

The main idea behind these requirements is to simplify upgrade procedure by minimizing the number of changes from customer side.

In this post I would like to share some thoughts about one of the requirement we have - breaking changes.

Let's start from the definition:

breaking change - a change in one part of a software system that potentially causes other components to fail; occurs most often in shared libraries of code used by multiple applications.

As you can see from the definition it is not clear of what exactly can be considered as breaking. It can be any change that could break existing solution. Personally I differentiate changes by categories.

Note: it could vary from one project to another. Current categorization is project specific.

Categories of breaking changes:

• Breaking Changes in API
• Breaking Changes in UI
• Breaking Changes in Configuration
• Breaking Changes in Database schema
• Breaking Changes in data, stored in database

1. Breaking Changes in API

API change - a change in the publicly visible definition of a type, including any of its public members. This includes changing type and member names, changing base type of a type, adding/removing interfaces from list of implemented interfaces of a type, adding/removing members (including overloads), changing member visibility, renaming method and type parameters, adding default values for method parameters, adding/removing attributes on types and members, and adding/removing generic type parameters on types and members, etc. This does not include any changes in member bodies, or any changes to private members (i.e. we do not take into account Reflection).

Source: http://stackoverflow.com/questions/1456785/a-definitive-guide-to-api-breaking-changes-in-net

Check for the most of the breaking API changes can be automated. For example, you can configure you CI to execute LibCheck tool on every build and send a report in case any breaking changes detected.

However there are some changes in API that could break the API but not detected by the tool. In some sources this is called binary level break - an API change that results in client assemblies compiled against older version of the API potentially not loading with the new version. Example: changing method signature, even if it allows to be called in the same way as before (ie: void to return type / parameter default values overloads).

There are more types of breaking changes in API (changes in behavior, semantic, changes in abstract classes that just throw an exception when implementation is missing, etc.).

Note: I am going to have a separate post, with various examples, related to breaking changes in API.

The strategy is our company is to use a combination of code review, the tool that could detect the breaking changes and running unit and integration tests on CI. The better coverage you have in your tests the higher chance to catch breaking API change before the release.

2. Breaking Changes in UI

In relation to UI we are trying to avoid of making any changes here. Only minor fixes with styles are possible.

Such approach is related to several things:

• Any changes in UI requires review and updating documentation (at least we have to regenerate screenshots).
• Customers might use own versions of existing pages or dialogs with some modifications. Usually this is done by copying existing pages and inheriting form existing classes. In case we remove any controls or move them to a different place this might break their customization.

Changes we usually allow to do in updates:

• Make changes in styling and sizes.
• Move controls inside the container, however we should avoid of changing the control order.
• In some situations adding new controls might be Ok (depends on situation).

To say the truth any changes in UI should be analyzed carefully.

Possible way to track changes in UI:

• Analysis of changes before fix.
• Code Review.
• Automated UI testing.
• Manual UI testing.

3. Breaking Changes in Configuration

The general rule in relation to changes in configuration is "Do not change the configuration."

The changes in configuration are almost always done manually and bring additional headache for the customers who are doing an upgrade.

In case you do need to make changes in configuration files, the only allowed changes are:

• Changes in comments :).
• Adding new nodes (preferably at the end of the node set).
• Adding new attributes (preferably at the end of the node set).

Any other changes, like changing the order of the nodes, renaming nodes, changing node or attribute values, etc., will change the node xPath which means that this can potentially break the custom logic or custom patch files.

The check for changes in configuration can be automated by comparing with configuration from previous version.

Another good indicator for breaking changes in configuration is upgrade procedure. In two words - updating the configuration files during the upgrading between Updates should be optional, everything must be working at least not worth as it was before the upgrade. If you see any exceptions related to configuration after the upgrade this is an indicator that something has been done in a wrong way.

The rule: situation when necessary configuration is missing is always possible. You should have a plan "B" and use fall-backs and default values.

4. Breaking Changes in Database schema

For the changes in database schema we should use the same rule to the one we have for changes in configuration - "Avoid of making any changes in schema for minor updates".

General requirements are:

• New API should be able to work with old schema, or in a different words - updating the schema in minor updates is not mandatory.
• Applying new schema must never break or even modify existing customer data.

Changes in indexes, stored procedures are also considered as breaking since they could already have customization.

There are some tools that could be used for automated check for changes in database. One of them called "DBComparer" but there might be a lot of other that could be much better :).

5. Breaking Changes in data, stored in database.

It is very difficult to say anything concrete here since guidelines might depend on the data you have in database. For example: in case you have any template data, that could be used by your customers to create own specific content, you should ensure that once you change your template you will not break all content created by customer. You should always think about custom data once decide to change anything in your data. New API should always be compatible with data from previous version.

Summary

Please, do not forget about customers when making changes to your system. Tracking of breaking changes is not an easy thing but your customers are really appreciate this. In case you do have to introduce breaking changes, please make it visible to a customer (highlight in Release Notes, put a warning\erorr to a log, show message, etc.) and ensure the instructions for further steps are clear and easy.

Try to automate the process for identifying breaking changes since fixing them at the late stage is very difficult or even impossible at all.