Showing posts with label optimization. Show all posts
Showing posts with label optimization. Show all posts

Thursday, 25 January 2018

Sitecore cache problem when iterating through the content using API

Sitecore uses a number of various caches to keep the most frequently used data in memory instead of making requests to a database every time API needs the data.

The most important thing here to remember is that the cache should keep the necessary data only. However, there are a number of cases when things might get wrong and cache will start keeping data that has been accessed once. In this case, next time, you need the item which is not stored in cache, Sitecore will make a database request and cache the item again. Later, the obsolete data will be removed from the cache (in other words, the cache will be recovered) but this requires some time. During the recovering time, the performance of the processing requests will be decreased.
Things might become even worth in case the database server is located far from the Web servers and latency start playing its role.

Which operations might cause problems with cache?


I would say that any that work with a great number of data items that are not supposed to be cached or requested again during a short period of time.

A few examples:

1. Reindexing data. 
In this case, the indexing API iterates thought the Sitecore data (tree) and perform an indexing operation. As a result, all indexing items will be added to Sitecore caches. Just imagine, what will happen in case you are reindexing the whole content tree with millions of items... The cache will be overfilled soon and Sitecore will start cleaning it up in the middle of the indexing operation. Thus, instead of just indexing data, the processor time is spent on cache operations and further data clean up.

2. Publishing data
Similarly to the indexing, the publishing process is accessing a number of data, thus a number of odd items are cached.

3. Sitecore initialization logic
Sitecore initialization logic performs a number of operations that also require access to the items. For example loading localization data, scanning some items to load application settings, etc.


What can we do to improve the situation?

The easiest solution that came to my mind is using a sort of cache disabling context that would prevent data to be cached.
Sitecore allows disabling data caches by using Sitecore.Data.DatabaseCacheDisabler:

using(new DatabaseCacheDisabler())
{
  // your code here...
}
All the code, executed in DatabaseCacheDisabler context will not get and put data from \ to Sitecore caches. Another good thing is that DatabaseCacheDisabler inherits from Switcher class that is thread static. This means that cache disabling logic will be performed in current context only and will not influence other threads.
The bad thing about this code is that it will not get data from the cache even if it is there! In other words, if part of the items, you need to work with, from your code is already in cache Sitecore will not use them and make a request to a database by introducing a performance penalty.

To improve the performance we would rather want to get data from cache if it is already there but do not add new data to Sotecore caches if we read it from a database.

After some investigation, I have managed to find the switcher that does the trick: Sitecore.Data.CacheWriteDisabler.
The usage of this class is quite the same as the previous one but the code in the scope of this disabler will be using cached data if it is available.
Performance tests confirmed that the code in the scope of this disabler works faster than when I was using just DatabaseCacheDisabler.

At the end, I would like to warn about using of disablers in Sitecore.
One should remember that in case the code inside the disabler not only reads the items but also modifies them, then changes you did with these items will not appear in Sitecore caches if these items have alreay been cached.
Thus, one should use disablers wisely without breaking the cache integrity.



Friday, 3 March 2017

Sitecore Fast query: performance problem

Sitecore Fast Query is designed for retrieving and filtering items from the Sitecore database. Sitecore
Fast Query uses the database engine to execute queries.
Original article with detailed information about the Sitecore Fast query you can find here.

Looks good, right?

The feature is cool, especially when you need to find all items with a specific name or field value which is not possible to do via Database API.
In Sitecore 7+ version one can use Content Search feature for this purpose but... sometimes the index is not up to date or field data might be stored in a different way than you expected, etc.

Getting data using Fast query could be very useful in some cases and do not introduce additional dependencies.

Let's have a look at the performance, is it good enough?

Sitecore Fast Query converts all the query conditions into SQL statements, and this places certain limitations on the use of queries.
It is more or less easy to generate an SQL statement for getting data for the statement that defines item fields and properties conditions (e.g. find items by some field value, by template ID or name and other combinations). But, what is happening when we need to search for items under the specific location?
For example: fast:/sitecore/content/Home//*[@@templatename='Sample Item']

Since Sitecore database structure does not keep item paths in tables and the Sitecore content tree is build basing on PrentID reference, we should create an SQL statement that should join Items table a number of times. 

To improve the performance of searching in the hierarchy and simplify the SQL statement, Sitecore databases have a special table that keeps item-parent relations.  This table updated on every item save \ move \ delete operations. This slows doewn item operations a little bit but allows to use item hierarchy search in Fast Query.

Let's have a look at the queries, corresponding SQL statements and execution time.
Test Data:
800k items in Items table
6M records in Descendants table
3M records in Versioned fields table

Fast Query
fast://*[@@templateid = '{0}']

This query is translated into the simple SQL that searches for all items with a specified template.
The execution time of such request is 0.47 ms

If we change this query a little bit and try to search items with a specific template in a specific location (under content item), the picture will be a little bit different.

Fast Query
fast:/sitecore/content//*[@@templateid = '{0}']

This request is executed ~1500 ms


I have got a little bit worth result with searching by field.

Fast Query
fast://*[@__lock='%\"" + Context.User.Name + "\"%'] is executing ~820 ms

while the same query but under specified location is executing ~3200ms

Fast Query
fast:/sitecore/content//*[@__lock='%\"" + Context.User.Name + "\"%']


Summary

Sitecore fast query can be very effective and useful for cases when one needs to select items using conditions that do not involve item hierarchies.

Some could say that Fast Query does not use cache and always perform requests to SQL server. Well, yes, this is true but fast query execution is not the common type of the requests to the database and spending resources on caching and invalidating the data in this cache might appear more expecnsive then just request the data from database. Also, the data constructed from the fast query request is taken from the cache. Thus, it might be not so bad.

As for the hierarchical data (queries that use item paths, I would recommend to review the query and search for all data that fits your conditions using fast query and then filter this data by location on the Web server side. For huge solutions with a lot of content, this approach will work much better. To test this I have replaced query
fast:/sitecore/templates/System/Analytics/External/Matchers/Client Matchers/* with just Database.GetItem(<item_path>).GetChildren() call and got 10 times faster execution.

Thanks.


Sitecore Content Serialization - first look

Agenda Preparations Configuration Module Configuration Performing Serialization Operations in CLI How to migrate from Unicorn to SCS Generat...