What is Sitecore update package?

As you probably know, Sitecore distributes updates via special packages, called Sitecore Update packages. These packages have ".update" extension and can be installed using special page, called UpdateInstallationWizard that can be found in /sitecore/admin folder.

What is in the package and how is it different from standard Sitecore package?

The standard Sitecore package is designed for moving content from one Sitecore instance (server) to another. Thus, the operations that must be supported for this scenario is just adding Sitecore items, files and security roles. In other words it should support Add operations.

The Sitecore update package is designed for upgrading Sitecore instances between minor Sitecore versions. The main difference of upgrading process in comparison to just copying data is that during upgrade the most common operation not Add (install new item or file) but upgrading already existing content with support for analyzing and resolving conflicts.

The main features of the Sitecore update package:

• The package supports such commands like:
  • Delete items and files;
  • Add items and files;
  • Change specific set of the item fields;
  • Change the file (actually for usual files it is implemented as a replace);
  • Patch Xml file (the feature is disabled by default and not used right now);
• Generate a "real" rollback (uninstall) package during the package installation. The rollback package will contain the only revert operation for actions that have been performed.
• Support for Analyze (Dry Run) mode, when you install the package without real installation. Very useful mode for analyzing and fixing potential problems with the package.
• Support for 2 different installation modes: Install (when we guarantee that Sitecore will contain all features form released Sitecore) and Update (when we guarantee that in case of conflict with customer data, the custom data will not be lost).
• Support for post installation instructions that allow you to execute a custom code after package installation is finished.
• Using API from Sitecore.Update assembly, one can generate a snapshot package - the package that contains information about files and items but not their content. This package has a small size can be used as a source for generating update packages. This is very useful since you don't need to have Sitecore instance up and running.

The package format.

Despite the fact the package has extension ".update" this is just usual Zip archive. In this archive you can see folder structure that keeps installation commands grouped by command type. For example, all add item commands will appear in "addeditems" folder, deleted items - in "deleteditems", etc. Every entry in these folders are either just regular folder or xml file that contains command metadata and instructions.

User scenarios we considered when working on update package framework.

Actually when designing the framework we wanted to cover one scenario - simplify the upgrade procedure between minor versions. However the scenario is currently can be achieved using 2 different approaches.

• Upgrading Sitecore instance.
• Extracting added and customized content with further installation to the new version of Sitecore (in other words - moving the custom data to a new instance).

The first scenario requires generating an update package that contains differences between Sitecore versions.

Props:

• A Single package that can work for a number of customers.
• Does not require access to customer solutions.
• The package is cumulative which means that using the same package you can upgrade from any version in between source and target.

Cons:

• It is too generic and does not reflect customization.

The second scenario requires generating an update package that contains only customized data that can be easily moved to a different Sitecore instance.

Props:

• In some cases it might be much easier to upgrade.
• Having a single package with custome data simplifies upgrade to any Sitecore version.

Cons:

• Package generation logic should be moved to a customer side which means that customers should have very strong knowledge about changes we done in Sitecore. Otherwise the upgraded solution might be broken.

How to generate an update package?

All API for generating packages, working with package data and installing update package is located in Sitecore.Update assembly. We just use a set of batch files that calls necessary API.

Also there are a number of custom applications that use our API and allow to integrate package related commands into explorer context menu or even in Visual Studio.

Is it possible to control of what is going to be added to the package?

Yes, the package generation API supports file filters (to filter scanned items) and custom C# filters (to filter or change commands in already generate package). All filters can be configured in configuration file or added from code.

Is there an easy way to explore the update package?

I know at least several tools that could represent the package data in a readable way.

Personally, I either unpack package files and analyze it's structure or use own tool for analyzing package.

The Package Analyzer tool features

• Has a set of filters to filter package commands.
• Can perform initial package analysis for a number of dangerous commands like changing configuration files, etc.
• Allows the extract filtered commands to a text file.
• Can show the command Xml.
• Can compare 2 update packages and use WinMerge app for showing diff between commands.

Breaking changes: Introduction

Since I am working in Sustained Engineering team, one of my activities in the company is working on Updates.

Update in our company must improve the quality of the specific major version of the product by resolving a number of associated issues.The update release should comply to a number of requirements. Some of them (that I personally consider as very important) are:

• Must improve the overall quality of the product by fixing issues received from our customers and internal teams. At the same time regressions are not allowed and must be fixed before the release.
• Must not contain breaking changes, thus customer should not rebuild (recompile) their solutions and\or workarounds after upgrading to the Update.
• Must be backward compatible, thus all customer applications and features should work without any changes.

The main idea behind these requirements is to simplify upgrade procedure by minimizing the number of changes from customer side.

In this post I would like to share some thoughts about one of the requirement we have - breaking changes.

Let's start from the definition:

breaking change - a change in one part of a software system that potentially causes other components to fail; occurs most often in shared libraries of code used by multiple applications.

As you can see from the definition it is not clear of what exactly can be considered as breaking. It can be any change that could break existing solution. Personally I differentiate changes by categories.

Note: it could vary from one project to another. Current categorization is project specific.

Categories of breaking changes:

• Breaking Changes in API
• Breaking Changes in UI
• Breaking Changes in Configuration
• Breaking Changes in Database schema
• Breaking Changes in data, stored in database

1. Breaking Changes in API

API change - a change in the publicly visible definition of a type, including any of its public members. This includes changing type and member names, changing base type of a type, adding/removing interfaces from list of implemented interfaces of a type, adding/removing members (including overloads), changing member visibility, renaming method and type parameters, adding default values for method parameters, adding/removing attributes on types and members, and adding/removing generic type parameters on types and members, etc. This does not include any changes in member bodies, or any changes to private members (i.e. we do not take into account Reflection).

Source: http://stackoverflow.com/questions/1456785/a-definitive-guide-to-api-breaking-changes-in-net

Check for the most of the breaking API changes can be automated. For example, you can configure you CI to execute LibCheck tool on every build and send a report in case any breaking changes detected.

However there are some changes in API that could break the API but not detected by the tool. In some sources this is called binary level break - an API change that results in client assemblies compiled against older version of the API potentially not loading with the new version. Example: changing method signature, even if it allows to be called in the same way as before (ie: void to return type / parameter default values overloads).

There are more types of breaking changes in API (changes in behavior, semantic, changes in abstract classes that just throw an exception when implementation is missing, etc.).

Note: I am going to have a separate post, with various examples, related to breaking changes in API.

The strategy is our company is to use a combination of code review, the tool that could detect the breaking changes and running unit and integration tests on CI. The better coverage you have in your tests the higher chance to catch breaking API change before the release.

2. Breaking Changes in UI

In relation to UI we are trying to avoid of making any changes here. Only minor fixes with styles are possible.

Such approach is related to several things:

• Any changes in UI requires review and updating documentation (at least we have to regenerate screenshots).
• Customers might use own versions of existing pages or dialogs with some modifications. Usually this is done by copying existing pages and inheriting form existing classes. In case we remove any controls or move them to a different place this might break their customization.

Changes we usually allow to do in updates:

• Make changes in styling and sizes.
• Move controls inside the container, however we should avoid of changing the control order.
• In some situations adding new controls might be Ok (depends on situation).

To say the truth any changes in UI should be analyzed carefully.

Possible way to track changes in UI:

• Analysis of changes before fix.
• Code Review.
• Automated UI testing.
• Manual UI testing.

3. Breaking Changes in Configuration

The general rule in relation to changes in configuration is "Do not change the configuration."

The changes in configuration are almost always done manually and bring additional headache for the customers who are doing an upgrade.

In case you do need to make changes in configuration files, the only allowed changes are:

• Changes in comments :).
• Adding new nodes (preferably at the end of the node set).
• Adding new attributes (preferably at the end of the node set).

Any other changes, like changing the order of the nodes, renaming nodes, changing node or attribute values, etc., will change the node xPath which means that this can potentially break the custom logic or custom patch files.

The check for changes in configuration can be automated by comparing with configuration from previous version.

Another good indicator for breaking changes in configuration is upgrade procedure. In two words - updating the configuration files during the upgrading between Updates should be optional, everything must be working at least not worth as it was before the upgrade. If you see any exceptions related to configuration after the upgrade this is an indicator that something has been done in a wrong way.

The rule: situation when necessary configuration is missing is always possible. You should have a plan "B" and use fall-backs and default values.

4. Breaking Changes in Database schema

For the changes in database schema we should use the same rule to the one we have for changes in configuration - "Avoid of making any changes in schema for minor updates".

General requirements are:

• New API should be able to work with old schema, or in a different words - updating the schema in minor updates is not mandatory.
• Applying new schema must never break or even modify existing customer data.

Changes in indexes, stored procedures are also considered as breaking since they could already have customization.

There are some tools that could be used for automated check for changes in database. One of them called "DBComparer" but there might be a lot of other that could be much better :).

5. Breaking Changes in data, stored in database.

It is very difficult to say anything concrete here since guidelines might depend on the data you have in database. For example: in case you have any template data, that could be used by your customers to create own specific content, you should ensure that once you change your template you will not break all content created by customer. You should always think about custom data once decide to change anything in your data. New API should always be compatible with data from previous version.

Summary

Please, do not forget about customers when making changes to your system. Tracking of breaking changes is not an easy thing but your customers are really appreciate this. In case you do have to introduce breaking changes, please make it visible to a customer (highlight in Release Notes, put a warning\erorr to a log, show message, etc.) and ensure the instructions for further steps are clear and easy.

Try to automate the process for identifying breaking changes since fixing them at the late stage is very difficult or even impossible at all.