Concurrent Update of SAP Commerce Cloud Item Relations

As you know, all interactions with the database in SAP Commerce Cloud is performed via the ORM layer called Persistence Layer. This layer has an API for developers so that they can perform create/read/update/delete operations with the objects using the database-agnostic language, and the Persistence layer translates it to the database queries.

This layer is able to detect and store the changes in the objects in a smart way so that the objects are only updated if there are unsaved changes in them. Also, this component allows a cascade update which helps you to store the changes in the several objects linked with each other with a single save command.

For example, if you create a new product, and a new category for a product, and then link a category to a product, and then save the product, the category will be created in the database automatically along with the product.

However, this mechanism has a flaw: when the updated are performed in multiple threads or in the clustered environment in multiple processes, it is not concurrent safe with the “out-of-the-box” setup.

SAP Commerce Cloud platform provides optimistic locking through a version property (hjmpTS) on your persistent items. This property is automatically managed by the platform. It is incremented each time you update the item and there are no conflicts.

The conflicts happen if some other process or thread tries to update the item as well. The first attempting submit an update wins, all the rest are rejected. This is how optimistic locking works in general, and in SAP Commerce Cloud in particular.

However, in many business scenarios, the second attempt should not be rejected automatically just based on the fact it changes the same item. This second (third, fourth, etc.) attempt may bring the latest changes which should not be ignored. The normal way is to queue these changes so that they are applied in the original order, but sequentially.

For a single-node use, there is a synchronized keyword in Java. However, for a multi-server setup, it is not enough. For the clustered environments, the nodes may use the shared storage to keep the information on what objects are locked to perform these update operations in sync. The database or NoSQL storage may be such storage too. There are mechanisms for the item update locking in MySQL and Oracle. 

However, the Commerce Cloud ORM is “too smart” and performs database writes that are not directly related to the objects we change. These writes are classified as “updates”, “removes”, and “inserts”.

“Updates” is the most problematic operation. Other than we can’t often ignore the concurrent updates, there are cascaded updates we can’t skip too.

But what is more problematic is cascade updates where the relations are involved. For example, you save a product item with changed supercategories.

Let’s say you have ONE CATEGORY and ONE THOUSAND PRODUCTS. All products are linked to the same category. For this, hybris creates one thousand records in the Cat2ProdRel table.

If you modify an existing product, and the attribute you modify is not a category, the system won’t touch the category object, so you can take care of product-related concurrency. However, if you create a new product (let’s say Product 1001), the system inserts a new item into Cat2ProdRel – it is safe because it is an insert operation, concurrent-safe.

However, SAP Commerce Cloud updates last modification time for objects involved in a relation, namely Category in this example. It is so because Category is a shared object. If you do 1000 product inserts per minute, the system should also make 1000 category updates per minute, and all these updates are basically unnecessary.

Another example is CategoryModel.setProducts(products) which triggers an update statement for each product in the set to update its last modified time.

When used in a multi-threaded or multi-process environment, it results in optimistic locking failure exceptions messages in the log.

ROOT CAUSE

For the many-to-many relations, like Product-to-Category, SAP Commerce Cloud updates the last modification time of both objects involved when the relation is updated. The class responsible for this is de.hybris.platform.persistence.links.jdbc.dml.relation.UnorderedRelation (or OrderedRelation for ordered). 

SOLUTION

The solution is simple: disabling updating last modified time for the relation. For the Product-to-Category relation this configuration parameter is:

relation.CategoryProductRelation.markmodified=false

POSSIBLE SIDE EFFECTS

Some system components, such as catalog synchronization or Solr indexing, may require the last modified time to know which items to process.