An incremental load is a form of ETL (Extract, Transform, Load) process that involves transferring only the data that has been altered since the last loading session. This approach is both more efficient and quicker than performing a full load because it minimizes the volume of data that needs to be moved and processed. 

However, implementing an incremental load necessitates mechanisms for recognizing and monitoring modifications in the source data, using methods like timestamps, flags, or versioning. Additionally, it must effectively manage issues like conflicts and duplications, for instance, when a single record undergoes multiple updates.

In this article I am focusing on my experience in implementing the optimized data loader integrated into the target system which was SAP Commerce. Simply put, it handles duplicates and minimizes the number of unnecessary database writes. 

When exploring any topic, I tend to focus more on limitations than on capabilities. This is because capabilities are widely promoted—everything from documentation to marketing materials talks about them. But limitations have to be gathered bit by bit.

When I interview developers, I often ask not about what can be done with the e-commerce platform but about what cannot be done easily or what is difficult or what is expensive to implement. Such questions significantly better reveal the potential and capabilities of a candidate.

And now, having recently delved into CommerceTools, I make notes in the process of studying the documentation about what is missing or what requires more serious investment of effort and time than it may seem.

In this article, I share these notes.

In this article, I want to talk about my experience with how to make a website or online store search more responsive to business requirements and user expectations.

It’s common to give too much attention to technical issues, especially now, with the advent of machine learning/AI in this field. But more often than not, the deepest problems are on the surface and their solution is much simpler than a functional refinement.

One of the main challenges in managing e-сommerce websites is ensuring the product search feature works effectively. Imagine this: you notice an issue with how products are being found through search, and you decide to fix it by tweaking some settings. However, altering these settings could unexpectedly impact your business. When we talk about something being ‘broken’, it’s not just about error messages popping up. Rather, it could mean that customers are having a harder time finding what they want, which might lead to fewer sales over time. This is a subtle effect that can spread across your entire platform. But how do you pinpoint the exact cause of such a decline? Testing and monitoring changes in an ecommerce environment are complex because even small modifications can have wide-ranging effects on customer behavior and, consequently, sales performance.

To monitor changes, it’s essential to understand what to collect and how to extract data from the gathered information. Specifically, for e-commerce search, there are no good ready-made monitoring tools available, but there are various DIY kits that address some needs. Such tools suggest collecting and analyzing search results for reference queries and visualizing changes over time. However, this approach falls short in many respects. For instance, it only covers the most frequent queries.

When it comes to modernizing site search, there’s a big question about whether this is done from scratch or if it’s possible to use accumulated data from existing sessions. Developing from scratch is always a risk—because any assumptions you have about how users will search for content may be wrong. Therefore, regardless of how good the first version of the search turned out to be, it will need to be revised and reworked many times.

Today, I want to share the approach that I have used in several projects. I call it Search Analytics. Read More »

This article focuses on how SAP Commerce stores models in memory, which is a vast topic. To begin with, we will explore a relatively simple aspect, such as the unusual structure that appears in the debugger when trying to view the contents of any SAP Commerce model. We will also discuss how to access all the object properties contained within it.

There are dozens, even hundreds of test automation frameworks available, and we developed our own. Why? Why? Was it worth the effort? 

This article describes our needs and our solution, E2E CLI: the architecture of the API testing tool which has been used by our team for about 2 years. Back in 2020, it was developed by me from scratch over the course of a weekend — In other words, it is a very simple piece based on very simple ideas. That first version was supporting 80% of the functionality available today in the last version (and described below). We don’t add features without good reason.

We use this tool to end-to-end test all our services and integration flows. Whenever we make changes to a software system, we need to ensure that they do not break what was already working.

As the architect and lead developer, I designed and developed it from the ground up, so, of course, I am very proud of my brainchild. But the field cannot be well seen from within the field. You know, reinventing wheels is not always bad. When you build something on your own, you are in full control over what is being built, what its purpose will be, and when it is finally conceived and executed as you wish. It has been two years since I have encountered any alternative to our solution. 

To reach a wider audience, I found it beneficial to share key concepts hoping to get some feedback from the community and enhance the product.

Technically, the tool has nothing to do with SAP Commerce; it’s for testing. However, our entire team, which uses this tool daily, is focused mainly on SAP Commerce development. And the tool is tailored to be used with SAP Commerce.

While the client (for whose benefit the tool is being used) has authorized me to share high-level details, I cannot disclose names or code. In spite of this, the conceptual view should be useful on its own and might become a good foundation for your own solution.

A shopping cart is the centerpiece of almost every e-commerce website.

In fact, this component is the first thing that makes an e-commerce store different from just an online catalog. Working with issues and challenges related to the cart is something you’ll never forget. 

The Russian novelist Leo Tolstoy, at the opening of his novel Anna Karenina, writes ‘All Happy families resemble one another, but each unhappy family is unhappy in its own way.’ You know, the diversity of cart-related issues I have been observing in the SAP Commerce projects, made me think that Tolstoy meant the developers as well. The diversity of the issues and challenges in this component is so vast that every case is different. However, when it comes to building or optimizing the shopping cart processes, understanding the limitations,  pitfalls, and peculiarities is a huge plus. 

Let’s have a look at how a shopping cart works in SAP Commerce under the hood and how to avoid making mistakes in shopping cart customization.

When the coronavirus outbreak happened in China, it boosted local e-commerce as many stuck at their homes forced to buy their goods online. Sales of fresh food on JD.com reportedly jumped 215% during the 10-day period and counting. The ability to serve a large number of customers is growing every year. The traffic may be doubled in a week. How to make sure your system is capable to serve high traffic demand? In today’s article I’m sharing my experience with Tsung, a high-performance testing framework.

Any SAP Commerce Cloud setup is based on a multi-server setup organized as web cluster. This product comes with clustering support out of the box since very early versions. However, many years in a row, it is still one of the most challenging topics. 

JGroups is an essential component of SAP Commerce Cloud. It is widely used as an embedded solution for distributed messaging and eventing in the cluster. It is hard to say that it is underdocumented or it has some quirks that nobody knows how to encounter. However, almost every large project stumbles with JGroups. This article is aimed to shed some light on the internals of the SAP Commerce Cloud clustering and JGroups.  

What content management system you use is of a much higher importance today than it was years ago. Today’s CMS needs to contend with a multi-device world with a wide variety of tablets, mobiles, laptops, and be architecturally ready for new not-yet-existing channels in the consumer’s universe, such as AR/VR and the Internet of Things.

A CMS typically consists of a frontend part, or a «head», and backend. The headless CMS, as their name says, don’t stick to a single predefined “head”. Theoretically, you can develop your own if it is good for your case. Another name for headless CMS is API-first because this interaction is based on APIs.  

For example, the old hybris CMS Cockpit had both parts built with ZK Framework, which orchestrates both the HTML/Javascript “head” and was the essential part of the CMS backend. These parts were tightly connected and intertwined into a single unit called CMS Cockpit.  Later, SAP announced Smartedit, which has been a big bet and a quantum leap for Commerce Cloud’s content management. Spartacus, the open-source Javascript storefront from SAP, well complemented Smartedit. I published the comprehensive reviews of both products in the blog earlier.

There are a number of headless content management systems that are mature enough to efficiently replace the built-in Smartedit CMS. Among them, I would highlight Contentful, Prismic, DotCMS, Cloud CMS, Amplience, and Contentstack as well-established and innovative products with a clear roadmap and good integration capabilities.

One of my recent research projects was aimed at using Contentstack as a headless content management system for SAP Commerce Cloud. This article uncovers the key highlights of the research.  As a result of that work, I also created a data migration tool for moving existing CMS content from Commerce Cloud to Contentstack.  

What is Contentstack? Contentstack is a SaaS headless CMS with an API-first approach that offers the approach known as Content-as-a-Service. It focuses on managing structured content into data feeds that other applications, such as Commerce Cloud, can consume. You can design a content data model interactively or via the Contentstack API and populate the structures with the content interactively or via the same API.

In this article, I pay special attention to the data migration from Commerce Cloud’s CMS to Contentstack. This exercise helped me to see the complexity in all its glory. Because when you design a system from scratch, you can evade the platform’s constraints by designing the structures compatible only with the new CMS. When you need to migrate data, you have to find a solution on how to adapt concepts and data models to get them compatible with Commerce Cloud.