An incremental load is a form of ETL (Extract, Transform, Load) process that involves transferring only the data that has been altered since the last loading session. This approach is both more efficient and quicker than performing a full load because it minimizes the volume of data that needs to be moved and processed. 

However, implementing an incremental load necessitates mechanisms for recognizing and monitoring modifications in the source data, using methods like timestamps, flags, or versioning. Additionally, it must effectively manage issues like conflicts and duplications, for instance, when a single record undergoes multiple updates.

In this article I am focusing on my experience in implementing the optimized data loader integrated into the target system which was SAP Commerce. Simply put, it handles duplicates and minimizes the number of unnecessary database writes. 

When exploring any topic, I tend to focus more on limitations than on capabilities. This is because capabilities are widely promoted—everything from documentation to marketing materials talks about them. But limitations have to be gathered bit by bit.

When I interview developers, I often ask not about what can be done with the e-commerce platform but about what cannot be done easily or what is difficult or what is expensive to implement. Such questions significantly better reveal the potential and capabilities of a candidate.

And now, having recently delved into CommerceTools, I make notes in the process of studying the documentation about what is missing or what requires more serious investment of effort and time than it may seem.

In this article, I share these notes.

In this article, I want to talk about my experience with how to make a website or online store search more responsive to business requirements and user expectations.

It’s common to give too much attention to technical issues, especially now, with the advent of machine learning/AI in this field. But more often than not, the deepest problems are on the surface and their solution is much simpler than a functional refinement.

One of the main challenges in managing e-сommerce websites is ensuring the product search feature works effectively. Imagine this: you notice an issue with how products are being found through search, and you decide to fix it by tweaking some settings. However, altering these settings could unexpectedly impact your business. When we talk about something being ‘broken’, it’s not just about error messages popping up. Rather, it could mean that customers are having a harder time finding what they want, which might lead to fewer sales over time. This is a subtle effect that can spread across your entire platform. But how do you pinpoint the exact cause of such a decline? Testing and monitoring changes in an ecommerce environment are complex because even small modifications can have wide-ranging effects on customer behavior and, consequently, sales performance.

To monitor changes, it’s essential to understand what to collect and how to extract data from the gathered information. Specifically, for e-commerce search, there are no good ready-made monitoring tools available, but there are various DIY kits that address some needs. Such tools suggest collecting and analyzing search results for reference queries and visualizing changes over time. However, this approach falls short in many respects. For instance, it only covers the most frequent queries.

When it comes to modernizing site search, there’s a big question about whether this is done from scratch or if it’s possible to use accumulated data from existing sessions. Developing from scratch is always a risk—because any assumptions you have about how users will search for content may be wrong. Therefore, regardless of how good the first version of the search turned out to be, it will need to be revised and reworked many times.

Today, I want to share the approach that I have used in several projects. I call it Search Analytics. Read More »