Introduction

The inventory management is one of the challenging topics in e-commerce. As a part of my research, I came up with some best practices in this area. In this article, you can find information about the inaccurate inventory records and inventory balance discrepancies as well as techniques to compensate the inventory record errors and stock update delays. I explain the details on inventory data importing: integration interfaces, data guaranteed delivery, failover and error handling, mass update specifics. I am going to go over the architecture of the lean stock update that could be useful for marketplaces with millions of products and distributed warehouses. You find out the details about data anomalies and how to process products with multiple product units, using the stock history, and rule-based product availability. Also I’ll provide a large topic about the stock management for different types of product bundles.

Inaccurate inventory records

This is an all-too-common problem when inventory data isn’t accurate. Outdated, damaged or spoiled inventory, wrong labels, lost items, delayed stock updates and other inventory-related issues occur everywhere.   Inaccuracy takes shape in many forms, such as inaccurate quantities, inaccurate storage locations, inaccurate pricing, and inaccurate identification. This happens very often with retail companies, especially big-box retailers that have millions of  SKUs. There are  various techniques you can use to compensate for inventory record errors. Two of them are widely used in e-commerce:

• Safety stock. Safety stock is often used as a protection against uncertainties in variables in inventory operations, such as the demand and supplier lead time. It can be extended to serve as a buffer against uncertainty in the inventory record.
• Constant Decrement of the Inventory Record. If you are aware of the presence of stock loss and also know its stochastic behavior, another way to compensate for the error is exporting the decremented value of the inventory records to hybris. How much? You can decrement by the average stock loss demand each period. Since the actual value of the stock loss at each period is unknown, simply decrementing the record will still not eliminate the error in the inventory record. However, over time, this corrective action can be expected to perform better than leaving the inventory record unadjusted. You can use average stock loss classes with different decrements and associate them with product categories.

• Manual stock verification. Once the checkout is completed, the special group of people validates the order, double check the availability, and eventually they confirm, edit or deny the order. These people may contact the customer to go over the details if something is not clear. After the process is over, the system captures the payment (for prepaid orders) and the fulfillment is started.  The customer is informed about the order is placed.  This approach is costly and old-fashioned, nobody likes it. However, there are millions of e-shops in the world that use this approach at least for the certain product groups.

Product variants

Sometimes the same product is available in different styles or types. From the ERP standpoint, these are different items. For customers, all of them are different versions of the same product. Hybris requires linking product variants with each other. There are a base product and optional variant products which have an attribute that points to their base product. In some ERP this grouping is achieved through product categories or the SKU# pattern. Sometimes the WMS knows nothing about variants. For example, cheap rubber bracelets may go in various colors with the same SKU#.  It is not a problem for a traditional retail: the customer picks the right color, there no need to have separate SKUs in the WMS. However, for e-commerce website you may want to ask the customer about the color. How will you push this information to the warehouse once the order is placed? In comments? How to determine the stock level for each color? These questions should be asked.

Merging and splitting products

Another pattern is based on the concept of backup suppliers. These warehouses should be 100% reliable in terms of product availability but at the cost of higher price for the seller. Using backup supplier is certainly a red flag for the procurement department. The last example of merging products is typical for furniture: The same approach is for the bundles that are available when all the components are available.

Inventory balance discrepancy

The e-commerce website is not an only consumer of WMS data. Other systems may update the information in it and the changes may propagate with some delay. Certainly, you need to minimize this delay. However, for various reasons this delay is long enough to have issues with the stock inaccuracy. There are two main types of delays:

• related to the technical constraints: frequency, throughput, memory or CPU limitations
• related to organizational constraints: for example, the stock will be updated only hard copies of the documents are registered in the system

In the example below “Warehouse reserve” is an and “hybris reserved” is an    (warehouse reserve) is tightly linked with the order management processes in WMS. Usually it takes some time to process the update. It might be a technical delay or a workflow that comprises a set of steps for validating the order and  process the reserves.  The initial state of the stock is 500 items. As we see, 3 items are still being processed on WMS side and another three items were reserved by hybris. We initially had 500 items, 10 items were sold, but at this moment hybris stock levels service shows 497 items. This value is more that the real stock has. It is a reason why you need to send a synchronize request to WMS from the shopping cart. However, the warehouse shows 494 that is also more than 490. In order to mitigate the discrepancy, you need to use WMS unprocessed requests in the hybris availability calculation.

• hybrisProductAvailability = (hybrisStock – hybrisReserved)
• hybrisRealtimeAvailabilityCheck(t) = (WarehouseStock(t) – WarehouseReserve(t))

hybrisRealtimeAvailabilityCheck(t) is mainly used in the shopping cart and in the checkout process. hybrisProductAvailability is mainly used for SOLR indexing and product pages. Let’s place another order, #2. There are two products in it. If we accept the order of 494 or 489 products, you won’t be able to fulfill the order, because the stock has only 488 items. To summarize it, you need two interfaces to your WMS service:

• Bulk updates. For each item
  • Current stock info /per item and warehouse/
  • Reserved amount /per item and warehouse/
• Hot current stock info
  • Current stock info for the item /per warehouse/
  • Reserved amount for the item /per warehouse/

Integration details

Hybris default integration diagram

This diagram shows the stock-related processes only. Exporting orders isn’t shown here as well as fulfillment-related processes.

Key integration interfaces

Async integration endpoints

• sync order/consignment updates (WMS -> hybris, hybis -> WMS)
• update hybris stock levels (WMS -> hybris)
• create WMS consignment (hybris -> WMS)

Sync integration endpoints

• check availability (hybris -> WMS)

Guaranteed Delivery

All asynchronous communication from and to hybris should flow through an Enterprise Message Bus.  It is important to make messages persistent so that they are not lost even if the messaging system crashes. All synchronous communication from and to hybris is recommended to flow through an API Gateway. Using the unified gateway for all API calls will allow you to monitor throughput and test API usage.

Failover and graceful error handling

I recommend using Netflix Hystrix for manage failures in the data exchange. This framework uses the circuit breaker pattern which will stop further requests if a configurable number of requests starts to fail. This will ensure that the app servers are not endlessly hung up allowing for a graceful error handling.

Mass update with Impex

If you need to update large amounts of items using ImpEx, I recommend to take into account the following :

• INSERT_UPDATE performs three flexiblesearch queries: two SELECTs and one UPDATE. See one of the previous articles for the details.
• All unique attributes and reference object’s lookup attributes mentioned in ImpEx header as constraints should have a database index for fast lookup.

For example:

INSERT_UPDATE StockLevel;available;warehouse(code)[unique=true];inStockStatus(code);productCode[unique=true]
...

For this example, let’s check the indexes for the following lookup attributes:

• For Warehouse object: by the
  code
   attribute,
• For Product object; by the
  code
   attribute.

If you don’t have the significant number of warehouses, reading sequentially is faster than working through an index. Let’s look at the OOTB hybris configuration:

• there is no index for warehouse.code; Full scan; Frequently used -> in cache;
• Product.code is in index;

Lean stock update

According to this concept, the WMS updates SOLR rather than hybris. SOLR updates hybris database. Some events may force this update. It will reduce the database load significantly. I used this concept in my hybris Marketplace PoC.

Incremental updates

Very often the external product stock data provider (ERP or WMS) is a bottleneck for the data exchange. Pulling all data is too expensive and slow. Incremental updates allow to make this process more efficient: transfer the changed data only. As a rule, something like a attribute is available in the source system.  Using this attribute as a filter will significantly reduce the server load and increase the performance.

SOLR and caching

I encourage you to have a look on my article about the hybris Marketplace. I used a set of 2M products, 16K categories and 6000 facets. The numbers are too large to be used with the conventional hybris. It became possible to use these with hybris just because all data are in fast document-oriented SOLR database rather than relatively slow relational database. Availability information is also in SOLR. In another PoC I demonstrated that it takes seconds to update the availability information for large amount of products, warehouses and even customer groups.

Data Anomalies

There are three types of anomalies: update, deletion and insertion anomalies.

Updates

An update anomaly is a data inconsistency that results from data redundancy and a partial update.

Normally, there shouldn’t be any update anomalies, if the system designed properly. See the Inserts section below as well.

Deletes

A deletion anomaly is the unintended loss of data due to deletion of other data. Don’t remove objects to avoid data inconsistency. Use special statuses to mark items as non-relevant.

Inserts

An insertion anomaly is the inability to add data to the database due to absence of other data.

For our topic, there is a common issue about the insertion anomaly, as follows:

• There is a new warehouse in ERP, but the warehouse hasn’t been added to the hybris warehouse list yet.
• There is a new product in ERP, but the product hasn’t been added to the hybris product catalog yet.
• There is product availability data (stock levels) which have references to the products and/or warehouses mentioned above.

As a result, the stock update may end up with errors. It is clear that these situations should be planned well beforehand. Certainly, you need to insert the stock levels items only if all referencing objects are in place. However, you need to check it for each StockLevel item imported from WMS that is not good for performance. I recommend to pre-process the incoming data and split them into two parts:

update

 and

insert

.   Products and warehouses should be cached in memory to make the pre-processing faster. The Stocklevel items which were sorted out for insertion should be processed if all the referenced components are found (product#, warehouse).

Historical data

Think about collecting the historical data of all availability data updates in the external NoSQL storage. It is a good source for the further big data processing to implement such features as

• Restock evaluation: “The product is temporary not available now, The estimated arrival date is next week. You can pre-order it”,
• Availability alerts: “Inform me when this item becomes available”.
• Mark products as discontinued. If a product is unavailable for a long time (configured), the system should send a message to administrators about that fact to mark this product as dicontinued (to redirect the customer to to the replacement product).
• System health monitoring: “Is there any anomalies with the product stock updates?” (see the next section)

Also, the collected information is great for troubleshooting.

Bundles

There are four different types of bundles in hybris:

	hybris product catalog	ERP
Type 1. Fully virtual	no bundle SKU#	no bundle SKU#
Type 2. ERP product grouping	no bundle SKU#	bundle SKU# is present
Type 3. Hybris product grouping	bundle SKU# is present	no bundle SKU#
Type 4. Bundles are regular products	bundle SKU# is present	bundle SKU# is present

Type 1. Fully virtual. 

ERP is not aware of bundles; only of bundles’ components. Hybris doesn’t have bundles in the product catalog. Example: Build your PC product configurator: For this option, the availability of the bundle is based on the availability of bundle components. This option is used when the bundles are supposed to build individually for the customers based on their selection.

Type 2. ERP product grouping

Like the previous option, Hybris is not aware of bundles. However, ERP has bundles as a separate SKUs. This pattern is common for the situations when ERP has SKUs for all possible combinations of components generated by hybris-powered product builder (like shown above).  Unlike the previous option, the bundles are stock keeping units on the ERP/WMS side. If some bundles are unavailable, you will not be able to order them. In this option, the real-time availability requests are essential because there is no way to store the information about bundle’s availability in hybris. This pattern is good for systems, where ERP bundles are virtual and the absolute number of possible bundles is huge (Otherwise it is recommended to go with Type 4). ERP calculates the bundle availability dynamically, by request, according to some algorithm known to ERP/WMS.

Type 3. Hybris product grouping

In this option, ERP is not aware of bundles. Bundles exist in hybris only. The bundle availability depends on the information from ERP/WMS (component level) and from hybris (component-level and bundle level).

Type 4. Bundles are regular products

In this option, bundles exist both in ERP/WMS and hybris. Each bundle is a purchasable item with a separate SKU. The set of components of the bundle is just for the reference, the availability information is based on ERP/WMS data for the bundle SKU.

Monitoring

Collecting historical data is great for the system health monitoring. If you update your stock every day or every hour, you can create (and continuously update) a typical pattern of the singular update or group of sequential updates. Any deviations from this patterns can be flagged as real-time alerts to admins. It is also a very useful tool for early identification of the issues related to the data exchange.

Multiple units

In some cases, handling products in a different unit of measures is necessary. For example, if you buy products in a country where the metric system is of application and sell them in a country where the imperial system is used, you will need to convert the units. Warehouse units and purchase units can be different.

• We buy eggs by trays (30 eggs)
• We check all eggs individually when receiving it (quality control)
• We sell eggs by the dozen to the customers

For furniture, a kitchen set can consist of the several components. Some of these components are purchasable separately, but some of them can be sold only as a part of the other units. In addition to that, some components are common for different kitchen sets. How to determine a number of available kitchen sets by the known number and types of components?

• We buy wire in coils
• We sell wire by the foot/metre and in coils.

For the last example,  in ERP we deal with coils, but for the end customer, two different types of products are available:

• wire by the foot and
• wire in coils.

These products share the same stock items. From the customer’s perspective, these products are basically variants of one product, “wire”. In addition to that, there is one more challenging thing: you can’t connect two cuts to have a new longer cut. So there is no easy way to calculate the available stock for such products. So if you have 100 feet of wire, and your customer wants 90, you will possibly have difficulties with the remaining 10-feet cut. However, that is the topic for the separate article. Another example of products with multiple units is soap. Soap products can be sold in bars or by weight. When soap is sold in bulk, the customer is asked about the weight of the piece.  The same situation is with liquids and granular materials that are common for construction material B2B e-commerce websites.

Rule-based product availability

Sometimes the product availability should depend on the combination of many factors: Availability = Function ( RawStockData, CL, CS, PA, PPA, PM, DM)

• RawStockData, available items per warehouse
• CL, customer location (for example, some items are unavailable for some areas)
• CS, customer status (for example, some items are available only for gold customers)
• PA, product attributes (for example, a category where product is linked to)
• PPA, product price attributes (for example, low margin products are not available in some areas)
• PM, chosen payment method (for example, some products may not be available for pickup or delivery to distant areas when cash-on-delivery method is chosen)
• DM, chosen delivery method (some products may not be available for chosen delivery method)

The best way to implement product availability rules is using Drools (included into hybris 6.x). RawStockData can be wrapped into the Drools Globals methods. The remaining listed components can be implemented as Drools Facts. There are different ways to write rules. The hybris rule builder can be leveraged for this purpose as well. The results can be also cached for the faster lookup.

© Rauf Aliev, October 2016