Hybris WMS/ERP Integration: Stock Levels and Reserves
IntroductionThe inventory management is one of the challenging topics in e-commerce. As a part of my research, I came up with some best practices in this area. In this article, you can find information about the inaccurate inventory records and inventory balance discrepancies as well as techniques to compensate the inventory record errors and stock update delays. I explain the details on inventory data importing: integration interfaces, data guaranteed delivery, failover and error handling, mass update specifics. I am going to go over the architecture of the lean stock update that could be useful for marketplaces with millions of products and distributed warehouses. You find out the details about data anomalies and how to process products with multiple product units, using the stock history, and rule-based product availability. Also I’ll provide a large topic about the stock management for different types of product bundles.
Inaccurate inventory recordsThis is an all-too-common problem when inventory data isn’t accurate. Outdated, damaged or spoiled inventory, wrong labels, lost items, delayed stock updates and other inventory-related issues occur everywhere. Inaccuracy takes shape in many forms, such as inaccurate quantities, inaccurate storage locations, inaccurate pricing, and inaccurate identification. This happens very often with retail companies, especially big-box retailers that have millions of SKUs. There are various techniques you can use to compensate for inventory record errors. Two of them are widely used in e-commerce:
- Safety stock. Safety stock is often used as a protection against uncertainties in variables in inventory operations, such as the demand and supplier lead time. It can be extended to serve as a buffer against uncertainty in the inventory record.
- Constant Decrement of the Inventory Record. If you are aware of the presence of stock loss and also know its stochastic behavior, another way to compensate for the error is exporting the decremented value of the inventory records to hybris. How much? You can decrement by the average stock loss demand each period. Since the actual value of the stock loss at each period is unknown, simply decrementing the record will still not eliminate the error in the inventory record. However, over time, this corrective action can be expected to perform better than leaving the inventory record unadjusted. You can use average stock loss classes with different decrements and associate them with product categories.
- Manual stock verification. Once the checkout is completed, the special group of people validates the order, double check the availability, and eventually they confirm, edit or deny the order. These people may contact the customer to go over the details if something is not clear. After the process is over, the system captures the payment (for prepaid orders) and the fulfillment is started. The customer is informed about the order is placed. This approach is costly and old-fashioned, nobody likes it. However, there are millions of e-shops in the world that use this approach at least for the certain product groups.
Product variantsSometimes the same product is available in different styles or types. From the ERP standpoint, these are different items. For customers, all of them are different versions of the same product. Hybris requires linking product variants with each other. There are a base product and optional variant products which have an attribute that points to their base product. In some ERP this grouping is achieved through product categories or the SKU# pattern. Sometimes the WMS knows nothing about variants. For example, cheap rubber bracelets may go in various colors with the same SKU#. It is not a problem for a traditional retail: the customer picks the right color, there no need to have separate SKUs in the WMS. However, for e-commerce website you may want to ask the customer about the color. How will you push this information to the warehouse once the order is placed? In comments? How to determine the stock level for each color? These questions should be asked.
Merging and splitting productsAnother pattern is based on the concept of backup suppliers. These warehouses should be 100% reliable in terms of product availability but at the cost of higher price for the seller. Using backup supplier is certainly a red flag for the procurement department. The last example of merging products is typical for furniture: The same approach is for the bundles that are available when all the components are available.
Inventory balance discrepancyThe e-commerce website is not an only consumer of WMS data. Other systems may update the information in it and the changes may propagate with some delay. Certainly, you need to minimize this delay. However, for various reasons this delay is long enough to have issues with the stock inaccuracy. There are two main types of delays:
- related to the technical constraints: frequency, throughput, memory or CPU limitations
- related to organizational constraints: for example, the stock will be updated only hard copies of the documents are registered in the system
- hybrisProductAvailability = (hybrisStock – hybrisReserved)
- hybrisRealtimeAvailabilityCheck(t) = (WarehouseStock(t) – WarehouseReserve(t))
- Bulk updates. For each item
- Current stock info /per item and warehouse/
- Reserved amount /per item and warehouse/
- Hot current stock info
- Current stock info for the item /per warehouse/
- Reserved amount for the item /per warehouse/
Hybris default integration diagramThis diagram shows the stock-related processes only. Exporting orders isn’t shown here as well as fulfillment-related processes.
Key integration interfaces
Async integration endpoints
- sync order/consignment updates (WMS -> hybris, hybis -> WMS)
- update hybris stock levels (WMS -> hybris)
- create WMS consignment (hybris -> WMS)
Sync integration endpoints
- check availability (hybris -> WMS)
Guaranteed DeliveryAll asynchronous communication from and to hybris should flow through an Enterprise Message Bus. It is important to make messages persistent so that they are not lost even if the messaging system crashes. All synchronous communication from and to hybris is recommended to flow through an API Gateway. Using the unified gateway for all API calls will allow you to monitor throughput and test API usage.
Failover and graceful error handlingI recommend using Netflix Hystrix for manage failures in the data exchange. This framework uses the circuit breaker pattern which will stop further requests if a configurable number of requests starts to fail. This will ensure that the app servers are not endlessly hung up allowing for a graceful error handling.
Mass update with ImpexIf you need to update large amounts of items using ImpEx, I recommend to take into account the following :
- INSERT_UPDATE performs three flexiblesearch queries: two SELECTs and one UPDATE. See one of the previous articles for the details.
- All unique attributes and reference object’s lookup attributes mentioned in ImpEx header as constraints should have a database index for fast lookup.
INSERT_UPDATE StockLevel;available;warehouse(code)[unique=true];inStockStatus(code);productCode[unique=true] ...For this example, let’s check the indexes for the following lookup attributes:
- For Warehouse object: by the
- For Product object; by the
- there is no index for warehouse.code; Full scan; Frequently used -> in cache;
- Product.code is in index;
Lean stock updateAccording to this concept, the WMS updates SOLR rather than hybris. SOLR updates hybris database. Some events may force this update. It will reduce the database load significantly. I used this concept in my hybris Marketplace PoC.
Incremental updatesVery often the external product stock data provider (ERP or WMS) is a bottleneck for the data exchange. Pulling all data is too expensive and slow. Incremental updates allow to make this process more efficient: transfer the changed data only. As a rule, something like a
SOLR and cachingI encourage you to have a look on my article about the hybris Marketplace. I used a set of 2M products, 16K categories and 6000 facets. The numbers are too large to be used with the conventional hybris. It became possible to use these with hybris just because all data are in fast document-oriented SOLR database rather than relatively slow relational database. Availability information is also in SOLR. In another PoC I demonstrated that it takes seconds to update the availability information for large amount of products, warehouses and even customer groups.
Data AnomaliesThere are three types of anomalies: update, deletion and insertion anomalies.
An update anomaly is a data inconsistency that results from data redundancy and a partial update.
Normally, there shouldn’t be any update anomalies, if the system designed properly. See the Inserts section below as well.
A deletion anomaly is the unintended loss of data due to deletion of other data. Don’t remove objects to avoid data inconsistency. Use special statuses to mark items as non-relevant.
An insertion anomaly is the inability to add data to the database due to absence of other data.
For our topic, there is a common issue about the insertion anomaly, as follows:
- There is a new warehouse in ERP, but the warehouse hasn’t been added to the hybris warehouse list yet.
- There is a new product in ERP, but the product hasn’t been added to the hybris product catalog yet.
- There is product availability data (stock levels) which have references to the products and/or warehouses mentioned above.
Historical dataThink about collecting the historical data of all availability data updates in the external NoSQL storage. It is a good source for the further big data processing to implement such features as
- Restock evaluation: “The product is temporary not available now, The estimated arrival date is next week. You can pre-order it”,
- Availability alerts: “Inform me when this item becomes available”.
- Mark products as discontinued. If a product is unavailable for a long time (configured), the system should send a message to administrators about that fact to mark this product as dicontinued (to redirect the customer to to the replacement product).
- System health monitoring: “Is there any anomalies with the product stock updates?” (see the next section)
BundlesThere are four different types of bundles in hybris:
|hybris product catalog||ERP|
|Type 1. Fully virtual||no bundle SKU#||no bundle SKU#|
|Type 2. ERP product grouping||no bundle SKU#||bundle SKU# is present|
|Type 3. Hybris product grouping||bundle SKU# is present||no bundle SKU#|
|Type 4. Bundles are regular products||bundle SKU# is present||bundle SKU# is present|
Type 1. Fully virtual.ERP is not aware of bundles; only of bundles’ components. Hybris doesn’t have bundles in the product catalog. Example: Build your PC product configurator: For this option, the availability of the bundle is based on the availability of bundle components. This option is used when the bundles are supposed to build individually for the customers based on their selection.
Type 2. ERP product groupingLike the previous option, Hybris is not aware of bundles. However, ERP has bundles as a separate SKUs. This pattern is common for the situations when ERP has SKUs for all possible combinations of components generated by hybris-powered product builder (like shown above). Unlike the previous option, the bundles are stock keeping units on the ERP/WMS side. If some bundles are unavailable, you will not be able to order them. In this option, the real-time availability requests are essential because there is no way to store the information about bundle’s availability in hybris. This pattern is good for systems, where ERP bundles are virtual and the absolute number of possible bundles is huge (Otherwise it is recommended to go with Type 4). ERP calculates the bundle availability dynamically, by request, according to some algorithm known to ERP/WMS.
Type 3. Hybris product groupingIn this option, ERP is not aware of bundles. Bundles exist in hybris only. The bundle availability depends on the information from ERP/WMS (component level) and from hybris (component-level and bundle level).
Type 4. Bundles are regular productsIn this option, bundles exist both in ERP/WMS and hybris. Each bundle is a purchasable item with a separate SKU. The set of components of the bundle is just for the reference, the availability information is based on ERP/WMS data for the bundle SKU.
MonitoringCollecting historical data is great for the system health monitoring. If you update your stock every day or every hour, you can create (and continuously update) a typical pattern of the singular update or group of sequential updates. Any deviations from this patterns can be flagged as real-time alerts to admins. It is also a very useful tool for early identification of the issues related to the data exchange.
Multiple unitsIn some cases, handling products in a different unit of measures is necessary. For example, if you buy products in a country where the metric system is of application and sell them in a country where the imperial system is used, you will need to convert the units. Warehouse units and purchase units can be different.
- We buy eggs by trays (30 eggs)
- We check all eggs individually when receiving it (quality control)
- We sell eggs by the dozen to the customers
- We buy wire in coils
- We sell wire by the foot/metre and in coils.
- wire by the foot and
- wire in coils.
Rule-based product availabilitySometimes the product availability should depend on the combination of many factors: Availability = Function ( RawStockData, CL, CS, PA, PPA, PM, DM)
- RawStockData, available items per warehouse
- CL, customer location (for example, some items are unavailable for some areas)
- CS, customer status (for example, some items are available only for gold customers)
- PA, product attributes (for example, a category where product is linked to)
- PPA, product price attributes (for example, low margin products are not available in some areas)
- PM, chosen payment method (for example, some products may not be available for pickup or delivery to distant areas when cash-on-delivery method is chosen)
- DM, chosen delivery method (some products may not be available for chosen delivery method)
© Rauf Aliev, October 2016
4 October 2016 at 06:49
Thanks for the overview, first of all!
Did You consider Hybrise in the role of global APO system – the system which manages all the information on availability as the prime source of information ? I am sure such an approach would produce hundreds less requests between systems + no online requests on checkout.
4 October 2016 at 08:26
In the most cases this approach will not work because e-commerce is not the only channel to sell for the client. However, for some rare cases it’s ok. I didn’t have any projects of the sort before.
4 October 2016 at 08:30