I want to focus on a requirement I have met on several projects over the years where the merchant has multiple instances of SAP-Commerce (hybris) and they want their non-production environments to be “more representative” of the production environment in terms of the data.

Many merchants have 3 such environments typically defined as development, staging and production and as time passes by, the production environment receives multiple BAU changes to core data areas such as web content, products/categories/classifications undertaken by merchandising teams or product management and of course a lot of data is generated through customer registrations and orders placed.

However, the non-production environments are frequently not kept in line in terms of web content, product data or representation in terms of quantities of customers or orders.  I’m not referring here to the code and core configuration related data that is included in builds.

This is a 2-part article:

In this part I look at the rationale of why data migration is important and the various initiatives that have endeavoured to support this over the years;

Part 2 to where I will look at the areas of data to consider for migration and the necessary concerns when migrating data.

 

Rationale of Data Migration

Merchants can understandably want their staging and possibly the development environments to be representative of production because these systems (especially staging) are frequently used for internal testing, showcasing and training and various other what-if situations that they would not necessarily want to just try or undertake on the production system.

With some clients and projects, this requirement may extend further in case there are more that the regular three environments (such as a dedicated environment for performance testing) or where  some of the data is needed for the developers themselves on their own local environments.

Before the SAP Commerce Cloud services, merchants were typically responsible for hosting their own systems and environments whether they were on-premise or hosted by service/cloud providers.

Now, we see that SAP Commerce provides typically three environments – their assignments/names are customer specific, but regularly called production, Staging and Development as seen in the screen shot below.

Fig. 1 – Example Commerce Cloud Portal showing three environments:

Why is Data Migration Important ? 

There are a number of reasons to have production-representative data migrated to other environments.

The dev/staging environments would be more realistic:

The users of the environment will see this realism from two directions:

• New features that are being added as part of new development work for test and training purposes will be clearly distinguishable against a familiar background giving a near-complete representation of what those features will look like or function on the production environment;
• Web content and products that have been manually created and implemented on the production environment as part of merchandising or product management will be reflected back to the staging and development environments. Examples could include new web content pages or imagery, new promotions, new categories, menus etc etc. Of course, it could be argued that any such changes to web content could be at first applied to staging, but some clients will argue that their business users do not have bandwidth to duplicate their work.

Integrations are conveying more realistic data:

Where the dev or staging environment is subject to integration testing, the data being conveyed in this testing is representative of production. Examples could include data being retrieved by a headless front end such as Spartacus, integrations of data to downstream systems such as an order management system, reporting services. Data that is more representative to production data will assist in validating non-functional requirements such as attribute validations, performance and error detection and using data that is new – such as recently added products.

Issues may be found during testing before a build is committed to production:

If the dev or staging environments are subject to rigorous testing then those tests are more likely to have a higher confidence level if the data is representative of production. This could be something simple such as the layout of product information in a PDP page, or more complex if new products are subjected to new promotional calculations.

New members of the development team:

These guys need to get the data in their local SAP Commerce systems somewhat representative of production but not usually to the size and scale of the production data volumes. They therefore need a suitable subset of the data to work with. Ideally, this data should be included in impex files development environment is initialized. However,  I have seen cases where dev teams are swapping copies of their local databases and media folders to get up to speed quicker and this shortcut approach can lead to situations where a local system will not initialize correctly.

 

What initiatives have there been to support data migration ?

There is no straightforward out of the box solution to this requirement ever since my experience with SAP Commerce (hybris) since hybris version 4.1 in 2010. However, historically this has clearly been a subject of much discussion and there have been a number of approaches to meet the requirement over the years. There are still a number of un-answered questions in the SAP help pages raised by various hybris developers, these are questions that have either migrated from the old Hybris.com wiki or raised anew on the SAP site. However, I have found 4 approaches that have been presented by either SAP or by the hybris organization before then:

CMS Migration Impex Scripts:

I remember an article on the old hybris Wiki “How to Export CMS Content” which discussed and gave an impex script to support CMS component transfer. Sadly, that wiki is no longer around, but it is possible to find some of the old wiki pages that SAP added to their site after they acquired the hybris business in 2013. I even checked on the Internet Archive Wayback Machine but because the old hybris wiki needed authentication to be fully accessed, that page is not preserved there either. In any case, I did find (for your curiosity) an export script that was originally designed for the old CMS (for hybris version 3) and that script was revised for version 4 and 5 but I could never get the approach to work properly. Here are those scripts that hybris published and you can see how the principal remains the same in both, but the feature richness added to over the years to content management has made this a much more complex process:

A sample of the script is shown below, and you can download the full script here (size is 3kB): 

Fig. 2 – Sample of Migration Script for Hybris 3

 

A sample of the script as revised for Hybris 5 and 5  is shown below, and you can download the full script here (size is 38kB): 

Fig. 3 – Sample of Migration Script for Hybris 4 and 5

 

HMC / Backoffice Export:

From as far back as hybris 4.0 the Hybris Management Console (HMC) has had an export facility to export data and media. This is a similar principle to option 1 above only it deals with media as well which are exported into a zip file. This facility is still available in latest releases of SAP Commerce Backoffice. There is a related feature that will generate an Impex template for all the data types in your system which significantly helps in defining impex definitions, but the default generated script needs substantial editing attention to remove many of the data type attributes that cannot be modified such as timestamps. Here it is in Hybris 4.2.2 that I have running in a linux virtual machine (don’t you just miss that old HMC ?):

Fig. 4 – CSV Export Wizard Feature in the Hybris 4 Management Console (HMC):

 

And below is the SAP Commerce 2205 Backoffice export facility:

Fig. 5 – Impex Export  Wizard Feature in the SAP Commerce Admin Backoffice:

 

Y2Ysync facility

This is still documented on the SAP Help Portal for versions up to 1808 (https://help.sap.com/docs/SAP_COMMERCE/d0224eca81e249cb821f2cdf45a82ace/8c8fb1468669101485da9c4d292cb103.html?locale=en-US&version=1811 ) is a reference to the SAP Commerce-to-SAP Commerce synchronization transferring data from a source system to a target system using the Data Hub in between. The y2ysync framework claimed continued performance of the SAP Commerce system during the synchronization which divided into independent stages comprising transfer of data to the Data Hub, and from the Data Hub to the target system. This is now however deprecated since SAP Commerce 1808.

Fig. 6 – Y2YSync Facility Data Flow:

As far as I know, there is no replacement for this citing deprecation because of “Dependency on deprecated parts of the product” since the Data Hub itself is now deprecated.

 

SAP Data Migration toolkit

I’ve included this one to avoid confusion as this is not a suitable tool for the subject of this article. This tool is intended for migration of an on-premise system database into a CCV2 system and not for between CCV2 systems. In case you find this tool, it used to be only available to SAP employees but is now published on github : https://github.com/SAP-samples/commerce-migration-toolkit including a user guide. This toolkit adds menus to the hybris admin console (HAC) which you then build into the CCV2 environment. You use those new HAC menus to setup integration attributes to the on-premise source database and it then fetches the data from that database and populates into the CCV2 database. You then follow up with an “update systems” build.

You could be forgiven to ask why you could therefore not deploy this in (say) a staging environment and pull data from the production environment. The reason why not is because in CCV2 environments you don’t have direct access to the database. However, the approach of having customized menus in HAC is attractive for such operations to be conducted without recourse to other complex and more manual procedure. SAP have also provided supporting videos: https://sapvideoa35699dc5.hana.ondemand.com/?entry_id=1_gxduwrl3

And for the media migration also: https://www.sap.com/cxworks/article/2589632453/migrate_to_sap_commerce_cloud_migrate_media_with_azcopy

 

Fig. 7 – Hybris Admin Console with Additional Menu Added from the Data Migration Toolkit

In addition, Rauf gave some good advice in 2018 in a HybrisMart page as a useful reference point for data exporting.

https://hybrismart.com/2018/03/18/cloning-catalogs-export-and-import-whole-catalogs/  

 

To Follow in Part 2

In the next concluding part article, I look at the areas of data to consider for migration and the necessary concerns when migrating data and draw some conclusions from this subject area.

Stay tuned!