Mastering hybris Impex: Recipes and Secrets

In this article, I am going to tell you about undocumented or under-documented capabilities of the ImpEx Engine.

CSV Format

The documentation says that values should be provided as Comma-Separated Values (CSV). SAP refers to RFC 4180 as a specification used in hybris.

However, the format specification is different for ImpEx headers and data blocks:

• You can use “\” to show that the next line is a continuation of the current line.
• You can use line breaks anywhere, but you need to quote the value, both in the header and data block.

Multi-line support

ImpEx extension provides the multi-line separator symbol for breaking lines. By default, this symbol is “\”. This symbol can’t be used in the data block as a multi-line separator. The following ImpEx is valid:

Don’t write it this way; it is bad style. However, it demonstrates that the hybris CSV parser ignores line breaks for quoted strings. By default, the multi-line separator is not used in the data block.

You can use that to create fancy-looking ImpEx files. The following example is not perfect, but it is valid and clearly demonstrates the idea:

There are two drawbacks:

• Backslashes make CSV invalid. The file is still readable by hybris, but CSV editors won’t be able to work with it.
• Leading spaces in each line in the green block will be written into the database. For the HTML fragment, the spaces don’t make any sense, but sometimes they do matter.

Removing all data/all records from the item type

The following ImpEx removes all data from TestItem (replace TestItem with your type):

$item=TestItem
REMOVE $item[batchmode=true];itemtype(code)[unique=true]
;$item

Conditional ImpEx

Some data fragments can be marked as conditional:

$var=value
...
INSERT_UPDATE TestItem;string[unique=true];integer
"#% if: ""$var"".equals(""value"")"
"key";5
#% endif:

In a similar way, you can execute fragments depending on configuration variables:

#% if: !"responsive".equalsIgnoreCase(Config.getParameter("commerceservices.default.desktop.ui.experience"));
...
#% endif:

Accessing Beans

Below is an example of accessing one of the standard hybris beans, flexibleSearch.

....
"#%
import  de.hybris.platform.servicelayer.search.FlexibleSearchQuery;
flexibleSearchService = Registry.getApplicationContext().getBean(""flexibleSearchService"");"

"#%  
query = ""SELECT...."";
flexibleSearchQuery = new FlexibleSearchQuery(query);
resultIterator = flexibleSearchService.search(flexibleSearchQuery).getResult().iterator();"

ImpEx object

There is an object used in many examples below. This object is an instance of ImpExImportReader. The following methods are used in ImpEx scripts more or less frequently:

• impex.setCurrentHeader(HeaderDescriptor)
  impex.setCurrentHeader(String)
  These methods can be used to replace the current header. For example, you can conditionally use INSERT or UPDATE modes depending on the data.
• impex.setValueLinesToSkip(int). How many value lines should be skipped.
• impex.getLastImportedItem(). Returns the last imported item of the Item type.
• impex.isSecondPass(). Checks if CSV data is parsed again to resolve previously unresolvable item references.
• impex.setCreateHMCSavedValues(boolean). Enables/disables the automatic update of item history data for created/removed/modified items.
• impex.discardNextLine(). The next line won’t be processed.
• impex.enableExternalSyntaxParsing(boolean). If false, ImpEx syntax in external data, such as headers, won’t be interpreted, and the lines will be interpreted as value lines.
• impex.includeSQLData(url,user,password,JDBCdriver,sql). Initializes parameters and fires an SQL query against the database.

#% impex.enableCodeExecution( true );
#% impex.setRelaxedMode( true );
INSERT_UPDATE Product; code[unique=true]; name
#% impex.includeSQLData("jdbc:mysql://localhost/mydb?user=rauf&password=raufpassword", "com.mysql.jdbc.Driver", "Select code,name from SomeTable");

• impex.includeSQLData(sql). The alternative way of firing a statement. To use with initDatabase(..).
• impex.includeSQLData(sql, int skip, int offset). Offset is the number of columns to skip from the left. Skip is the number of resulting lines to skip.
• impex.initDatabase(url,user,password,JDBCdriver). Initializes parameters to use with includeDatabase. It is useful when you have more than one includeSQLData in the ImpEx file.
• impex.includeExternalData(file, encoding, lineToSkip). Reads values from the file.
• impex.includeExternalDataMedia(code).

Example from the “bmecat” hybris module:

#% if: impex.findExternalDataMedia("regions.csv")!=null
"#% impex.info("" Creating regions"")"
INSERT_UPDATE REGION; country(isocode); isocode[unique=true]; active[virtual=true, default=true];
#% impex.includeExternalDataMedia("regions.csv");
#% endif:

• impex.includeExternalDataMedia(code, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, encoding, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, encoding, delimiter, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, columnOffset).
• impex.getCurrentLocation(). Returns the current line number.
• impex.insertLine(csvLine). Inserts a line into the parsing stream; it will be the next line processed.
• impex.addDefinition(definition). Creates/modifies a macro. The definition may look like “$a=33”.
• impex.setValidationMode(mode). Sets up a validation mode. The following modes are supported: import_strict, import_relaxed, export_only, export_reimport_relaxed, export_reimport_strict.
• impex.setRelaxedMode(). The same as setValidationMode("import_relaxed").
• impex.debug(msg). Shows a debug message in the console. Internally, it triggers log.debug(msg), which means that your message will be shown only if you have enabled debug mode in the configuration.
• impex.info(msg). Shows a debug message in the console (log.info).
• impex.warn(msg). Shows a debug message in the console (log.warn).
• impex.error(msg). Shows a debug message in the console (log.error).

Setting up ImpEx variables from BeanShell code

The following fragment demonstrates how variables can be set up dynamically. The text part of the addDefinition method can be constructed using the data from bean method calls (see Accessing Beans).

"#% impex.getReader().addDefinition(""$a=33"")";

Using variables in the code

To access ImpEx variables from ImpEx BeanShell code, just put the variable name into the BeanShell code. The ImpEx engine replaces all macros before running your code, so just:

$cronjob=abcd
#% java.lang.System.out.println("$cronjob");

Mapping all configuration variables into ImpEx macros

UPDATE GenericItem[processor=de.hybris.platform.commerceservices.impex.impl.ConfigPropertyImportProcessor];pk[unique=true]

$variable=$config-your.config.property

INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$variable

# OR you can just directly use the config macro
INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$config-your.config.property

Importing a property with a map

INSERT_UPDATE BTGConfig[unique=true];\
               defaultTimeUnit(code);\
                 usedRuleTypes(code);\
                    operandMapping(key(code),value(code))[map-delimiter=|]
BTGConfig;     WEEK;\
                 ORDER,CART,USER,WCMS;\
                    ORDER->BTGOrganizationTotalSpentInCurrencyRelativeDatesOperand,\
                           BTGOrganizationTotalSpentInCurrencyLastYearOperand,\
                           BTGNumberOfOrdersAboveAmountRelativeDateOperand,\
                           BTGCategoriesInOrdersOperand,BTGEachOrderTotalSumOperand,\
                           BTGOrderTotalSumOperand,\
                           BTGProductsInOrdersOperand,\
                           BTGNumberOfOrdersOperand,
                           BTGNumberOfOrdersRelativeDateOperand,
                           BTGLastOrderDateOperand\
                           |\
                     CART->BTGCartIsEmptyOperand,\
                           BTGCartTotalSumOperand,\
                           BTGCategoriesInCartOperand,\
                           BTGProductsInCartOperand,\
                           BTGQuantityOfProductInCartOperand\
                           |\
                     WCMS->BTGViewedProductsOperand,\
                           BTGViewedCategoriesOperand,\
                           BTGVisitedContentpagesOperand,\
                           BTGReferalUrlOperand,BTGUrlParameterOperand\
                           \|
                     USER->BTGOrganizationUIDOfUserOperand,BTGUserAddressPostalCodeOperand,BTGUserCountryOperand,BTGUserGenderOperand,BTGUserMemberOfGroupsOperand|SCRIPT->BTGMediaScriptOperand,\
                           BTGStringScriptOperand

The last column contains the values for a property defined as a map.

Importing two objects which depend on each other with mandatory fields

Suppose that the following relation is defined:

The following ImpEx adds a user and a book at the same time:

INSERT_UPDATE Author;code[unique=true];book(code);
#% impex.setValidationMode("import_relaxed");
;author1;Book1

INSERT_UPDATE Book;code[unique=true];name;author(code)
;Book1;BookName;author1

Relaxing the validation means that Hybris doesn’t check the mandatory fields.

Importing data from an external data source/SQL database

"#%
impex.initDatabase(""jdbc:mysql://localhost/testdb?user=testuser&password=testpass"",""com.mysql.jdbc.Driver"");
impex.setRelaxedMode(false);
impex.includeSQLData("Select * from Products");"""

Importing language versions of the content in separate lines

#% import java.lang.String;
#% import java.lang.Integer;
 INSERT_UPDATE TestItemType2;\
    bar[unique=true];\
    localizedFooBar[lang=en];\
    localizedFooBar[lang=fr];
   "#% beforeEach:
 String lang = line.get(Integer.valueOf(2));
 if (lang.equals(""en"")) {
      //ignore french language version
      line.put(Integer.valueOf(2), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(3), """");
 } else {
      //ignore french language version
      line.put(Integer.valueOf(3), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(2), """");
 }";
; foobarCode1;en;Horse
; foobarCode1;fr;cheval

The result:

Processing the last imported item

This example comes with hybris:

INSERT_UPDATE Order;code[unique=true];user(uid);date[dateformat=dd.MM.yyyy HH:mm];currency(isocode);net;deliveryMode(code[default='premium-gross']);paymentMode(code);Discounts(code);calculated;site(uid);store(uid);status(code)
;zohan-00-1;zohan@customer.com;01.04.2011 15:10;USD;false;;advance;;false;electronics;electronics;CREATED
"#%   impex.getLastImportedItem().setDeliveryAddress(impex.getLastImportedItem().getUser().getDefaultDeliveryAddress());";
"#%   impex.getLastImportedItem().setPaymentAddress(impex.getLastImportedItem().getUser().getDefaultPaymentAddress());";
"#%   impex.getLastImportedItem().calculate();";

As you can see, the setter and operations share the same object. This is because it deals with Jalo AbstractOrder. This example uses the deprecated Jalo method calculate. It is better to use calculationService.calculate(impex.getLastImportedItem()) instead.

Collections

There is an undocumented way to work with collections:

UPDATE Language ; isoCode[unique=true]; fallbackLanguages(isoCode)
; en ; (+) ru
; en ; (+) fr
; en ; (+) it
; en ; (-) fr

Result:

© Rauf Aliev, October 2016