Mastering hybris Impex: Recipes and Secrets

In this article, I am going to tell you about undocumented or under-documented capabilities of the Impex Engine.

CSV Format

The documentation says that values should be provided as Comma-Separated Values (CSV). SAP refers to RFC 4180 as a specification used in hybris. However, the format specification is different for the impex headers and data blocks:

• you can use “\” to show that the next line is a continuation of the current line.
• you can use line breaks anywhere but you need to quote the value (both in the header and data block);

Multi-line support

ImpEx extension provides the multi-line separator symbol for breaking lines. By default this symbol is “\”. This symbol can’t be used in the data block as multi-line separator. The following ImpEx is valid: Don’t write in such a way, that is a wrong style. However, it demonstrates that the hybris CSV parser ignores line breaks for quoted strings. By default, the multi-line separator is not used in the data block. You can use that for creating fancy-looking impex files. The following example is not perfect, but valid and clearly demonstrates the idea: There are two drawbacks:

• Backslashes makes CSV non-valid. The file is still readable by hybris, but all CSV editors won’t be able to work with it.
• Leading spaces in the each line in the green block will be written into the database. Specifically for the HTML fragment the spaces don’t make any sense, but sometimes it does matter

Removing all data (all records) from the itemtype

The following impex removes all data from TestItem (replace TestItem with your type):

$item=TestItem
REMOVE $item[batchmode=true];itemtype(code)[unique=true]
;$item

Conditional ImpEx

Some  data fragments can be marked as conditional:

$var=value
...
INSERT_UPDATE TestItem;string[unique=true];integer
"#% if: ""$var"".equals(""value"")"
"key";5
#% endif:

In similar way, you can execute the fragments depending on the configuration variables

#% if: !"responsive".equalsIgnoreCase(Config.getParameter("commerceservices.default.desktop.ui.experience"));
...
#% endif:

Accessing Beans

Below is an example of accessing one of the standard hybris beans, flexibleSearch.

....
"#%
import  de.hybris.platform.servicelayer.search.FlexibleSearchQuery;
flexibleSearchService = Registry.getApplicationContext().getBean(""flexibleSearchService"");"

"#%  
query = ""SELECT...."";
flexibleSearchQuery = new FlexibleSearchQuery(query);
resultIterator = flexibleSearchService.search(flexibleSearchQuery).getResult().iterator();"

Impex object

There is an object in many examples below. This object is an instance of ImpExImportReader. The following methods are used in impex sctipts more or less frequently:

• impex.setCurrentHeader(HeaderDescriptor). impex.setCurrentHeader(String) These methods can be used to replace the current header. For example, you can conditionally use INSERT or UPDATE modes depending on the data.
• impex.setValueLinesToSkip(int). How many value lines should be skipped.
• impex.getLastImportedItem(). Returns the last imported item (of Item type)
• impex.isSecondPass(). If csv data are parsed again for resolving previously unresolvable item references.
• impex.setCreateHMCSavedValues(boolean). Enables/disables the automatic update of item history data for created/removed/modified items.
• impex.discardNextLine(). Next line won’t be processed.
• impex.enableExternalSyntaxParsing(boolean). If false, Impex syntax in external data (like headers) won’t be interpreted (and the lines will be interpreted as value lines)
• impex.includeSQLData(url,user,password,JDBCdriver,sql). Initialize parameters and fires an SQL against the database.

#% impex.enableCodeExecution( true );
#% impex.setRelaxedMode( true );
INSERT_UPDATE Product; code[unique=true]; name
#% impex.includeSQLData("jdbc:mysql://localhost/mydb?user=rauf&password=raufpassword", "com.mysql.jdbc.Driver", "Select code,name from SomeTable");

• impex.includeSQLData(sql). The alternative way of firing a statement. To use with initDatabase(..).
• impex.includeSQLData(sql, int skip, int offset). Offset is a number of columns to skip from left. Skip  is a number of resulting lines to skip. 
• impex.initDatabase(url,user,password,JDBCdriver). Initialize parameters to use with includeDatabase. It is useful when you have more than one includeSQLData in the impex file.
• impex.includeExternalData(file, encoding, lineToSkip). Reads values from the file. 
• impex.includeExternalDataMedia(code).

Example from “bmecat” hybris module:

#% if: impex.findExternalDataMedia("regions.csv")!=null
"#% impex.info("" Creating regions"")"
INSERT_UPDATE REGION; country(isocode); isocode[unique=true]; active[virtual=true, default=true];
#% impex.includeExternalDataMedia("regions.csv");
#% endif:

• impex.includeExternalDataMedia(code, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, encoding, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, encoding, delimiter, linesToSkip, columnOffset).
• impex.includeExternalDataMedia(code, columnOffset).
• impex.getCurrentLocation(). Returns current line number.
• impex.insertLine(csvLine). Inserts a line into a parsing stream (it will be a next processing line)
• impex.addDefinition(definition).  Creates/modifies a macro.  Definition may look like “$a=33”.
• impex.setValidationMode(mode). Sets up a validation mode. The following are supported: import_strict, import_relaxed, export_only, export_reimport_relaxed, export_reimport_strict.
• impex.setRelaxedMode(). The same as setValidationMode(“import_relaxed”)
• impex.debug(msg).  Shows a debug message in the console. Internally triggers log.debug(msg), that means that your message will be shown only if you turned the debug mode on in the configuration. 
• impex.info(msg).  Shows a debug message in the console (log.info)
• impex.warn(msg).  Shows a debug message in the console (log.warn)
• impex.error(msg).  Shows a debug message in the console (log.error)

Setting up impex variables from the beanshell code

The following fragment demonstrates how variables can be set up dynamically. The text part of the addDefinition method can be constructed using the data from the bean method calls (see Accessing Beans).

"#% impex.getReader().addDefinition(""$a=33"")";

Using variables in the code

To access ImpEx variables from ImpEx BeanShell code just put its name into the beanshell code. Impex engine replaces all of the macros before running your code so just

$cronjob=abcd
#% java.lang.System.out.println("$cronjob");

Mapping all configuration variables into impex macros

UPDATE GenericItem[processor=de.hybris.platform.commerceservices.impex.impl.ConfigPropertyImportProcessor];pk[unique=true]

$variable=$config-your.config.property

INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$variable

# OR you can just directly use the config macro
INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$config-your.config.property

Importing a property with a map 

INSERT_UPDATE BTGConfig[unique=true];\
               defaultTimeUnit(code);\
                 usedRuleTypes(code);\
                    operandMapping(key(code),value(code))[map-delimiter=|]
BTGConfig;     WEEK;\
                 ORDER,CART,USER,WCMS;\
                    ORDER->BTGOrganizationTotalSpentInCurrencyRelativeDatesOperand,\
                           BTGOrganizationTotalSpentInCurrencyLastYearOperand,\
                           BTGNumberOfOrdersAboveAmountRelativeDateOperand,\
                           BTGCategoriesInOrdersOperand,BTGEachOrderTotalSumOperand,\
                           BTGOrderTotalSumOperand,\
                           BTGProductsInOrdersOperand,\
                           BTGNumberOfOrdersOperand,
                           BTGNumberOfOrdersRelativeDateOperand,
                           BTGLastOrderDateOperand\
                           |\
                     CART->BTGCartIsEmptyOperand,\
                           BTGCartTotalSumOperand,\
                           BTGCategoriesInCartOperand,\
                           BTGProductsInCartOperand,\
                           BTGQuantityOfProductInCartOperand\
                           |\
                     WCMS->BTGViewedProductsOperand,\
                           BTGViewedCategoriesOperand,\
                           BTGVisitedContentpagesOperand,\
                           BTGReferalUrlOperand,BTGUrlParameterOperand\
                           \|
                     USER->BTGOrganizationUIDOfUserOperand,BTGUserAddressPostalCodeOperand,BTGUserCountryOperand,BTGUserGenderOperand,BTGUserMemberOfGroupsOperand|SCRIPT->BTGMediaScriptOperand,\
                           BTGStringScriptOperand

The last column contains the values for a property defined as map

Importing two objects which depend on each other with mandatory fields

Suppose that the following relation is defined: The following Impex adds  a user and a book at the same time:

INSERT_UPDATE Author;code[unique=true];book(code);
#% impex.setValidationMode("import_relaxed");
;author1;Book1

INSERT_UPDATE Book;code[unique=true];name;author(code)
;Book1;BookName;author1

Relaxing the validation means that Hybris doesn’t check the mandatory fields.

Importing data from the external data source (sql database)

"#%
impex.initDatabase(""jdbc:mysql://localhost/testdb?user=testuser&password=testpass"",""com.mysql.jdbc.Driver"");
impex.setRelaxedMode(false);
impex.includeSQLData("Select * from Products");"""

Importing language versions of the content in the separate lines

#% import java.lang.String;
#% import java.lang.Integer;
 INSERT_UPDATE TestItemType2;\
    bar[unique=true];\
    localizedFooBar[lang=en];\
    localizedFooBar[lang=fr];
   "#% beforeEach:
 String lang = line.get(Integer.valueOf(2));
 if (lang.equals(""en"")) {
      //ignore french language version
      line.put(Integer.valueOf(2), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(3), """");
 } else {
      //ignore french language version
      line.put(Integer.valueOf(3), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(2), """");
 }";
; foobarCode1;en;Horse
; foobarCode1;fr;cheval

The result:

Processing the last imported item

This example comes with hybris:

INSERT_UPDATE Order;code[unique=true];user(uid);date[dateformat=dd.MM.yyyy HH:mm];currency(isocode);net;deliveryMode(code[default='premium-gross']);paymentMode(code);Discounts(code);calculated;site(uid);store(uid);status(code)
;zohan-00-1;zohan@customer.com;01.04.2011 15:10;USD;false;;advance;;false;electronics;electronics;CREATED
"#%   impex.getLastImportedItem().setDeliveryAddress(impex.getLastImportedItem().getUser().getDefaultDeliveryAddress());";
"#%   impex.getLastImportedItem().setPaymentAddress(impex.getLastImportedItem().getUser().getDefaultPaymentAddress());";
"#%   impex.getLastImportedItem().calculate();";

As you see the setter and operations share the same object. It is so because it deals with jalo AbstractOrder. This example uses deprecated jalo method (calculate). It is better to use calculationService.calculate(impex.getLastImportedItem()) instead.

Collections

There is undocumented way to work with the collections:

UPDATE Language ; isoCode[unique=true]; fallbackLanguages(isoCode)
; en ; (+) ru
; en ; (+) fr
; en ; (+) it
; en ; (-) fr

Result:

© Rauf Aliev, October 2016