Mastering hybris Impex: Recipes and Secrets


In this article, I am going to tell you about undocumented or under-documented capabilities of the Impex Engine.

CSV Format

The documentation says that values should be provided as Comma-Separated Values (CSV). SAP refers to RFC 4180 as a specification used in hybris. However, the format specification is different for the impex headers and data blocks:
  • you can use “\” to show that the next line is a continuation of the current line.
  • you can use line breaks anywhere but you need to quote the value (both in the header and data block);

Multi-line support

ImpEx extension provides the multi-line separator symbol for breaking lines. By default this symbol is “\”. This symbol can’t be used in the data block as multi-line separator. The following ImpEx is valid: 2016-10-31_18h30_58 Don’t write in such a way, that is a wrong style. However, it demonstrates that the hybris CSV parser ignores line breaks for quoted strings. By default, the multi-line separator is not used in the data block. You can use that for creating fancy-looking impex files. The following example is not perfect, but valid and clearly demonstrates the idea: 2016-10-31_19h04_25.png There are two drawbacks:
  • Backslashes makes CSV non-valid. The file is still readable by hybris, but all CSV editors won’t be able to work with it.
  • Leading spaces in the each line in the green block will be written into the database. Specifically for the HTML fragment the spaces don’t make any sense, but sometimes it does matter

Removing all data (all records) from the itemtype

The following impex removes all data from TestItem (replace TestItem with your type):
$item=TestItem
REMOVE $item[batchmode=true];itemtype(code)[unique=true]
;$item

Conditional ImpEx

Some  data fragments can be marked as conditional:
$var=value
...
INSERT_UPDATE TestItem;string[unique=true];integer
"#% if: ""$var"".equals(""value"")"
"key";5
#% endif:
In similar way, you can execute the fragments depending on the configuration variables
#% if: !"responsive".equalsIgnoreCase(Config.getParameter("commerceservices.default.desktop.ui.experience"));
...
#% endif:

Accessing Beans

Below is an example of accessing one of the standard hybris beans, flexibleSearch.
....
"#%
import  de.hybris.platform.servicelayer.search.FlexibleSearchQuery;
flexibleSearchService = Registry.getApplicationContext().getBean(""flexibleSearchService"");"
"#%  
query = ""SELECT...."";
flexibleSearchQuery = new FlexibleSearchQuery(query);
resultIterator = flexibleSearchService.search(flexibleSearchQuery).getResult().iterator();"

Impex object

There is an object in many examples below. This object is an instance of ImpExImportReader. The following methods are used in impex sctipts more or less frequently:
  • impex.setCurrentHeader(HeaderDescriptor). impex.setCurrentHeader(String) These methods can be used to replace the current header. For example, you can conditionally use INSERT or UPDATE modes depending on the data.
  • impex.setValueLinesToSkip(int). How many value lines should be skipped.
  • impex.getLastImportedItem(). Returns the last imported item (of Item type)
  • impex.isSecondPass(). If csv data are parsed again for resolving previously unresolvable item references.
  • impex.setCreateHMCSavedValues(boolean). Enables/disables the automatic update of item history data for created/removed/modified items.
  • impex.discardNextLine(). Next line won’t be processed.
  • impex.enableExternalSyntaxParsing(boolean). If false, Impex syntax in external data (like headers) won’t be interpreted (and the lines will be interpreted as value lines)
  • impex.includeSQLData(url,user,password,JDBCdriver,sql). Initialize parameters and fires an SQL against the database.
#% impex.enableCodeExecution( true );
#% impex.setRelaxedMode( true );
INSERT_UPDATE Product; code[unique=true]; name
#% impex.includeSQLData("jdbc:mysql://localhost/mydb?user=rauf&password=raufpassword", "com.mysql.jdbc.Driver", "Select code,name from SomeTable");
  • impex.includeSQLData(sql). The alternative way of firing a statement. To use with initDatabase(..).
  • impex.includeSQLData(sql, int skip, int offset). Offset is a number of columns to skip from left. Skip  is a number of resulting lines to skip. 
  • impex.initDatabase(url,user,password,JDBCdriver). Initialize parameters to use with includeDatabase. It is useful when you have more than one includeSQLData in the impex file.
  • impex.includeExternalData(file, encoding, lineToSkip). Reads values from the file. 
  • impex.includeExternalDataMedia(code).
Example from “bmecat” hybris module:
#% if: impex.findExternalDataMedia("regions.csv")!=null
"#% impex.info("" Creating regions"")"
INSERT_UPDATE REGION; country(isocode); isocode[unique=true]; active[virtual=true, default=true];
#% impex.includeExternalDataMedia("regions.csv");
#% endif:
  • impex.includeExternalDataMedia(code, linesToSkip, columnOffset).
  • impex.includeExternalDataMedia(code, encoding, linesToSkip, columnOffset).
  • impex.includeExternalDataMedia(code, encoding, delimiter, linesToSkip, columnOffset).
  • impex.includeExternalDataMedia(code, columnOffset).
  • impex.getCurrentLocation(). Returns current line number.
  • impex.insertLine(csvLine). Inserts a line into a parsing stream (it will be a next processing line)
  • impex.addDefinition(definition).  Creates/modifies a macro.  Definition may look like “$a=33”.
  • impex.setValidationMode(mode). Sets up a validation mode. The following are supported: import_strict, import_relaxed, export_only, export_reimport_relaxed, export_reimport_strict.
  • impex.setRelaxedMode(). The same as setValidationMode(“import_relaxed”)
  • impex.debug(msg).  Shows a debug message in the console. Internally triggers log.debug(msg), that means that your message will be shown only if you turned the debug mode on in the configuration. 
  • impex.info(msg).  Shows a debug message in the console (log.info)
  • impex.warn(msg).  Shows a debug message in the console (log.warn)
  • impex.error(msg).  Shows a debug message in the console (log.error)

Setting up impex variables from the beanshell code

The following fragment demonstrates how variables can be set up dynamically. The text part of the addDefinition method can be constructed using the data from the bean method calls (see Accessing Beans).
"#% impex.getReader().addDefinition(""$a=33"")";

Using variables in the code

To access ImpEx variables from ImpEx BeanShell code just put its name into the beanshell code. Impex engine replaces all of the macros before running your code so just
$cronjob=abcd
#% java.lang.System.out.println("$cronjob");

Mapping all configuration variables into impex macros

UPDATE GenericItem[processor=de.hybris.platform.commerceservices.impex.impl.ConfigPropertyImportProcessor];pk[unique=true]
$variable=$config-your.config.property
INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$variable

# OR you can just directly use the config macro
INSERT_UPDATE SampleItem;code[unique=true];name
;sample1;$config-your.config.property

Importing a property with a map 

INSERT_UPDATE BTGConfig[unique=true];\
               defaultTimeUnit(code);\
                 usedRuleTypes(code);\
                    operandMapping(key(code),value(code))[map-delimiter=|]
BTGConfig;     WEEK;\
                 ORDER,CART,USER,WCMS;\
                    ORDER->BTGOrganizationTotalSpentInCurrencyRelativeDatesOperand,\
                           BTGOrganizationTotalSpentInCurrencyLastYearOperand,\
                           BTGNumberOfOrdersAboveAmountRelativeDateOperand,\
                           BTGCategoriesInOrdersOperand,BTGEachOrderTotalSumOperand,\
                           BTGOrderTotalSumOperand,\
                           BTGProductsInOrdersOperand,\
                           BTGNumberOfOrdersOperand,
                           BTGNumberOfOrdersRelativeDateOperand,
                           BTGLastOrderDateOperand\
                           |\
                     CART->BTGCartIsEmptyOperand,\
                           BTGCartTotalSumOperand,\
                           BTGCategoriesInCartOperand,\
                           BTGProductsInCartOperand,\
                           BTGQuantityOfProductInCartOperand\
                           |\
                     WCMS->BTGViewedProductsOperand,\
                           BTGViewedCategoriesOperand,\
                           BTGVisitedContentpagesOperand,\
                           BTGReferalUrlOperand,BTGUrlParameterOperand\
                           \|
                     USER->BTGOrganizationUIDOfUserOperand,BTGUserAddressPostalCodeOperand,BTGUserCountryOperand,BTGUserGenderOperand,BTGUserMemberOfGroupsOperand|SCRIPT->BTGMediaScriptOperand,\
                           BTGStringScriptOperand
The last column contains the values for a property defined as map

Importing two objects which depend on each other with mandatory fields

Suppose that the following relation is defined:

GeSHi Error: GeSHi could not find the language html (using path /var/www/html/hybrismart.com/public_html/wp-content/plugins/codecolorer/lib/geshi/) (code 2)
The following Impex adds  a user and a book at the same time:
INSERT_UPDATE Author;code[unique=true];book(code);
#% impex.setValidationMode("import_relaxed");
;author1;Book1

INSERT_UPDATE Book;code[unique=true];name;author(code)
;Book1;BookName;author1
Relaxing the validation means that Hybris doesn’t check the mandatory fields.

Importing data from the external data source (sql database)

"#%
impex.initDatabase(""jdbc:mysql://localhost/testdb?user=testuser&password=testpass"",""com.mysql.jdbc.Driver"");
impex.setRelaxedMode(false);
impex.includeSQLData("Select * from Products");"""

Importing language versions of the content in the separate lines

#% import java.lang.String;
#% import java.lang.Integer;
 INSERT_UPDATE TestItemType2;\
    bar[unique=true];\
    localizedFooBar[lang=en];\
    localizedFooBar[lang=fr];
   "#% beforeEach:
 String lang = line.get(Integer.valueOf(2));
 if (lang.equals(""en"")) {
      //ignore french language version
      line.put(Integer.valueOf(2), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(3), """");
 } else {
      //ignore french language version
      line.put(Integer.valueOf(3), line.get(Integer.valueOf(3)));
      line.put(Integer.valueOf(2), """");
 }";
; foobarCode1;en;Horse
; foobarCode1;fr;cheval
The result: 2016-11-02_20h33_49.png

Processing the last imported item

This example comes with hybris:
INSERT_UPDATE Order;code[unique=true];user(uid);date[dateformat=dd.MM.yyyy HH:mm];currency(isocode);net;deliveryMode(code[default='premium-gross']);paymentMode(code);Discounts(code);calculated;site(uid);store(uid);status(code)
;zohan-00-1;zohan@customer.com;01.04.2011 15:10;USD;false;;advance;;false;electronics;electronics;CREATED
"#%   impex.getLastImportedItem().setDeliveryAddress(impex.getLastImportedItem().getUser().getDefaultDeliveryAddress());";
"#%   impex.getLastImportedItem().setPaymentAddress(impex.getLastImportedItem().getUser().getDefaultPaymentAddress());";
"#%   impex.getLastImportedItem().calculate();";
As you see the setter and operations share the same object. It is so because it deals with jalo AbstractOrder. This example uses deprecated jalo method (calculate). It is better to use calculationService.calculate(impex.getLastImportedItem()) instead.

Collections

There is undocumented way to work with the collections:
UPDATE Language ; isoCode[unique=true]; fallbackLanguages(isoCode)
; en ; (+) ru
; en ; (+) fr
; en ; (+) it
; en ; (-) fr

Result:
2016-11-02_22h44_01

© Rauf Aliev, October 2016

9 Responses

  1. Lenar Fatikhov

    Lenar Fatikhov

    Reply

    3 November 2016 at 13:41

    Cool, thank you!
    I would also add a hint about the possibility to use not only BeanShell, but Groovy in impex.

  2. Julio Argüello

    Julio Argüello

    Reply

    9 November 2016 at 03:54

    Good Job!

  3. seabook

    seabook

    Reply

    28 July 2017 at 03:05

    Amazing articles for hybris. Btw, here seems missed ImpexTranslator

  4. badtenant

    badtenant

    Reply

    12 October 2017 at 18:06

    How do you handle a relation with a date?

    INSERT_UPDATE table;relationField(code,date)[unique=true,dateFormat=dd-MM-yyyy];value
    ;codevalue:”01-01-1970″;value

    It thinks the colons are more values…Is there a solution to this?

    I get errors like invalid date and too many referenced values trying all kinds of combinations….

  5. camelofcode

    camelofcode

    Reply

    12 December 2017 at 08:42

    I’ve been able to do some really useful stuff thanks to this article. A real lifesaver, thanks

  6. Stacie Beckham

    Stacie Beckham

    Reply

    26 September 2018 at 18:32

    With thanks! Valuable information!

Leave a Reply