Hybris Impex Preprocessor: Impex PLUS

A note from 2026: This article was published in 2018, when SAP Commerce Cloud was still commonly referred to as SAP Hybris. ImpEx remains supported, but some platform extension, setup, and packaging APIs have changed across SAP Commerce Cloud releases, so adapt integration code to your target version.

ImpEx is a hybris-specific language built on top of CSV for importing/exporting data to and from the database. The hybris ImpEx engine converts ImpEx statements into SQL. Developers work with ImpEx only; SQL statements are not used at all.

The syntax of ImpEx is quite flexible, but there are cases that it does not cover nicely. I developed an ImpEx preprocessor to make up for such shortfalls.

This article explains my enhanced version of the ImpEx scripting language. For some tasks, however, the default capabilities are not enough. The additions explained in this article are especially useful in multi-market setups, where many objects must be created with the same or similar configuration.

Macro definitions plus: key concepts

Out of the box, ImpEx allows you to define macros so that you do not have to type repeating strings. For example, you can define a variable $A=3 and use it one or more times in the same file. The ImpEx engine replaces these variables with their values.

$catalogVersion = catalogVersion(catalog(id), version)[unique=true,default=CatA:Online]
UPDATE MyComponent; $catalogVersion;uid[unique=true];title
;;componentId1;title1
;;componentId2;title2
;;componentId3;title3

It is a very useful feature, but some things are not possible to implement using macros. The out-of-the-box macro functionality does not help you iterate over a collection in ImpEx. There is an out-of-the-box way to do that: you can use embedded code with BeanShell. However, the scripts look awful, are inconvenient, and are error-prone.

Another option is using the Velocity engine to convert ImpEx templates into ImpEx files during a build phase. It requires customization. Some examples of that can be seen in hybris OOTB modules.

My solution is to convert the ImpEx templates on the fly and use a dedicated macro language, which could be better suited for this purpose than Velocity.

The concept is easier to demonstrate by example. Suppose you have four catalogs: A, B, C, and D. You need to create all of them in one ImpEx. It is assumed that the number of catalogs is higher than four and growing. The purpose is to create a maintainable set of data load/update scripts.

INSERT_UPDATE Catalog;id[unique=true]
;A
;B
;C
;D
INSERT_UPDATE CatalogVersion;catalog(id)[unique=true];version[unique=true];active
;A;Online;true
;B;Staged;false
;C;Online;true
;D;Staged;false

It is pretty straightforward, but when you have a dozen catalogs and need to add another one, you need to duplicate lines in different files. This approach is standard but error-prone and inconvenient.

In my solution, there is a special kind of macro substitution. It is enclosed in percent signs and consists of the macro name and one or more optional attributes. With my macro enhancements, the script will look like this:

INSERT_UPDATE Catalog;id[unique]
;%ALLCATALOGS.ID%

INSERT_UPDATE CatalogVersion;catalog(id)[default=%ALLCATALOGS.ID%, unique=true];version[unique=true];active
;%ALLCATALOGVERSIONS.ID%; %ALLCATALOGVERSIONS.ACTIVE%

To get it converted into ImpEx, this template requires a macro definition, which is defined separately and can be shared among different scripts:

{ 
  "ALLCATALOGS": [ { ID: A }, { ID: B }, { ID: C }, { ID: D } ],
  "ALLCATALOGVERSIONS" : [{ ID:Online, ACTIVE:true }, {ID:Staged, ACTIVE:false}]
}

This macro and the template above generate the following ImpEx:

INSERT_UPDATE Catalog;id[unique]
;A
;B
;C
;D

INSERT_UPDATE CatalogVersion;catalog(id)[default=A, unique=true];version[unique=true];active
;;Staged; false
;;Online; true

INSERT_UPDATE CatalogVersion;catalog(id)[default=B, unique=true];version[unique=true];active
;;Staged; false
;;Online; true

INSERT_UPDATE CatalogVersion;catalog(id)[default=C, unique=true];version[unique=true];active
;;Staged; false
;;Online; true

INSERT_UPDATE CatalogVersion;catalog(id)[default=D, unique=true];version[unique=true];active
;;Staged; false
;;Online; true

We saved a lot of ImpEx code and moved the configuration to a separate file. This file can be shared across other ImpEx files.

The example above shows the following features of the preprocessor:

Repeating whole blocks (INSERT_UPDATE) as many times as the macro values.
Repeating the data block lines as many times as the macro values.
Using the hierarchical structure of the macro definition.

The first macro attribute in a line is an iterator. All other attributes are used for fetching data from the JSON block for the macro identified by the first attribute.

Only one macro object per line is allowed. However, you can use any number of attributes of the object there.

References

References are used if you need to avoid duplication in macro definitions. For example, you need to iterate over product catalogs and refer to some catalog attributes defined in the list of all catalogs. You can create a reference by using a special character, &, in the macro value. The syntax of this value is the following: &OBJ.ATTRIBUTE/VALUE

{
 "ALLPRODUCTCATALOGS" : [ 
 { "ID" : "A", "DETAILS": "&ALLCATALOGS.ID/A" },
 { "ID" : "B", "DETAILS": "&ALLCATALOGS.ID/B" } ],
 "ALLCATALOGS": [ 
 { "ID": "A", "NAME": "ProductCatalogA" }, 
 { "ID": "B", "NAME": "ProductCatalogB" }, 
 { "ID": "C", "NAME": "ContentCatalogC" }, 
 { "ID": "D", "NAME": "ContentCatalogD" } ]
}

With this macro, the following script:

INSERT_UPDATE Catalog;id[unique=true]
;%ALLCATALOGS.ID%

INSERT_UPDATE Catalog;id[unique=true];name
;%ALLPRODUCTCATALOGS.ID%;%ALLPRODUCTCATALOGS.DETAILS.NAME%;

will generate the following ImpEx:

INSERT_UPDATE Catalog;id[unique=true]
;A
;B
;C
;D

INSERT_UPDATE Catalog;id[unique=true];name
;A;ProductCatalogA;
;B;ProductCatalogB;

Global Constants

References can be used to define and use constants shared between macros:

{ "CAT" : [ 
    { ID : A, 
      TYPE:"&TYPE.ID/TYPE1" }, 
    { ID : B, 
      TYPE:"&TYPE.ID/TYPE2"  } 
   ],
  "TYPE" : [
    { "ID" : "TYPE1", "VALUE" : "TYPE_1" },
    { "ID" : "TYPE2", "VALUE" : "TYPE_2" }
   ]
}

The following script uses these global types:

UPDATE Obj;
;%CAT.ID%;%CAT.TYPE%;

Resulting ImpEx:

UPDATE Obj;
;A;TYPE_1;
;B;TYPE_2;

Localizations and data maps

This feature allows you to refer to complex objects. It is easier to demonstrate by example. Let’s take the following macro definition:

{
 "ALLPRODUCTCATALOGS" :  [ { "ID" : "A", "DETAILS": "&ALLCATALOGS.ID/A" },
 { "ID" : "B", "DETAILS": "&ALLCATALOGS.ID/B" } ],
 "LANGUAGES" : [
     { "ID" : "fr" },
     { "ID" : "en" }
 ],
 "ALLCATALOGS": [
      { "ID": "A",
        "NAME": [ { "en" : "ProductCatalogA" } ,
                  { "fr" : "CatalogueDeProduitsA" }
            ] } ,
      { "ID": "B",
        "NAME": [  { "en" : "ProductCatalogB" },
                   { "fr" : "CatalogueDeProduitsB" }
         ]  } ]
}

As you can see, in this macro I added localizations for the catalog names. You can refer to these translations by specifying a language ID in square brackets in the data lines.

INSERT_UPDATE Catalog;id[unique=true];name[lang=%LANGUAGES.ID%]
;%ALLPRODUCTCATALOGS.ID%;%ALLPRODUCTCATALOGS.DETAILS.NAME[LANGUAGES.ID]%;

The template above will generate the following ImpEx file:

INSERT_UPDATE Catalog;id[unique];name[lang=fr]
;A;CatalogueDeProduitsA;
;B;CatalogueDeProduitsB;

INSERT_UPDATE Catalog;id[unique];name[lang=en] 
;A;ProductCatalogA; 
;B;ProductCatalogB;

So, the macro in the header makes the whole ImpEx block repeat as many times as the macro values there. The macro in the block makes the data lines repeat. The parameter in square brackets can refer only to a macro in a header.

Lists

Lists and maps are converted into strings as comma-separated values:

{
 "ALLPRODUCTCATALOGS" :  [ { "ID" : "A", "DETAILS": "&ALLCATALOGS.ID/A" },
 { "ID" : "B", "DETAILS": "&ALLCATALOGS.ID/B" } ],
 "LANGUAGES" : [
     { "ID" : "fr" },
     { "ID" : "en" }
 ],
 "ALLCATALOGS": [
      { "ID": "A",
        "NAME": [ { "en" : "ProductCatalogA" } ,
                  { "fr" : "CatalogueDeProduitsA" }
            ] } ,
      { "ID": "B",
        "NAME": [  { "en" : "ProductCatalogB" },
                   { "fr" : "CatalogueDeProduitsB" }
         ]  } ]
}

ImpEx template:

INSERT_UPDATE Object;languages 
;%LANGUAGES%;
INSERT_UPDATE Object;languages 
;%LANGUAGES.ID%;
INSERT_UPDATE Object;names 
;%ALLCATALOGS.NAME["en"]%;
INSERT_UPDATE Object;names 
;%ALLCATALOGS.NAME%;

The resulting ImpEx below shows the difference:

INSERT_UPDATE Object;languages 
;fr, en;
INSERT_UPDATE Object;languages 
;fr;
;en;
INSERT_UPDATE Object;names 
;ProductCatalogA;
;ProductCatalogB;
INSERT_UPDATE Object;names 
;ProductCatalogA,CatalogueDeProduitsA;
;ProductCatalogB,CatalogueDeProduitsB;

If the value contains spaces or special characters, it will be enclosed in double quotes automatically. Let’s assume we changed the macro definition file to meet this case:

INSERT_UPDATE Object;names 
;"Product Catalog A","CatalogueDeProduitsA";
;"Product Catalog B","CatalogueDeProduitsB";

Configuration variables

Some values can be set as configuration variables.

{
 "ALLPRODUCTCATALOGS" : [ 
   { "ID" : A, NAME:"&config.catalogA.Name" },
   { "ID" : B, NAME:"&config.catalogB.Name" } 
 ],
}

These variables can be defined in project.properties or local.properties:

catalogA.Name=Catalog A Name
catalogB.Name=Catalog B Name

Impex Variables and Macros

Because it is a preprocessor, we need to have a valid ImpEx file as a result of applying macros. The examples above demonstrated how to use macros in headers and data lines. But how can we use them in variables? For example:

$catalogVersion=catalogVersion(catalog(id), version)[default=%CAT%]
UPDATE Obj; $catalogVersion

Unfortunately, macro substitutions will not work in this case if used as above. The main reason is that it is a stream one-pass preprocessor, which creates limitations. The allowed syntax is the following:

$catalogVersion%CAT.ID%=catalogVersion(catalog(id), version)[default=%CAT.ID%]
UPDATE Obj; 
;$catalogVersion%CAT.ID%

Let’s assume we use the following macro definition:

{ "CAT" : [ { ID : A }, { ID : B } ] }

In this case, you will have the following ImpEx file as a result:

$catalogVersionA=catalogVersion(catalog(id), version)[default=A]
$catalogVersionB=catalogVersion(catalog(id), version)[default=B]
UPDATE Obj; 
;$catalogVersionA
;$catalogVersionB

Impex Plus Validators

The macros explained above are especially useful for a large number of catalogs or languages. However, the macro definition file tends to grow over time, and it is easy to overlook one or several missing blocks that are supposed to be mandatory.

Templates:
- Only allowed symbols are in the macro definition. If any unallowed symbols are found, the fragment will not be considered a macro and will be processed as simple ImpEx content.
- Only one macro per line. However, you can use any number of attributes there.
- All macros in ImpEx are defined in the macro definition file. If a macro is present in ImpEx but not defined in the macro definition file, the system throws an error.
Macro definitions:
- JSON is valid.
- All macro references (&obj.attr/value) are valid.
- All macro names and attributes conform to the requirements: allowed characters, etc.

Integrating with Hybris

Unfortunately, the SAP Hybris ImpEx Processor is not extendable. Using processors does not help much either. I used an ImpEx converter class that transforms an ImpEx template + macro definition into an ImpEx data stream, which the SAP hybris default ImpEx engine is able to execute.

The extension setup class was extended to use the enhanced mechanism instead of the default one, so that the OOTB updates support the ImpEx templates and macro language.