Understanding SAP Hybris Commerce Persistence Layer: Database tables and their structure
The purpose of this document is to shed a light on the internal data structures used by SAP Hybris Commerce. I explain system tables and attributes, their purpose and meaning.
You know that SAP Hybris Commerce has a powerful type system built on top of the database layer. It means that you can’t (you are not recommended to) write to the database directly. Direct read is also not recommended. So basically the information about the specific attributes and table names is not used by developers directly.
However, understanding of the internal structure is very useful in troubleshooting and low-level data analysis. For example, direct access to the database may show that your system has a lot of unused items, and it affects the performance. Analyzing the data is also extremely helpful with data integrity issues.
Looking at the underlying data model you will understand the processes better. You will see that SAP Hybris Commerce is not a black box with hidden algorithms and logic under the hood.
) and new style data model definitions (
).
Hybris generates database schema using
. The schema is created as a set of SQL files in the temporary directory using Apache DDLUtils SQLBuilder. The generator uses the information loaded from the XML model definitions. Hybris creates three files,
will go there, getting their values stored as blob. There is not much information about it, here in the reference is mentioned (https://wiki.hybris.com/display/release5/items.xml+Element+Reference#items.xmlElementReference-ComplexType:modifiersType).
Basically if you sent the
modifier the object will be stored in this props table, making it more expensive to search by, but releasing the original object of what is expected to be a heavy column in the database.
Typically you would want to add attributes to this table when they are too big and you would never want to search by this attribute, for instance the full hmc xml text, or the comments that could be added to the items.
Anyway, it is not a very good practice to store attributes in this table, as there are cases where it can grow way too much and have a very bad impact on performance, it is more recommended to specify the type of the database you are going to be working on, as explained here.
annotation.
Basically when you select a row with from the database and that returns you the current version. When the row is updated then the version column is incremented. When you update the row you use a WHERE clause on the query to match the version that you selected. If the update doesn’t modify any rows then you know something else has updated the row (and version) in the meantime.
(default false) is config parameter which either enables or disables checking that upon update an item row must have the same version that has been read when fetching the HJMP entity object.
This means setting a
parameter to
in properties activates optimistic locking and of course increasing the risk of seeing optimistic concurrency exceptions such as HJMPException(PK was modified concurrently – expected version … ), which will force a transaction rollback
P.S: If you don’t enable optimistic locking, only the dirty attributes will be written and the final result will be a merge.” (source)
Summary Don’t write to the database directly in hybris. Use the hybris data access APIs in all situations when you need to modify things. There are no exceptions. Otherwise, you will definitely have issues with the caching subsystem. The explanations below are important for troubleshooting and to early detect the data issues. |
Hybris database internals
Creating tables
Hybris creates tables during the initialization and update processes. It loads old-style ([-.]advanced-deployments.xml
-items.xml
HybrisSchemaGenerator.initialize()
- DropDDL – with DropTable statements
- DDL – with CREATE statements (tables and indexes)
- DML – with INSERT/UPDATE statements
Table types
- Item type tables
- Localized values tables
- Generic Audit Tables
- Property Tables
- System Tables
Item type tables
The major part of the list of hybris database tables is item type tables. These tables are created automatically from *-items.xml during the system init or update process. Their names are specified in the ‘deployment’ tag, so the database tables can be of any name chosen by the developer. Actually, in this process some of other tables are created or updated too. For example, localized values tables are created if you have at least one localized attribute. Generic audit tables are created for all item type tables automatically (the feature is available in 6.6+). Each item type table has at least the following columns:-
. Version (bigint). See the section “HJMPTS”. It is incremented each time either the item table, the *props table or the ACLEntry table is changed belonging to a specific item. Example: 3hjmpTS
-
. Created time (datetime). Example: 2017-07-07 18:22:49.000000createdTS
-
. Modified time (datetime). Example: 2017-07-07 18:33:26.000000modifiedTS
-
. Reference to a type (*types.pk). See the section PKs. Example: 8796093087826TypePkString
-
. Supposed to be a reference to a type, but it is null. Possibly, deprecatedOwnerPkString
-
. Unique identifier (See the section “PKs”) Example: 8796093055058PK
-
sealed
-
Obsolete by now. It used to be for optimistic locking covering the ACLEntry table. This is now solely relying on the hjmpTS columnaCLTS
-
Obsolete by now. It used to be for optimistic locking covering the *PROPS tables. This is now solely relying on the hjmpTS columnpropTS
Localized values tables
If the table contains localized properties (having values in different languages), these values are stored separately from the main item type table. The table with such data has a suffix “lp”. For example, productslp has the following attributes:- p_name
- p_description
- p_manufactureretypedescription
- p_segment
- p_articlestatus
- p_summary
- p_style
- p_size
- p_color
Generic Audit tables
Their names are constructed using the item table name, code and a suffix “sn”. For example, for Products, the name of the change history table will be “products1sn” where 1 is a type code of Product and “sn” is a fixed suffix. For the tables having a long name, the name will be truncated. For example, for the table “products4restriction”, the change history table will have a name “products4restri1081sn”. Not all changes are saved. You need to activate this feature for a particular type. Of course, it will affect the performance, but you will be able to track changes. So the set of participating types depends on particular projects and its requirements and constraints. It has been activated out-of-the-box for the following types: AbstractContactInfo, Address, B2bunit, BaseSite, Btgsegment,Btgsegmentresult, Cart, CartEntry, City, Comment, Consent, ConsentTemplate, Country, CsAgentGroup, CsTicket, Currency, Customerreview, CxSegment, CxUsertosegment, District, Employee, Language, Order, Orderentry, Partneraddress, Paymentinfo, Paymentmode, Principalgrouprelation, Product, Quote, Quoteentry, Region, SapB2bdocument, SapOrder, Title, Unit, User, Usergroup, UserPasswordChangeAudit. Read more about Generic Audit feature: https://help.hybris.com/6.7.0/hcd/3e2b8491449d43b391eb1ce9c73cbb7f.html The structure of the change history tables:-
. Unique autoincrement ID.ID
-
. The Type Instance PK (for products it is a PK of the particular product).ITEMPK
-
. The Type PK. For Products it is a PK from *typesITEMTYPEPK
-
. Change time stamp.timestamp
-
. Audit Record Timestamp.currenttimestamp
-
. Who changes the item. It is NOT a PK of the user, it is a string, the user login (because the user record can be removed)changinguser
-
. Additional context (JSON)context
-
. The whole business payload of audited type before a persistence operation (JSON).payloadbefore
-
. The whole business payload of audited type after a persistence operation (JSON)payloadafter
-
. 0 – deletion, 1 – creation, 2 – modificationoperationtype
-
. Source Item PK.sourcePK
-
. Target Item PKtargetPK
-
. Language PKlanguagePK
- timestamp and currenttimestamp will represent the date and time of a change
- itempk and itemtypepk will contain the 13-digit PKs of the language item added and its type (Language)
- changinguser – your current backoffice user, such as “admin”
- operationtype = 1, it means “creation”
- payloadbefore and payloadafter – see below
Property tables
Property tables are specified for item types in *-items.xml as part of ‘deployment’ tag (‘propertytable’ attribute).- addressprops
- quoteentryprops
- orderentryprops
- typesystemprops
- usergroupprops
- userprops
- quoteprops
- orderprops
- mediaprops
- productprops
System tables
- aclentries,
- Types:
- atomictypes,
- collectiontypes,
- composedtypes,
- maptypes,
- attributedescriptors,
- configitems,
- enumerationvalues,
- genericitems,
- gentestitems,
- junit_numberseries,
- links,
- metainformations,
- numberseries,
- orderdiscrels,
- pgrels,
- props,
- searchrestrictions
- ydeployments
- tasks_aux_queue
- tasks_aux_scheduler
- tasks_aux_workers
- testitem
Props tables
Props tables are a little bit special, as all attributes that you specify in the items.xml file with the modifierdontOptimize="true"
dontOptimize
- ‘addressprops’
- ‘mediaprops’
- ‘orderentryprops’
- ‘orderprops’
- ‘productprops’
- ‘props’
- ‘quoteentryprops’
- ‘quoteprops’
- ‘typesystemprops’
- ‘usergroupprops’
- ‘userprops’
ACL Entries (Table «aclentries»)
Purpose: Access Control Lists (Permission-User-Item). Defined by core-advanced-deployment.xml Used by JALO AccessManager. The User/Object permissions are stored here.-
. Version. See the section “HJMPTS”.ID
-
. Reference to UserRight Model.ITEMPK
-
. Stands for negative permissions.negative
-
. Reference to Principal Model.principalPK
-
. Reference to Item Model.itemPK
Type System records
Structure
The type system is represented by the following set of tables:- AtomicTypes
- ComposedTypes
- MapTypes
- CollectionTypes
- EnumerationValues
- AttributeDescriptors
- PropsMap
- yDeployments
- Country
Common attributes
This set of attributes is used in the following tables:- Atomictypes
- Collectiontypes
- Commenttypes
- Composedtypes
- MapTypes
-
. Version (bigint). See the section “HJMPTS”. It is incremented each time either the item table, the *props table or the ACLEntry table is changed belonging to a specific item. Example: 3hjmpTS
-
. Created time (datetime). Example: 2017-07-07 18:22:49.000000createdTS
-
. Modified time (datetime). Example: 2017-07-07 18:33:26.000000modifiedTS
-
. Reference to a type (*types.pk). See the section PKs. Example: 8796093087826TypePkString
-
. Supposed to be a reference to a type, but it is null. Possibly, deprecatedOwnerPkString
-
. Unique identifier (See the section “PKs”) Example: 8796093055058PK
-
. Name of the extension (string). Example: “core” or “cockpit”p_extensionname
-
. Create a database table/model or not (flag). Example: 1p_autocreate
-
. Generate Jalo classes for the model or not (flag). Example: 0p_generate
-
. String representation of the Item name.Examples: “Item”, “ComposedType”, “Media”, “UserRight” (core-advanced-deployment.xml)internalCode
-
Default value. Examples: 0.p_defaultvalue
-
Obsolete by now. It used to be for optimistic locking covering the ACLEntry table. This is now solely relying on the hjmpTS columnaCLTS
-
Obsolete by now. It used to be for optimistic locking covering the *PROPS tables. This is now solely relying on the hjmpTS columnpropTS
Atomic Types (Table «Atomictypes»)
- All common attributes+
-
. Comma-separated list of PKs which are parent to the item. Example: “,8796093055058,8796094431314,8796094464082,”InheritancePathString
-
. Java Class Name. Example: “de.hybris.platform.jalo.media.MediaContainer” (core-advanced-deployment.xml)JavaClassName
-
. Supertype PK. Example: 8796094496850SuperTypePK
-
Example: “java.lang.Long” (core-advanced-deployment.xml)InternalCodeLowerCase
The following atomic types are created by default with the hybris accelerator:
Collection Types (Table «collectiontypes»)
- All common attributes +
-
. Element type PK. Example: 8796094496850 (core-advanced-deployment.xml)ElementTypePK
-
. Type of collection. 0=Collection, 1=Set, 2=List. Example: 1 (core-advanced-deployment.xml)typeOfCollection
-
. Example: ”InternalCodeLowerCase” (core-advanced-deployment.xml)user2owneurope1discountsowneurope1discountscoll
Composed Types (Table «composedtypes»)
- All common attributes+
-
. Comma-separated list of PKs which is a parent to the item. Example: “,8796093055058,8796094431314,8796094464082,” (core-advanced-deployment.xml)inheritancePathString
-
. Example: “de.hybris.platform.jalo.Item” (core-advanced-deployment.xml)jaloClassName
-
. Example: “de.hybris.platform.persistence.Item” (core-advanced-deployment.xml)itemJNDIName
-
. If ‘true’, type will be evaluated with that allowing only one instance per system. (core-advanced-deployment.xml)singleton
-
. DEPRECATED. If ‘true’, the item will only exists in the jalo layer and isn’t backed by an entity bean. Default is ‘false’.p_jaloonly
-
. “1″ for dynamic attributes.p_dynamic
-
. Super type PK (core-advanced-deployment.xml)SuperTypePK
-
. If true, the system uses Jalo strategy for managing ACLs. Otherwise, it uses Service Layer Direct (SLD) strategy.p_legacypersistence
-
. “1″ for system types or nullp_systemtype
-
. “1” for catalog item types or nullp_catalogitemtype
-
. name of the attribute that is treated as catalog Version OR null. For the most items-xml catalog-aware defined types the value is catalogVersion. “systemVersion” is also used, for classification catalogs.p_catalogversionattributequali
-
. Comma-separated list of unique attributes of the typep_uniquekeyattributequalifier
-
. lowercase representation of internalCodeinternalCodeLowerCase
-
. “1″ if removable, “0” is not removable. (core-advanced-deployment.xml)removable
-
. Assumingly, it is a leftover from the time when SAP hybris had one more type item related table, holding unlocalized properties. This obviously wasn’t really needed since those properties can live inside the main item table (as they do today). This field was telling which of the two modes were used. In the current versions, it is always “1”. (core-advanced-deployment.xml)propertyTableStatus
-
. A type code (a number) associated with the composed type (core-advanced-deployment.xml)itemTypeCode
-
. Used only for enumeration types; 0 for sorted.p_comparationattribute
-
. Used only for relation types. 1 for localized relations (see the localized attribute of relation type declaration). A localized n-m relation can have a link between two items for each language.p_localized
-
. Used only for relation types. Source attribute for the relations.p_sourceattribute
-
. Used only for relation types. Target attribute for the relations.p_targetattribute
-
. Used only for relation types. Source type for the relations.p_sourcetype
-
. Used only for relation types. Target type for the relations.p_targettype
-
. Used only for relation types. Is the relation navigable from this side. Can only be disabled for one side of many to many relation.p_sourcenavigable
-
. The same as p_sourcenavigable, but for the target object.p_targetnavigable
-
. Used only for relation types. reflects the “ordering” attribute of the relation tag of items.xml. Contains PK of the attribute used for ordering (*pos).p_orderingattribute
-
. Unknown for me (always null after init for all types)p_localizationattribute
-
. Added by cms2-items.xml for CMSPageType that uses composedtypep_previewdisabled
- ~190 core item types
- about 318 relations
- about 168 enums
Map Types (Table «maptypes»)
- All common attributes +
-
. PK of the Argument type (core-advanced-deployment.xml)ArgumentTypePK
-
. Return type PK (core-advanced-deployment.xml)ReturnTypePK
-
. Internal code lowercased (core-advanced-deployment.xml)InternalCodeLowerCase
Localized Properties (Tables «maptypeslp», «composedtypeslp», «atomictypeslp», «collectiontypeslp»)
These tables contain the information about the item types properties in different languages.
-
. PK of the attributeITEMPK
-
. PK of the item typeITEMTYPEPK
-
. PK of the languageLANGPK
-
. Localized namep_name
-
. Localized descriptionp_description
Attribute Descriptors (Table «attributedescriptors»)
Attribute descriptors contain the information about the type attributes.
- All common without Internal Code
-
. Name of the attribute (from the xml)qualifierInternal
-
. PK of the attribute type. Example: 8796095053906 (UserDiscountGroup)attributeTypePK
-
. Name of the field in the database. Example: “p_europe1pricefactory_udg”columnName
-
. Examples: “Boolean.FALSE”, “em(). getEnumerationValue( “CmsApprovalStatus”, “check”)”p_defaultvaluedefinitionstring
-
. PK of the type this attribute belongs to customerListenclosingTypePK
-
. The cmp (container managed persistence) value for persistence type; it was deprecated back in 4.4 and it only remains for internal development use.persistenceQualifierInternal
-
. Atomic Type used for storing data. For example: java.util.DatepersistenceTypePK
-
. Name of attribute handler, if defined. For example,p_attributehandler.principalGroupDisplayNameLocalizedAttributeHandler
-
. Used mainly for addresses. For example, Address.contactAddress has selectionDescriptorPK? = AddressCollection.contactAddressesselectionDescriptorPK
-
. Bitmask: READ(0), WRITE(1), REMOVE(2), OPTIONAL(3), SEARCH(4), PARTOF(5), PRIVATE(7), PROPERTY(8), LOCALIZED(9), INHERITED(10), INITIAL(11), DO NOT OPTIMIZE(13), ENCRYPTED(14), PRIMITIVE (16)modifiers
-
. “1” if uniquep_unique
-
. “1” if the attribute should be hidden from UI. Example: OrgUnit.path Boolean.TRUEp_hiddenforui
-
. Example: Product.untypedFeatures Boolean.TRUEp_readOnlyForUI
-
. Example: ProductFeature.stringValue Boolean.TRUEp_dontCopy
-
. Lowercase version of QualifierqualifierLowerCaseInternal
-
. “1” if the attribute is hidden. For example, attributes for storing a relative position in ordered lists are hiddenisHidden
-
. “1” if the attribute is property. Attributes with isProperty = 0 are not exposed as item type properties (they are considered as system properties, such as modified time or savedValues)isProperty
-
. Example: CustomerList.name -> Principal.name because CustomerList extends UserGroup, UserGroup extends PrincipalGroup, PrincipalGroup extends Principal.superAttributeDescriptorPK
- inheritancePathString. Comma-separated list of parent types Example: “,8796093841490, 8796094988370, 8796095021138, 8796095479890,”
-
. External qualifier is used to get or set via de.hybris.platform.util.Config. Used only by LDAP module and HMC. Used for Config Attribute Descriptors. Example: “ldap.jndi.connection.timeout”p_externalqualifier
-
. Used for Config Attribute Descriptors. Used only by LDAP module and HMC. Example: Boolean.TRUEp_storeindatabase
-
. Used only by LDAP module and HMC. Example: Boolean.TRUEp_needrestart
-
. Defines if the attribute is either a parameter (param=true) or if it describes a result row attribute (param=false). Example: Boolean.FALSE Deprecated.p_param
-
. Defines the position of the attribute descriptor. Example: Integer.valueOf (2) Deprecated.p_position
-
. Contains the expression for the default value. Example: de.hybris.platform.jalo.JaloSession.getCurrentSession().getUser()p_defaultvalueexpression
-
. “1” if it is a source field in the relationp_issource
-
. “1” for ordered relationsp_ordered
-
. Name of the relation (for relations)p_relationname
-
. PK of the relation type (for relations)p_relationtype
- AttributeDescriptorsLP
Other tables
Common attributes
All objects below contain a fixed set of attributes, namely:-
. A unique keyPK
-
. Version (bigint). See the section “HJMPTS” Example: 3hjmpTS
-
. Created time (datetime). Example: 2017-07-07 18:22:49.000000createdTS
-
. Modified time (datetime). Example: 2017-07-07 18:33:26.000000modifiedTS
-
.aCLTS
-
. Reference to a type (*types.pk). See the section PKs. Example: 8796093087826TypePkString
-
. PK of the ownerOwnerPkString
-
.propTS
Config Items (Table «configitems»)
- (all the common attributes above)
Enumeration Values (Table «enumerationvalues»)
- (all the common attributes above)
-
. An enum value (string). Example: “CANCELLING”code
-
. The code in lowercase. Example: “cancelling”codeLowerCase
-
. A sequence number of the value in enum. Example: 3sequenceNumber
-
. Extension name. Example: basecommercep_extensionname
-
. Media Item PK.p_icon
-
. “1” for the editable item.editable
GenericItems
This table contains the objects of the GenericItem type.-
. Unique identifier (See the section “PKs”) Example: 8796093055075PK
-
.aCLTS
-
.propTS
-
.p_code
-
p_actiontemplate
-
.p_action
-
.p_comparator
-
.p_emptyhandling
-
.p_valuetype
-
.p_searchparametername
-
.p_joinalias
-
.p_lower.p_wherepart
-
.p_typedsearchparameter
-
<strong>p_enclosingtype</strong>`
- GenericItemLP
gentestitems
This table contains the instances of GenericTestItem used in the test classes.-
. Version (bigint). See the section “HJMPTS” Example: 3hjmpTS
-
. Created time (datetime). Example: 2017-07-07 18:22:49.000000createdTS
-
. Modified time (datetime). Example: 2017-07-07 18:33:26.000000modifiedTS
-
. Reference to a type (*types.pk). See the section PKs. Example: 8796093087826TypePkString
-
.OwnerPkString
-
. Unique identifier (See the section “PKs”) Example: 8796093055075PK
-
.aCLTS
-
.propTS
junit_numberseries
NumberSeries is used for JUnit to test a unique PK pool. The structure is the same as in NumberSeries, see below.links
This table is used for the objects of the ExtensibleItem type. This table is used by LinkManager for non-root relations.-
.hjmpTS
-
.createdTS
-
.modifiedTS
-
.TypePkString
-
.OwnerPkString
-
.PK
-
.languagepk
-
.Qualifier
-
.SourcePK
-
.TargetPK
-
.SequenceNumber
-
.RSequenceNumber
-
.aCLTS
-
.propTS
mediaprops
See the information about props table above.-
.hjmpTS
-
.ITEMPK
-
.ITEMTYPEPK
-
.NAME
-
.LANGPK
-
.REALNAME
-
.TYPE1
-
.VALUESTRING1
-
.VALUE1
metainformations
Hybris saves the tenant information here.-
.hjmpTS
-
.<strong>PK</strong>
-
.createdTS
-
.modifiedTS
-
.aCLTS
-
.TypePkString
-
.OwnerPkString
-
.propTS
-
.SystemPK
-
.SystemName
-
This fields says if the system initialized or notisInitialized.
-
.LicenceID
-
.LicenceName
-
.LicenceEdition
-
.AdminFactor
-
.LicenceExpiration
-
.LicenceSignature
numberseries
This table is used by NumberSeries API for generating unique keys. From the service layer, it is used by PersistentKeyGenerator. For example, it is used for orders, comments, media, cronjobs, processes, workflows and workflow actions. See the class NumberSeries for the details. It is used by various components to generate unique identifiers.-
hjmpTS
-
the key. “pk_” is used for generating PKs of the objects.serieskey
-
There are two types, alphanumeric (0) and numeric (1).seriestype
-
Current value. The unique ids are generated in blocks of the specified size, currentvalue contains the last id of the generated sequence.currentValue
-
Template defines the key format. ‘@’ is used to inject PK, $ is used to inject a numeric value of the key. .template
pgrels and orderdiscrels
pgrels is used for the objects of the PrincipalGroupRelation type (extends Link). ordersiscrels is used for the objects of the OrderDiscountRelation type (extends Link). Both have the following structure:-
hjmpTS
-
createdTS
-
modifiedTS
-
TypePkString
-
OwnerPkString
-
PK
-
LanguagePk
-
Qualifier
-
SourcePK
-
TargetPK
-
SequenceNumber
-
RSequenceNumber
-
aCLTS
-
propTS
searchrestrictions
This table is used for the objects of the SearchRestrictions type (extends Link).-
hjmpTS
-
createdTS
-
modifiedTS
-
TypePkString
-
OwnerPkString
-
PK
-
p_extensionname
-
p_autocreate
-
p_generate
-
p_code
-
p_active
-
principal
-
query
-
RestrictedType
-
aCLTS
-
propTS
- SearchRestrictionLP
testitem
TestItem is used in unit tests. There are fields of different types.-
hjmpTS
-
createdTS
-
modifiedTS
-
TypePkString
-
OwnerPkString
-
PK
-
fieldA
-
fieldB
-
fieldBoolean
-
fieldByte
-
fieldCharacter
-
fieldDate
-
fieldDouble
-
fieldFloat
-
fieldInteger
-
fieldLong
-
fieldPrimitiveBoolean
-
fieldPrimitiveByte
-
fieldPrimitiveChar
-
fieldPrimitiveDouble
-
fieldPrimitiveFloat
-
fieldPrimitiveInteger
-
fieldPrimitiveLong
-
fieldPrimitiveShort
-
fieldSerializable
-
fieldString
-
fieldLongString
-
p_testproperty0
-
aCLTS
-
propTS
-
fieldBigDecimal
-
p_testproperty1
-
p_foo
-
p_bar
-
p_xxx
-
p_itemtypetwo
-
p_itemstypetwo
ydeployments
This table contains the information about type deployments: what table corresponds to what type and what configuration options should be taken into account.-
hjmpTS
-
Typecode
-
TableName
-
PropsTableName
-
Name
-
PackageName
-
SuperName
-
ExtensionName
-
Modifiers
-
TypeSystemName
Special attributes
hjmpTS
This attribute contains a number which is used for concurrent writes management. There are some extra information relevant to the topic that you may find useful: “It’s meant to support optimistic locking in the same way as a persistence layer like Hibernate does with a@Version
hjmp.throw.concurrent.modification.exceptions
hjmp.throw.concurrent.modification.exceptions
true
Kiran Kumar Vemula
7 May 2018 at 00:57
Excellent. Good work.
Yury Bich
7 May 2018 at 02:32
“Read more about Generic Audit feature” link is broken.
Devraj Kumar
7 May 2018 at 12:45
Very Very useful information. Thanks