Connecting the Dots with SAP Cloud Platform Integration

Integration topics are one of the cornerstones of any e-commerce solutions. The challenge of integrating applications and systems is growing with the number of customizations and systems involved. There is a great demand for customer self-service and automation which would not be possible without deep integration of ERP and e-commerce platform. There are different ways to integrate SAP Commerce Cloud with SAP and non-SAP systems. The choice would depend upon the capabilities and limitations of the systems involved. These are related to the integration interfaces, data formats, extensibility, cost of changes, strategic plans and many other factors. One of the most recurrent problems is the integration of SAP Commerce with SAP ERP for master and transactional data. Of course, for SAP Commerce the vendor provided a solution many years ago and keep it updated. However, right now we are facing to a kind of uncertainty because one solution, SAP Commerce Data Hub, is declared retiring and another, Cloud Platform Integration, is defined as strategic and developing that may give the impression that the product is immature and in beta. SAP Cloud Platform Integration is not too young as many from SAP Commerce/Hybris community think. The platform has been continuously evolving since June 2013. It went by another name, SAP HANA Cloud Integration (SAP HCI). The product was renamed at the beginning of 2017 as part of a general rebranding of SAP’s cloud offering. In 2018, SAP SCPI started showing up in the context of Commerce projects. In this article, I want to address the most common myths and objections associated with this product and explain the fundamental concepts behind the solution.

Integration approaches

There are at five six different integration options recommended by SAP for connecting SAP and non-SAP systems:

• SAP App Center – for point-to-point integrations,
• SAP Data Hub – integration middleware, for SAP Commerce,
• Integration API and SAP Cloud Platform Integration Cloud – cloud integration middleware for a wide range of SAP products,
• Platform Extension Factory (project “Kyma”),
• Greenfield/custom.

The following flowchart is presented by SAP to help with decision-making around integrations for SAP Commerce Cloud. What we see here is that for the standard SAP ERP integration and greenfield development, SAP recommends SAP Cloud Platform Integration, #3. However, such approach is considered by many as risky because SCPI is a relatively new word in the Commerce/Hybris area. Many suspect that it brings a bunch of unknowns into the picture and introduce new challenges and limitations. It’s certainly up there, but the devil is not so black as he is painted. Moreover, SAP Cloud Platform Integration is definetely the best choice among the suitable alternatives.

What is Cloud Platform Integration in the nutshell?

Cloud platform integration facilitates the orchestration, routing, processing, transformation of messages between systems. It allows you to integrate processes based on the exchange of messages. It can be understood as a mix of Enterprise Service Bus in the cloud and a kind of ETL system where the extraction step is event-driven and staging (intermediate storage area used for data processing) is architecturally avoided to make the integration processes stateless. The Cloud Platform Integration offers two feature sets: connectivity features (adapters) and data integration (ETL). All adapters and data integration features are tailored for a wide range of SAP products that make the ideal choice for connecting SAP systems with each other. The key capability of SCPI is that it supports the implementation of Enterprise Integration Patterns. For that, it uses Apache Camel under the hood. SCPI resides in the same cloud where SAP Commerce Cloud is. So the integration platform is physically installed in your cloud, contrary to SaaS, where the same middleware is used for a bunch of customers. The resources can be used on-demand. You can flexibly adapt resource consumption according to the business needs. Such additional resources can be allocated quickly because of horizontal scaling and cloud-native architecture of the product.

Key differences with other ESBs

• SCPI is the only integration platform capable to work on SAP Cloud. If you need to replicate data between the cloud-based systems, such as Commerce Cloud and S/4HANA, you need to have the integration platform in the same cloud as they are. If all of your systems are not cloud-based, the SCPI is possibly not the best solution. For the hybrid setups, such as SAP Commerce Cloud and SAP ERP on-premise, other ESBs may work well too, but because of many of them don’t have a ‘starter pack’, it will most likely result in huge customization efforts and performance issues.
• The predefined integration packages are kept up-to-dated by SAP. If a new version of the participating systems is released, the integration package can be also updated by SAP to get the most from the introduced changes. The 3rd party ESBs supports many vendors, and the vendors are not able to keep their connector library up to date. Also, the connectors are often too generic and high-level.
• SCPI is tightly integrated into SAP Cloud that commonizes many processes, such as user provisioning and security settings.

Key difference with SAP Commerce Data Hub

• Changes in the Commerce Data Hub configuration needs more thorough testing than SCPI’s. The customizations in a single class can disrupt the work of the whole platform. SCPI integration flows are executed in an isolated environment. There is a graphical editor to model the message processing steps and specify in detail what happens to the message during processing. It supports versioning and testing.
• You might need to redeploy Commerce Data Hub even for simple data transformations. You need to restart Data Hub to enable any configuration changes. The restart can interfere with the running tasks. The data processing in Data Hub is not transaction-aware. Data Hub doesn’t have a fail-safe way of handling shutdown and this operation may lead to issues with incomplete data and abandoned/unprocessed/unsent objects.
• Commerce Data Hub has limited scalability and flexibility. The common challenge is handling intensive data flows. If Data Hub is to operate at peak performance then sometimes, after a long period of operation, you will need to repair or overhaul data integrity and reconcile the data in the integrating systems.
• Stability of Data Hub relies heavily on data integrity and well-planned cleanup processes. Unlike SCPI, Commerce Data Hub SAP integration architecture involves the use of a staging database where all received objects are stored and compartmentalized to be used in other processes. Internal processes also actively use the database. That often results in poor performance and issues difficult to diagnose and properly treat.

Apache Camel Framework

From a modeling perspective, the integration flow is based on the Business Process Model and Notation (BPMN), a graphical representation for specifying business processes in a business process model. To interpret and execute the integration flows during runtime, Cloud Platform Integration relies on the framework called Apache Camel. Apache Camel is a payload-agnostic message routing and mediation engine. Each message has a header, body (payload) and optional attachments. A header contains additional values associated with the message, in the form of name-value pairs, such as message sender, content encoding, authentication information. In Apache Camel, Body and attachments can have values of any type, but SCPI standard components are taught to work with the XML. Using an open-source framework under the hood makes SCPI extremely extensible and architecturally clear.

Integration Content / Integration templates

The SAP provides predefined integration content as reference templates. The integration flows, defined by these templates, can be extended according to the specific needs of the integration project. The number of integration flows, interfaces, mapping programs, and other objects are collectively define how messages are exchanged through SCPI in the context of the integration scenario. This template library, or Integration Content Catalog, has about 200 integration packages. Each package addresses the integration challenge of a specific business case. Almost all of them are designed for SAP products, and about one half of them involves SAP ERP and SAP S/4 HANA products. SAP Commerce Cloud is involved in about 20 integration packages including the following systems are integration counterparts to SAP Commerce Cloud:

• SAP Marketing Cloud
• SAP ERP
• SAP S/4HANA
• SAP S/4HANA Cloud
• SAP S/4HANA Service
• SAP Customer Data Cloud
• SAP CPQ
• SAP Claims Management
• SAP Customer Engagement Center
• SAP Subscription Billing
• SAP CRM
• SAP Cloud for Customer

The package can be understood as a distributable set of artifacts related to the specific integration. You can copy the package to your project and modify it according to your needs. The packages are updateable by the package vendor (mostly, SAP), but the project copies are not. For example, in April this year, SAP added support for multiple tax countries for material replication, and the integration flow copied into the project was automatically updated if no custom changes were performed against the template. If there are custom changes, the update process has to be manual. There are following types of artifacts the package can consist of:

• Documents
  • Files – a changelog file, for example
  • URLs – mainly to the package documentation and relevant links
• Configuration
  • Integration Flow, which contains such components as
  • BPMN (Business Process Model and Notation) diagram
  • Data schema and data mapping in the EDMX format
  • Data transformation scripts (Groovy)
  • WDSL definitions (XSD schemas) – for connectors
  • Configuration Data
  • Value Mapping definitions
  • OData Service – oData schemas and bindings
  • Data Integration etc.

The integration flows, value mappings and OData service definitions can be downloaded as a zip.

Integration Flow Triggers

You can create an integration flow which:

• Event-driven: Listens to the incoming request, processes it and send data to the target system. So, the flow is triggered by the incoming message or event.
• Scheduled: The system actively checks the data regularly according to the configured schedule (for example, every 30 seconds). The results are delivered to the target system.

For example, the following process represents the order replication from SAP Commerce to SAP ERP where the event-driven approach is implemented. Each block in this process is configurable. The arrows represent the data flows. The inbound interface is OData, the outbound interface is IDOC. OData is a protocol for building and using RESTful APIs that the SAP Commerce Integration API uses and extends. It is a mechanism of choice for SAP now. The Commerce Cloud Integration API module is a new set of extensions providing interfaces for data integration with SAP Commerce Cloud, both inbound and outbound. Previously we have IMPEX, OCC and Platform Web Services for all integration tasks. The key difference between OData-based Integration API and “old-school” OCC/Platform Web Services is that the first integration method is completely data-driven, opposite to old approach where Java development was required to introduce new data management interfaces.

Message Processing Steps

SCPI supports a wide range of integration patterns. Each pattern has a specific set of processing steps. The following message processing steps are currently supported:

• Participants. Sender and Receiver.
• Process. Enables you to define subprocesses and exception processes.
• Event. Start/end events, Start/End message, Error start/end, Escalation, a Terminate message event and a Timer.
• Mapping. The message, XSLT, and operation mapping.
• Transformation. Message transformers, such as Content Modifier, Converters (CSV, EDI, JSON, XML), Encoders and Decoders (Base64, Gzip, MIME, ZIP),
• Scripts (Groovy and JS), Filter, and Message Digest.
• Calls. External (Send, Content Enricher, and Request-Reply) and Local (Process call and Looping Process Call)
• Routing. Aggregator, Gather, Join, Multicast (Sequential and Parallel), Router, and Splitter (EDI, general, IDOC, Iterating, PKCS#7/CMS).
• Security. Encryptors and Decryptors (PKCS7, PGP), Signer and Verifier (PKCS7, XML Digital Signer)
• Persistence. Data Store CRUD Operations, Persist, Write Variables.
• XML Validator.

Message Flows and Channels are used to connect the steps which each other. The channels can be configurable. The channels can be inbound and outbound.

Inbound adapters (channels)

The incoming data come from the Sender channel via one of the “sender adapters”. Using them you can specify which technical protocols should be used to connect a sender to the integration flow. SCPI comes with the out-of-the-box adapters for HTTPS, IDOC, JMS, Mail, OData, SFTP, SOAP, XI, and some others. For example, the SFTP adapter can check for the new files every 10 seconds, pull them into the system and push the contents to the next component. It is similar to how Hot Folder works. After the file is pulled from the SFTP, the system can move or remove the file or check it as processed in the local repository. The HTTPS sender adapter allows you to accept incoming HTTPS requests on a specific address. For example, such a service can be exposed as a REST API. For processing such requests, you can add a Script component with which can convert incoming data info a serialized stream for consumption by the subsequent components in the flow. There is a “CSV to XML converter” component which can transform comma-separated data into the XML data which are understood by, for example, the message mapping component. One of the important adapters is IDOC which is a common format used to pass the data across SAP systems. For the asynchronous master data replication, the system uses IDOC adapters to get connected to SAP ERP, and OAuth adapters to interact with the SAP Commerce Cloud. The SOAP adapter is used for the SOAP protocol supported by many 3rd party systems for data integration. OData (stands for Open Data Protocol) is a relatively new protocol SAP adopted for data access. This REST-based protocol is built on standardized technologies such as HTTP, Atom/XML, and JSON. The main difference from other REST-based web services is in that it provides a uniform way to describe both the data and the data model. In our example, the OData inbound adapter is used. It is configured using an EDMX schema that is based on the SAPCpiOutboundOrder item type (SAP Commerce Cloud). The EDMX is a container for all things about the Order data model. This structure is pre-generated from the WSDL with the use of Import Wizard.

Outbound adapters (channels)

The outbound adapters are used to send data to the partner systems or services. SCPI supports OData, HTTP, IDOC, JMS, LDAP, Mail, ODC, SFTP, SOAP, XI, SAP RFC/BAPI, social networks (Facebook and Twitter), sending data to the subprocess (ProcessDirect).

Data transformation

After receiving some input from the Sender, the SCPI passes it through an integration flow. In our example, the following components are in this chain:

• Sender adapter – OData. We discussed it above. It contains an order from SAP Commerce. Configuration. It is the instance of the Content Modifier component. In this flow, it enriches the order structure received from SAP Commerce.
• Log XML Payload. This is the instance of the Groovy Script component. The script writes the payload from the OData service to the log file to ease troubleshooting.
• Mapping. It defines which and how data from the inbound structure should be mapped to the outbound structure (IDOC in our example). This is an instance of Message Mapping.
• Log IDOC. This is the instance of the Groovy Script component. The script writes the IDOC structure in the log.
• Read credentials. This is the instance of the Groovy Script component. The script reads credentials from the SCPI configuration and adds an HTTP Header (“Authorization”) to the message with the encoded username and password.
• Request/Reply ERP. This component sends the IDOC message to the SAP ERP. The URL is taken from the Integration flow configuration.
• Process ERP reply. This component transforms the XML from the service using the XLT transformation.
• OData Response message. This Content Modifier component creates a response message to the OData call.
• Log OData Response. This is the instance of the Groovy Script component. The script writes the OData response to the log file to ease troubleshooting.

The steps above can be simplified as “Create IDOC (Mapping) -> Sending IDOC”. The integration flow creates an OData listener which trigger the IDOC export if the order is placed. The IDOC structure is created using a Message Mapping component. It declares the rules on what fields from the source structure (from OData) are converted to what fields of the target structure (IDOC). The majority of mappings are simple, 1:1. If necessary, additional transformations can be applied. For example, EntryNumber -> E1SALESORDER_CREATEFROMDAT2/ E1BPSDITM/ PO_ITM_NO has a formula: PO_ITM_NO = formatNumber(EntryNumber + 1, template=“000000”, separator=“,”) For the following ProductCode->E1SALESORDER_CREATEFROMDAT2/ E1BPSDITM/ MATERIAL mapping, there is a condition “if length(productCode)>18, return empty” All these transformations can be changed or extended according to the project-specific customizations.

Tuning/Configuring the Integration Flow

There are two approaches to configuring the integration package imported (copied) from the integration content library: content edit and configure-only. The configure-only option provides an easy-to-use method of adapting an integration flow to the specific requirements. The developer can expose some configurable attributes (externalized parameters), such as mapping configuration, adapter-specific attributes, such as adapter-specific endpoints, or timer start event (scheduler). Externalizing a parameter helps to avoid hard-coded values om the integration flow and provides the flexibility to change the parameter values at the configuration time.

Synchronous and Asynchronous processing

The order replication integration process showed above as an example is synchronous. A sender, SAP Commerce Cloud, opens a connection to the SAP Cloud Platform Integration and sends a request message, with the order in the payload. After sending this message, Commerce Cloud doesn’t close the connection because a reply is expected. SAP Cloud Platform Integration opens a connection to the receiver, which is an SAP ERP and sends the transformed message to the IDOC channel on the SAP ERP side. The connection from Commerce Cloud to Cloud Integration is still open, as long as message processing is ongoing. Asynchronous message handling would work differently. After SAP Cloud Platform Integration receives the message correctly, and the message is valid, it acknowledges its reception to Commerce Cloud. After receiving the acknowledgment, Commerce Cloud closes the connection. With this approach, the connections are closed immediately after the reception is confirmed by the receiving party. Cloud Platform Integration supports both methods. For the order replication, the integration content developers chose the synchronous process (OData+IDOC), while for the master data replication the asynchronous message handling is used (IDOC-based).

SCPI: SAP Commerce and SAP ERP Integration

The SAP provides a pre-defined package for SAP ERP/Commerce Cloud integration. It contains the following integration flows:

Method	What	From	To
ASYNC	Products (Materials) From ERP’s MATMAS05, CLFMAS To CC’s Products, ERPVariantProducts via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Product Classification  From ERP’s CHRMAS, CLFMAS, CLSMAS IDocs To CC’s Attributes, ClassAttributeAssignments, ClassificationAttributeValues, ClassificationClass via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Price Row and Discount Row  From ERP’s COND_A04 IDocs To CC’s PriceRows and DiscountRows via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	B2B Customers  From ERP’s ADR3MAS, ADRMAS, DEBMAS07 IDocs to CC’s B2BUnit, B2BCustomer via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	B2C Customer Creation Confirmations From ERP’s DEBMAS07 To CC’s Customers via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Stock Level From ERP’s LOISTD01 IDocs To CC’s StockLevels via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	OMS Order Notification From ERP’s ORDERS05 and DELVRY07 IDocs To CC’s Order and OrderEntry via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Cart and Checkout Order Notifications From ERP’s DELVRY07 and ORDERS05 To CC’s Orders via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Invoices  From ERP’s INVOIC02 IDocs To CC’s SapB2BDocuments via OData	SAP ERP	SAP Commerce Cloud
ASYNC	Return Order Notifications From ERP’s DELVRY07 and ORDERS05 To CC’s ReturnRequests via OData	SAP ERP, S/4HANA	SAP Commerce Cloud
ASYNC	Order Cancellations From CC’s OrderCancellations via OData To CC’s Orders via OData	SAP Commerce Cloud	SAP ERP, S/4HANA
SYNC/ASYNC	Orders From CC’s Order via OData To ERP’s SALESORDER_CREATEFROMDAT202	SAP Commerce Cloud	SAP ERP, S/4HANA
SYNC/ASYNC	Orders with Product Configuration From CC’s Orders via OData To ERP’s SALESORDER_CREATEFROMDAT202	Commerce Cloud	ERP, S/4HANA
SYNC/ASYNC	B2C-Customer From CC’s B2CCustomer via OData To ERP’s ORDERS05	Commerce Cloud	ERP, S/4HANA
SYNC/ASYNC	B2B Customers From CC’s B2BCustomer via OData To ERP’s ADR2MAS03, ADR3MAS, DEBMAS07	Commerce Cloud	ERP, S/4HANA

Conclusion

SAP Cloud Platform Integration is a mature integration solution with a rich feature set. It is natively integrated into the cloud environment where SAP cloud products are, that makes it the best option in terms of performance, scalability, ease of deployment, stability, and reliability. It supports subscription and consumption-based licensing that gives a lot of flexibility and provides a “pay-as-your-business-grows” model. There is an extendable starter pack for all key SAP integration scenarios.