Connecting the Dots with SAP Cloud Platform Integration

Integration topics are one of the cornerstones of any e-commerce solution. The challenge of integrating applications and systems grows with the number of customizations and systems involved. There is great demand for customer self-service and automation, which would not be possible without deep integration between ERP and an e-commerce platform.

There are different ways to integrate SAP Commerce Cloud with SAP and non-SAP systems. The choice depends on the capabilities and limitations of the systems involved. These are related to integration interfaces, data formats, extensibility, cost of changes, strategic plans, and many other factors.

One of the most recurring problems is the integration of SAP Commerce with SAP ERP for master and transactional data. Of course, for SAP Commerce, the vendor provided a solution many years ago and has kept it updated. However, right now we are facing a kind of uncertainty because one solution, SAP Commerce Data Hub, has been declared retiring, while another, Cloud Platform Integration, is defined as strategic and developing, which may give the impression that the product is immature and in beta.

SAP Cloud Platform Integration is not as young as many in the SAP Commerce/Hybris community think. The platform has been continuously evolving since June 2013. It went by another name: SAP HANA Cloud Integration (SAP HCI). The product was renamed at the beginning of 2017 as part of a general rebranding of SAP’s cloud offering. In 2018, SAP SCPI started showing up in the context of Commerce projects.

In this article, I want to address the most common myths and objections associated with this product and explain the fundamental concepts behind the solution.

Integration approaches

There are five different integration options recommended by SAP for connecting SAP and non-SAP systems:

• SAP App Center — for point-to-point integrations,
• SAP Data Hub — integration middleware for SAP Commerce,
• Integration API and SAP Cloud Platform Integration Cloud — cloud integration middleware for a wide range of SAP products,
• Platform Extension Factory, project “Kyma”,
• Greenfield/custom.

The following flowchart is presented by SAP to help with decision-making around integrations for SAP Commerce Cloud.

What we see here is that for standard SAP ERP integration and greenfield development, SAP recommends SAP Cloud Platform Integration, #3. However, this approach is considered risky by many because SCPI is a relatively new term in the Commerce/Hybris area. Many suspect that it brings a bunch of unknowns into the picture and introduces new challenges and limitations.

It is certainly true to an extent, but the devil is not so black as he is painted. Moreover, SAP Cloud Platform Integration is definitely the best choice among the suitable alternatives.

What is Cloud Platform Integration in a nutshell?

Cloud Platform Integration facilitates the orchestration, routing, processing, and transformation of messages between systems. It allows you to integrate processes based on the exchange of messages. It can be understood as a mix of an Enterprise Service Bus in the cloud and a kind of ETL system, where the extraction step is event-driven and staging — the intermediate storage area used for data processing — is architecturally avoided to make the integration processes stateless.

Cloud Platform Integration offers two feature sets: connectivity features, or adapters, and data integration, or ETL. All adapters and data integration features are tailored for a wide range of SAP products, which makes them an ideal choice for connecting SAP systems with each other.

The key capability of SCPI is that it supports the implementation of Enterprise Integration Patterns. For that, it uses Apache Camel under the hood.

SCPI resides in the same cloud where SAP Commerce Cloud is. So the integration platform is physically installed in your cloud, contrary to SaaS, where the same middleware is used for a bunch of customers. The resources can be used on demand. You can flexibly adapt resource consumption according to business needs. Such additional resources can be allocated quickly because of the product’s horizontal scaling and cloud-native architecture.

Key differences from other ESBs

• SCPI is the only integration platform capable of working on SAP Cloud. If you need to replicate data between cloud-based systems, such as Commerce Cloud and S/4HANA, you need to have the integration platform in the same cloud as they are. If all of your systems are not cloud-based, SCPI is possibly not the best solution. For hybrid setups, such as SAP Commerce Cloud and SAP ERP on-premise, other ESBs may work well too, but because many of them do not have a “starter pack,” it will most likely result in huge customization efforts and performance issues.
• The predefined integration packages are kept up to date by SAP. If a new version of the participating systems is released, the integration package can also be updated by SAP to get the most from the introduced changes. Third-party ESBs support many vendors, and the vendors are not able to keep their connector libraries up to date. Also, the connectors are often too generic and high-level.
• SCPI is tightly integrated into SAP Cloud, which commonizes many processes, such as user provisioning and security settings.

Key differences from SAP Commerce Data Hub

• Changes in the Commerce Data Hub configuration need more thorough testing than SCPI’s. Customizations in a single class can disrupt the work of the whole platform. SCPI integration flows are executed in an isolated environment. There is a graphical editor to model the message processing steps and specify in detail what happens to the message during processing. It supports versioning and testing.
• You might need to redeploy Commerce Data Hub even for simple data transformations. You need to restart Data Hub to enable any configuration changes. The restart can interfere with running tasks. Data processing in Data Hub is not transaction-aware. Data Hub does not have a fail-safe way of handling shutdown, and this operation may lead to issues with incomplete data and abandoned, unprocessed, or unsent objects.
• Commerce Data Hub has limited scalability and flexibility. The common challenge is handling intensive data flows. If Data Hub is to operate at peak performance, then sometimes, after a long period of operation, you will need to repair or overhaul data integrity and reconcile the data in the integrating systems.
• The stability of Data Hub relies heavily on data integrity and well-planned cleanup processes. Unlike SCPI, the Commerce Data Hub SAP integration architecture involves the use of a staging database where all received objects are stored and compartmentalized to be used in other processes. Internal processes also actively use the database. That often results in poor performance and issues that are difficult to diagnose and properly treat.

Apache Camel Framework

From a modeling perspective, the integration flow is based on Business Process Model and Notation (BPMN), a graphical representation for specifying business processes in a business process model. To interpret and execute the integration flows during runtime, Cloud Platform Integration relies on the framework called Apache Camel.

Apache Camel is a payload-agnostic message routing and mediation engine. Each message has a header, body (payload), and optional attachments. A header contains additional values associated with the message in the form of name-value pairs, such as message sender, content encoding, and authentication information. In Apache Camel, the body and attachments can have values of any type, but SCPI standard components are designed to work with XML.

Using an open-source framework under the hood makes SCPI extremely extensible and architecturally clear.

Integration Content / Integration templates

SAP provides predefined integration content as reference templates. The integration flows defined by these templates can be extended according to the specific needs of the integration project. The number of integration flows, interfaces, mapping programs, and other objects collectively define how messages are exchanged through SCPI in the context of the integration scenario.

This template library, or Integration Content Catalog, has about 200 integration packages. Each package addresses the integration challenge of a specific business case. Almost all of them are designed for SAP products, and about one half of them involve SAP ERP and SAP S/4HANA products. SAP Commerce Cloud is involved in about 20 integration packages, including the following systems as integration counterparts to SAP Commerce Cloud:

• SAP Marketing Cloud
• SAP ERP
• SAP S/4HANA
• SAP S/4HANA Cloud
• SAP S/4HANA Service
• SAP Customer Data Cloud
• SAP CPQ
• SAP Claims Management
• SAP Customer Engagement Center
• SAP Subscription Billing
• SAP CRM
• SAP Cloud for Customer

The package can be understood as a distributable set of artifacts related to the specific integration. You can copy the package to your project and modify it according to your needs.

The packages are updateable by the package vendor, mostly SAP, but the project copies are not. For example, in April this year, SAP added support for multiple tax countries for material replication, and the integration flow copied into the project was automatically updated if no custom changes were performed against the template. If there are custom changes, the update process has to be manual.

The package can consist of the following types of artifacts:

• Documents
  • Files — a changelog file, for example
  • URLs — mainly links to the package documentation and relevant resources
• Configuration
  • Integration Flow, which contains such components as
  • BPMN (Business Process Model and Notation) diagram
  • Data schema and data mapping in the EDMX format
  • Data transformation scripts (Groovy)
  • WSDL definitions (XSD schemas) — for connectors
  • Configuration Data
  • Value Mapping definitions
  • OData Service — OData schemas and bindings
  • Data Integration, etc.

The integration flows, value mappings, and OData service definitions can be downloaded as a zip.

Integration Flow Triggers

You can create an integration flow that is:

• Event-driven: Listens to the incoming request, processes it, and sends data to the target system. So the flow is triggered by the incoming message or event.
• Scheduled: The system actively checks the data regularly according to the configured schedule, for example, every 30 seconds. The results are delivered to the target system.

For example, the following process represents order replication from SAP Commerce to SAP ERP, where the event-driven approach is implemented.

Each block in this process is configurable. The arrows represent the data flows. The inbound interface is OData; the outbound interface is IDOC. OData is a protocol for building and using RESTful APIs that the SAP Commerce Integration API uses and extends. It is the mechanism of choice for SAP now.

The Commerce Cloud Integration API module is a new set of extensions providing interfaces for data integration with SAP Commerce Cloud, both inbound and outbound. Previously, we had IMPEX, OCC, and Platform Web Services for all integration tasks. The key difference between the OData-based Integration API and “old-school” OCC/Platform Web Services is that the first integration method is completely data-driven, as opposed to the old approach where Java development was required to introduce new data management interfaces.

Message Processing Steps

SCPI supports a wide range of integration patterns. Each pattern has a specific set of processing steps. The following message processing steps are currently supported:

• Participants. Sender and Receiver.
• Process. Enables you to define subprocesses and exception processes.
• Event. Start/end events, Start/End message, Error start/end, Escalation, a Terminate message event, and a Timer.
• Mapping. The message, XSLT, and operation mapping.
• Transformation. Message transformers, such as Content Modifier, Converters (CSV, EDI, JSON, XML), Encoders and Decoders (Base64, Gzip, MIME, ZIP).
• Scripts (Groovy and JS), Filter, and Message Digest.
• Calls. External (Send, Content Enricher, and Request-Reply) and Local (Process call and Looping Process Call).
• Routing. Aggregator, Gather, Join, Multicast (Sequential and Parallel), Router, and Splitter (EDI, general, IDOC, Iterating, PKCS#7/CMS).
• Security. Encryptors and Decryptors (PKCS7, PGP), Signer and Verifier (PKCS7, XML Digital Signer).
• Persistence. Data Store CRUD Operations, Persist, Write Variables.
• XML Validator.

Message Flows and Channels are used to connect the steps with each other. The channels can be configurable.

The channels can be inbound and outbound.

Inbound adapters (channels)

The incoming data comes from the Sender channel via one of the “sender adapters.” Using them, you can specify which technical protocols should be used to connect a sender to the integration flow. SCPI comes with out-of-the-box adapters for HTTPS, IDOC, JMS, Mail, OData, SFTP, SOAP, XI, and some others. For example, the SFTP adapter can check for new files every 10 seconds, pull them into the system, and push the contents to the next component. It is similar to how Hot Folder works. After the file is pulled from the SFTP, the system can move or remove the file, or mark it as processed in the local repository.

The HTTPS sender adapter allows you to accept incoming HTTPS requests on a specific address. For example, such a service can be exposed as a REST API. To process such requests, you can add a Script component that can convert incoming data into a serialized stream for consumption by the subsequent components in the flow. There is a “CSV to XML converter” component that can transform comma-separated data into XML data, which is understood by, for example, the message mapping component.

One of the important adapters is IDOC, which is a common format used to pass data across SAP systems. For asynchronous master data replication, the system uses IDOC adapters to connect to SAP ERP and OAuth adapters to interact with SAP Commerce Cloud.

The SOAP adapter is used for the SOAP protocol supported by many third-party systems for data integration.

OData, which stands for Open Data Protocol, is a relatively new protocol SAP adopted for data access. This REST-based protocol is built on standardized technologies such as HTTP, Atom/XML, and JSON. The main difference from other REST-based web services is that it provides a uniform way to describe both the data and the data model.

In our example, the OData inbound adapter is used. It is configured using an EDMX schema that is based on the SAPCpiOutboundOrder item type in SAP Commerce Cloud. The EDMX is a container for all things related to the Order data model. This structure is pre-generated from the WSDL using the Import Wizard.

Outbound adapters (channels)

The outbound adapters are used to send data to the partner systems or services.

SCPI supports OData, HTTP, IDOC, JMS, LDAP, Mail, ODC, SFTP, SOAP, XI, SAP RFC/BAPI, social networks (Facebook and Twitter), and sending data to the subprocess (ProcessDirect).

Data transformation

After receiving some input from the Sender, SCPI passes it through an integration flow.

In our example, the following components are in this chain:

• Sender adapter — OData. We discussed it above. It contains an order from SAP Commerce. Configuration. It is an instance of the Content Modifier component. In this flow, it enriches the order structure received from SAP Commerce.
• Log XML Payload. This is an instance of the Groovy Script component. The script writes the payload from the OData service to the log file to ease troubleshooting.
• Mapping. It defines which data from the inbound structure should be mapped to the outbound structure, and how, IDOC in our example. This is an instance of Message Mapping.
• Log IDOC. This is an instance of the Groovy Script component. The script writes the IDOC structure to the log.
• Read credentials. This is an instance of the Groovy Script component. The script reads credentials from the SCPI configuration and adds an HTTP Header (“Authorization”) to the message with the encoded username and password.
• Request/Reply ERP. This component sends the IDOC message to SAP ERP. The URL is taken from the integration flow configuration.
• Process ERP reply. This component transforms the XML from the service using XSLT transformation.
• OData Response message. This Content Modifier component creates a response message to the OData call.
• Log OData Response. This is an instance of the Groovy Script component. The script writes the OData response to the log file to ease troubleshooting.

The steps above can be simplified as “Create IDOC (Mapping) -> Sending IDOC.” The integration flow creates an OData listener that triggers the IDOC export if the order is placed.

The IDOC structure is created using a Message Mapping component. It declares the rules for which fields from the source structure, from OData, are converted to which fields of the target structure, IDOC.

The majority of mappings are simple, 1:1. If necessary, additional transformations can be applied. For example, EntryNumber -> E1SALESORDER_CREATEFROMDAT2 / E1BPSDITM / PO_ITM_NO has a formula:

PO_ITM_NO = formatNumber(EntryNumber + 1, template=“000000”, separator=“,”)

For the following ProductCode -> E1SALESORDER_CREATEFROMDAT2 / E1BPSDITM / MATERIAL mapping, there is a condition: “if length(productCode) > 18, return empty.”

All these transformations can be changed or extended according to project-specific customizations.

Tuning/Configuring the Integration Flow

There are two approaches to configuring an integration package imported, or copied, from the integration content library: content edit and configure-only. The configure-only option provides an easy-to-use method of adapting an integration flow to specific requirements. The developer can expose some configurable attributes, or externalized parameters, such as mapping configuration, adapter-specific attributes, such as adapter-specific endpoints, or timer start event, or scheduler. Externalizing a parameter helps to avoid hard-coded values in the integration flow and provides the flexibility to change parameter values at configuration time.

Synchronous and Asynchronous processing

The order replication integration process shown above as an example is synchronous. A sender, SAP Commerce Cloud, opens a connection to SAP Cloud Platform Integration and sends a request message with the order in the payload. After sending this message, Commerce Cloud does not close the connection because a reply is expected. SAP Cloud Platform Integration opens a connection to the receiver, which is SAP ERP, and sends the transformed message to the IDOC channel on the SAP ERP side. The connection from Commerce Cloud to Cloud Integration is still open as long as message processing is ongoing.

Asynchronous message handling would work differently. After SAP Cloud Platform Integration receives the message correctly, and the message is valid, it acknowledges its reception to Commerce Cloud. After receiving the acknowledgment, Commerce Cloud closes the connection. With this approach, the connections are closed immediately after reception is confirmed by the receiving party.

Cloud Platform Integration supports both methods. For order replication, the integration content developers chose the synchronous process, OData+IDOC, while for master data replication, asynchronous message handling is used, IDOC-based.

SCPI: SAP Commerce and SAP ERP Integration

SAP provides a predefined package for SAP ERP/Commerce Cloud integration. It contains the following integration flows:

Conclusion

SAP Cloud Platform Integration is a mature integration solution with a rich feature set. It is natively integrated into the cloud environment where SAP cloud products are, which makes it the best option in terms of performance, scalability, ease of deployment, stability, and reliability. It supports subscription- and consumption-based licensing, which gives a lot of flexibility and provides a “pay-as-your-business-grows” model. There is an extendable starter pack for all key SAP integration scenarios.