Transaction Handling .?

In this blog, we will see what is Transaction-Handling and how to use it.?

Transaction-Handling:

It refers to the management of a series of operations so they behave as a single unit of work — where either all steps succeed or the process handles failures gracefully to maintain data integrity.

When you enable transaction-handling enabled, it hold locks on resources (e.g., database records, queue entries) for their duration. While this is necessary for consistency, effective transaction management encourages short, well-defined transactions. This minimizes the time resources are locked, reducing contention and improving overall system performance and scalability.

Options:

• No Transaction: Each step is committed independently. Only suitable for very simple, non-critical scenarios where data consistency is not paramount.
• Required for JDBC: This ensures that all operations interacting with CPI's internal database (e.g., Data Store operations, Write Variables used with Data Stores) are part of a single transaction. If one fails, all are rolled back.
• Required for JMS: Similarly, this applies transactional behavior to operations involving JMS queues managed by CPI.

Main Iflow:

Local / Child Iflow:

Transactional processing means that the message (as defined by the steps contained in a process) is processed within one transaction.

For example, consider a process with a Data Store Write operation. If transaction handling is activated, the Data Store entry is only committed if the whole process is executed successfully. In an error case, the transaction is rolled back and the Data Store entry isn't written. If transaction handling is deactivated, the Data Store entry is committed directly when the integration flow step is executed. In an error case, the Data Store entry is nevertheless persisted (and not removed or rolled back).

Note:

• It's crucial to understand that CPI's transaction handling primarily covers its internal resources (like Data Stores and JMS queues). 
• It does not provide a distributed transaction coordinator (like a two-phase commit) across heterogeneous external systems (e.g., atomically committing changes in an SAP ERP system and an external Salesforce system). 

For such complex cross-system consistency, you often need to implement patterns like Saga patterns, compensating transactions, or ensure idempotent operations on the external systems.

In conclusion, transaction handling in SAP CPI is a critical enabler for building reliable, fault-tolerant, and data-consistent integration solutions that can gracefully handle errors and maintain the integrity of your business data across interconnected systems.

Source: community.sap.com

Thanks for reading :-)

Data Store | Importance

In this blog, I am going to explain what is Data Store, Use Cases and various operations..?

Data Store plays an important role in handling persisted messages or data during integration flows. It allows temporary or long-term storage of messages in a structured way. 

Importance and use cases:

Message Persistence

• Stores messages that may need to be retried in case of failure (e.g., during asynchronous communication).
• Useful in scenarios where message delivery guarantees (like at-least-once delivery) are required.

Decoupling Processing Logic

• Enables decoupling between sender and receiver systems.
• Data can be collected from multiple sources and stored, then processed later in batches or at scheduled times.

Reliable Integration

• Acts as a temporary queue or buffer, improving the reliability of the integration process.
• Helps in recovery from transient failures, since messages can be reprocessed from the data store.

Error Handling and Retry Mechanism

• Failed messages can be stored for later inspection or manual/automated retry.
• Greatly enhances monitoring and troubleshooting capability.

Audit and Tracking

• Supports audit trails, allowing you to track message payloads for compliance or debugging.
• Messages stored can be indexed with metadata (e.g., ID, timestamp) for searching and reporting.

Considerations:

• Storage Limits: Be mindful of the overall storage limit (e.g., 32 GB) for data on your CPI tenant. Data stores are for temporary storage, not for long-term archiving or acting as a persistent database.
• Performance: While data stores offer performance benefits through decoupling, excessive or inefficient use of SELECT operations on very large data stores can impact performance.
• Security: CPI provides options to encrypt stored messages for enhanced security.

Types of operations:

1. Write
2. Get
3. Select
4. Delete

Write Operation: It stores entries in the data store.

• Data Store Name: Name of the data store where you want to store the message.
• Visibility: Indicates if the data store is only visible within the integration flow where it is defined or Global - to all the integration flows that are deployed on that tenant.
• Entry ID: A unique identifier to identify the entry. It can be picked from a header or from the message using XPath. If nothing is provided, the system generates a random GUID for this entry.
• Retention Threshold for Alerting (in d): Defines the number of days by when the entry must be read. Else an alert is generated in the cloud platform account. Default – 2days.
• Expiration Period (in d): The number of days (from when the entry is written into the data store) after which the entry must be deleted from the data store.
• Encrypt Stored Message: Check this if you want to encrypt the message before storing.
• Overwrite Existing Message:  Check this if you want to overwrite a message (if one already exists) with the same Entry ID in this data store.

Get Operation: To retrieve one entry from the data store

• Data Store Name:  The name of the data store from where the entry needs to be retrieved.
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Entry ID: the unique identifier of the entry you want to retrieve from the data store. If this is left empty, the last entry that was added to the data store is retrieved. It can also be picked from a header or from the message using XPath.
• Delete On Completion: check this if you want to delete the entry from the data store after retrieving.

Select Operation: To retrieve multiple messages from the data store in one bulk.

• Data Store Name: The name of the data store from where the entries need to be retrieved
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Number of Polled Messages: enter the number of messages that you would like to retrieve from the data store.
• Delete On Completion: check this if you want to delete the entry from the data store after retrieving.

Delete Operation: To delete messages from the data store.

• Data Store Name:  The name of the data store from where an entry needs to be deleted.
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Entry ID: the unique identifier of the entry you want to delete from the data store. If this is left empty, the last entry that was added to the data store is retrieved. It can also be picked from a header or from the message using XPath.

Notes:

• Never use the name sap_global_store for your data store, because it is used by the system to store variables created by the Write Variable step.
• If the message processing fails, the transaction is rolled back and the entry added to the data store will be deleted.
• If you try to Write an entry to the data store with an entry ID that already exists and “Overwrites Existing Message” is not checked, the message processing will fail.

Thanks for reading :-)

Groovy - Script

Introduction

Groovy Script is an important component in the SAP Integration Suite, especially in the place of SAP Cloud Platform Integration (CPI). It's a dynamic scripting language for the Java platform that enhances the productivity of integration developers.

Purpose: Groovy Script is used to implement custom logic during the integration process or mapping. It provides flexibility to manipulate messages, transform data formats, and handle complex integration flows.

Integration: It seamlessly integrates with Java, allowing developers to use existing Java libraries and frameworks in their scripts.

Features: Groovy offers features like easy-to-read syntax, error handling, and native support for XML and JSON, making it ideal for integration jobs.

Use Cases: Common use cases in SAP CPI include message mapping, data transformation, content enrichment and dynamic routing.

Why Groovy Script.?

• In the place of Groovy script, you can choose Jave script also. But Groovy is built on Java Platform and allowing seamless integration with Java libraries and functionalities, crucial in Java-heavy SAP environments.
• Groovy has excellent built-in support for XML and JSON, simplifying parsing and manipulation of these data formats in integration processes.
• Groovy provides superior error handling, especially useful in managing integration-specific exceptions.

Common use cases..

• Dynamic Header and Property manipulation
• Content transformation like XML to JSON, XML to CSV or vice-versa
• Conditional Logic
• Payload Enrichment or cleanup
• Custom Logging
• Data Mapping and Lookup
• Custom Error Handling
• Signature Generation
• Date and Time Calculation
• Complex math calculation

Build a simple flow and place Groovy script step in iFlow and copy/paste the following script..

import com.sap.gateway.ip.core.customdev.util.Message

 

def Message processData(Message message) {

    //code here

    message.setBody('Hello World')

    return message

}

Then, deploy the flow and test it.

Ref.. for more examples:

https://community.sap.com/t5/technology-q-a/best-way-to-learn-groovy/qaq-p/12746937

https://sapzero2hero.com/2022/03/21/sap-cpi-basic-example-groovy-script/

Thanks for reading :-)

XSLT - Mapping

In this blog, we will see what is XSLT mapping and how to use it in SAP CPI.? 

What is XSLT.?

XSLT is Extensible Stylesheet Language Transformations

XSLT is a language designed for transforming XML documents. In SAP Cloud Integration, it allows for dynamic modifications to messages, including altering headers and properties. This capability is particularly useful for applying conditional logic or handling complex transformations that go beyond simple field mapping.

Benefits of XSLT Mapping in SAP CPI

• Data Consistency: Data consistency can be achieved by XSLT mapping, which is one of its main benefits. You can preserve the integrity of your data as it flows across various systems and apps by establishing precise rules for transformation.
• Flexibility: Data transformations are flexible when using XSLT mapping. The mapping rules can be adjusted and changed as your company's needs.
• Integration Capabilities: The smooth integration of SAP CPI with multiple systems is a key feature. This integration is made possible through XSLT mapping, which makes sure that data is formatted correctly for each target system.
• Efficiency: A crucial component of XSLT mapping is efficiency. Processes for transforming data are streamlined, which lowers the chance of mistakes.

Example:

At the target side, or in the output we want that the Employee whose age is not mentioned should be excluded from the xml. So in this case we used the XSLT Mapping.

Incoming payload is:

<root>

<employee>

<name>Jay</name>

<age>20</age>

</employee>

<employee>

<name>Kiran</name>

</employee>

<employee>

<name>Kisan</name>

<age>21</age>

</employee>

</root>

XSLT:

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:template match="node()|@*">

<xsl:copy>

<xsl:apply-templates select="node()|@*"/>

</xsl:copy>

</xsl:template>

<xsl:template match = "employee[not(Age)]"/>

</xsl:stylesheet>

Save the iflow and deploy it. Test it from Postman.

In the response payload, employee details are excluded only when <age> is missing from incoming payload.

Source: community.sap.com

Thanks for reading :-)

Externalization / External Parameters - How to use it

Externalization refers to the practice of storing configuration parameters, credentials, and other dynamic values outside the integration flow (iFlow) itself. This approach provides several important benefits and is crucial for maintainable, secure, and scalable integration design.

Various Features:

Promotes Reusability

• By externalizing values like endpoints, credentials, or message headers, you can reuse the same iFlow across different environments (Dev, Test, Prod) without modifying the iFlow logic.
• This minimizes duplication and development effort.

Environment-Specific Configurations

• Externalized parameters allow you to configure iFlows differently for each environment using configuration values defined in the “Configuration” tab of the iFlow.
• Example: Different API keys, URLs, or proxy settings for Dev and Prod.

Separation of Logic and Configuration

• Keeps business logic clean and separate from configuration data.
• Makes integration flows easier to understand and maintain.

Simplifies Maintenance and Updates

• Updates to configuration (e.g., endpoint changes) don’t require redeploying or modifying the iFlow.
• You can just change the externalized value in the runtime configuration.

Improves Security

• Sensitive values like passwords and tokens can be stored securely using:
• Secure Parameters (in CPI)
• Credential Artifacts (e.g., User Credentials or OAuth2 Client Credentials)

• Reduces the risk of exposing credentials in code.

Supports Parameterization

• Externalization enables dynamic behavior based on parameters (like setting headers or file names).
• You can pass values into iFlows during runtime via headers or query parameters and map them using externalized parameters.

Facilitates DevOps and CI/CD

• Makes it easier to integrate with DevOps pipelines by avoiding hardcoded values.
• Configuration files or scripts can inject the right values during deployment.

How Externalization Works in SAP CPI:

When designing an iFlow, you can mark certain fields (e.g., an adapter's "Address" field, properties in a Content Modifier, or parameters in a script) as "externalized." This creates a parameter that can then be configured with a "Default Value" during design time. Once the iFlow is deployed, you can access the "Configure" view for that iFlow in the CPI tenant and provide "Configured Values" for these externalized parameters. These configured values override the default values at runtime for that specific tenant.

Let's build a small iFlow..

Click on OData connection..

Click on Externalize, as shown above..

And, then specify connection parameters that are being changed based on target environment like QA, PROD etc and leave remaining parameters like time-out, CSRF Protected..etc.

Then, save the iFlow and download it.

Now, go to QA or PROD environment, upload the iFlow 

Select Upload and specify Integration Flow and then click on Add.

Then, click on "Configure" option and select Receiver (drop down) and enter the values as per the target environment (QA or PROD).

Recommendations:

1. Externalize all the fields of the integration flow that you envisioned and provide the appropriate default values.

2. Be sure not to provide the tenant/landscape-specific value as a default parameter value.

3. Validate the default value of parameters through validation checks. Saving the integration flow will run validation checks.

4. Always provide the tenant/landscape-specific value in the Configure view.

5. Before downloading the integration flow or exporting the content package, always leverage the benefit of the Externalized Parameters view to compare the default and configured value of parameters for quality assurance. Update the default parameter value from the Externalized Parameters view or externalization editor for any correction.

6. Download the integration flow with Default Values Only if you do not want to reuse the configurations of the source system while importing into the target system.

7. Download the integration flow from source system with Merged Configured and Default Values if you want to reuse the configurations of the source system while importing into the target system.

Source: community.sap.com

Thanks for reading :-)

 

APIM (API Management) - APIKey Policy

What is an API-M?

In SAP CPI (Cloud Platform Integration), APIM stands for API Management, which is a capability within the SAP Integration Suite. It is designed to help organizations create, publish, and manage APIs that connect applications and systems both within and outside the enterprise. APIM provides a centralized platform to secure APIs with policies (such as authentication, rate limiting, IP whitelisting), monitor API usage, control access, and expose APIs through developer portals for external consumption..

Key components: 

API Provider: Represents backend systems or services such as iFlow end point alias (which will do basic transformation or get the data from back end systems like DB, ECC, S4 Hana, Salesforce or any) or any other system that expose APIs.

API Proxy: The actual API interface for consumers/third pary systems, which are created to apply policies, transformations, and security before exposing the backend API URL.

API Product: A collection or bundle of one or more API proxies made available together to developers.

Security and Monitoring: Features like OAuth2, API keys, rate limiting, IP whitelisting, and built-in protections against attacks to secure APIs, plus analytics and monitoring to track API performance and usage.

Case Study:

1. Create a iflow, which will call back-end system to fetch the data either by using any connector or HTTP call.

Sample third party URL: (dummy)

https://dummy.restapiexample.com/api/v1/employees

(response is subject to URL availability)

Method : GET

Now, deploy it and test it.

2. Create API Provider 

Go to Configure > API and then go to API Provider click on "Create"

Specify the details as shown below, save it and test the connection

(API  - Service Key details..)

3. Create API Proxy

    Click on Create 

Select API from drop down list and click on "Discover"

Select the required one and click on Next

Select OAuth and specify PRT Service Keys - Client ID/ Client-Secret, Token URL ..

Proxy URL will be created as shown below..

Now test the end-point from Postman...

It means, anyone can access. But we need to protect it by using Application Key or APIKey...

For that, we need to create a Product..

4. Go to Engage and click Create..

Enter Name and Tile and Quota and its interval as required..

Go to APIs..select the API and then click on OK.

Then Save it and Publish it to Developer HUB

After successful Publish... you can see it in HUB

Click on the product, then create New Subscription for Application..

Specify the name and save it to get APIKey

This key is Unique to identify end consumer of the API.

In order to accept the request based on APIKey, we need to apply "Verify API Key" policy ..

Go to API Proxy, select the API and click on policies..

Accepting the API Key thru header

 <!--Specify in the APIKey element where to look for the variable containing the api key--> 

<VerifyAPIKey async='true' continueOnError='false' enabled='true' 

xmlns='http://www.sap.com/apimgmt'>

    <APIKey ref='request.header.APIKey '/>

</VerifyAPIKey>

Accepting the API Key thru queryparam

 <!--Specify in the APIKey element where to look for the variable containing the api key--> 

<VerifyAPIKey async='true' continueOnError='false' enabled='true' 

xmlns='http://www.sap.com/apimgmt'>

     <APIKey ref='request.queryparam.APIKey '/>

</VerifyAPIKey>

Save it and deploy the API Proxy.

Now test the API from Postman either APIKey as header or queryparam...

APIKey as queryparam

APIKey as header

That's all.

If you want to see list of subscriptions for the Product..

Go to HUB, select the product..

You can test from API portal also...and we can debug to identify the issue w.r.to policies / conversions and it's useful for complex flows.

Source: SAP Community.

Thanks for reading :-)

Amazon - S3 Integration

In this blog, we will see what is Amazon S3 and how to connect from CPI.

What is Amazon S3?

Amazon S3 is the go-to service for flexible, scalable, and cost-effective object storage in the cloud, forming the backbone for countless applications and data strategies across industries.

Objects: Each piece of data you store in S3 is an object. An object consists of the data itself (which can be any type of file like images, videos, documents, backups, etc.), a unique key (its identifier within the bucket, similar to a file name), and metadata (descriptive information about the object).

Buckets: Buckets are simply containers for your objects. You create buckets, give them unique names, and specify an AWS Region where the bucket will reside.

Use Cases:

1. S3 is very much useful, when you want to pull files from Source and place them in Target bucket.

    S3->CPI->S3

2. After CPI processed data and then place them in a S3 bucket.

Note: S3 is a storage space and can be connected (just like SFTP  server)

Example:

Transfer files from source S3 to target S3 bucket.

On Sender Side:

Specify Connection parameters

Region Name: AWS region where the Bucket reside.

Bucket Name:Name of the bucket on AWS where the Sender Adapter should poll files from.

Polling Interval (in ms) :

Polling Interval in milliseconds.

Default value: 60000

Access Key Alias: Name of the secured parameter which stores AWS Access Key.

Secret Key Alias: Name of the secured parameter which stores AWS Secret Key.

On Receiver Side:

Select Receiver Side Amazon web service connection and specify required parameters.

Specify required criteria as per business requirement. 

That's all, Deploy and test it.

Source: community.sap.com

Thanks for reading :-)

JDBC Adapter - Select, Insert, Delete

In this blog, I will explain you how to create JDBC alias (cloud/on-premise) connection and perform some basic SELECT, INSERT and DELETE Operations.

How to create JDBC Connection alias..?

In real-time, you will get DB details either from infra-team or admin team..

Basically, you need DB Host, Port, User-Name, Password and Database name.

and based on the data base, you need to prepare JDBC URL.

Go to Monitor>Integrations and APIs>Manage Security> JDBC Material

Click on Add, select Database type "PostgreSQL" or SAP HANA Cloud or SAP HANA Platform (On-Premise)..etc

Note: If the Database is On-Premise, then you need to install JDBC Driver.

Specify all the required details and Deploy the connection and note the connection alias name.

In my example, the connection alias is "postgresql_db"

For PostgreSQL, the JDBC URL is ..

JDBC URL: jdbc:postgresql://host:port/database

Based on your database, you need to build the URL (if you're not sure, just google it) :-)

Next, create simple iFlow as shown below..

Inside Content Modifier step, specify your SQL query.

select * from "doc"."emprecord" limit 100;

where "emprecord" is the table name, you need to change it as per your table name.

Content Modifier Step:

JDBC - Connection (specify JDBC Data Source Alias - created earlier)

Once flow is ready, then deploy it and test it.

DB results:

Sample queries for your ref:

• create table emprecord( empid int, emplastname varchar(255), empfirstname varchar(255), empage int);
• insert into "doc"."emprecord"(empid,emplastname, empfirstname, empage) values (1003,'HHR','Lalit',25);
• select empid,emplastname,empfirstname, empage from "doc"."emprecord" limit 100;
• select * from "doc"."emprecord" limit 100;

Thanks for reading :-)