Data Store | Importance

In this blog, I am going to explain what is Data Store, Use Cases and various operations..?

Data Store plays an important role in handling persisted messages or data during integration flows. It allows temporary or long-term storage of messages in a structured way. 

Importance and use cases:

Message Persistence

• Stores messages that may need to be retried in case of failure (e.g., during asynchronous communication).
• Useful in scenarios where message delivery guarantees (like at-least-once delivery) are required.

Decoupling Processing Logic

• Enables decoupling between sender and receiver systems.
• Data can be collected from multiple sources and stored, then processed later in batches or at scheduled times.

Reliable Integration

• Acts as a temporary queue or buffer, improving the reliability of the integration process.
• Helps in recovery from transient failures, since messages can be reprocessed from the data store.

Error Handling and Retry Mechanism

• Failed messages can be stored for later inspection or manual/automated retry.
• Greatly enhances monitoring and troubleshooting capability.

Audit and Tracking

• Supports audit trails, allowing you to track message payloads for compliance or debugging.
• Messages stored can be indexed with metadata (e.g., ID, timestamp) for searching and reporting.

Considerations:

• Storage Limits: Be mindful of the overall storage limit (e.g., 32 GB) for data on your CPI tenant. Data stores are for temporary storage, not for long-term archiving or acting as a persistent database.
• Performance: While data stores offer performance benefits through decoupling, excessive or inefficient use of SELECT operations on very large data stores can impact performance.
• Security: CPI provides options to encrypt stored messages for enhanced security.

Types of operations:

1. Write
2. Get
3. Select
4. Delete

Write Operation: It stores entries in the data store.

• Data Store Name: Name of the data store where you want to store the message.
• Visibility: Indicates if the data store is only visible within the integration flow where it is defined or Global - to all the integration flows that are deployed on that tenant.
• Entry ID: A unique identifier to identify the entry. It can be picked from a header or from the message using XPath. If nothing is provided, the system generates a random GUID for this entry.
• Retention Threshold for Alerting (in d): Defines the number of days by when the entry must be read. Else an alert is generated in the cloud platform account. Default – 2days.
• Expiration Period (in d): The number of days (from when the entry is written into the data store) after which the entry must be deleted from the data store.
• Encrypt Stored Message: Check this if you want to encrypt the message before storing.
• Overwrite Existing Message:  Check this if you want to overwrite a message (if one already exists) with the same Entry ID in this data store.

Get Operation: To retrieve one entry from the data store

• Data Store Name:  The name of the data store from where the entry needs to be retrieved.
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Entry ID: the unique identifier of the entry you want to retrieve from the data store. If this is left empty, the last entry that was added to the data store is retrieved. It can also be picked from a header or from the message using XPath.
• Delete On Completion: check this if you want to delete the entry from the data store after retrieving.

Select Operation: To retrieve multiple messages from the data store in one bulk.

• Data Store Name: The name of the data store from where the entries need to be retrieved
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Number of Polled Messages: enter the number of messages that you would like to retrieve from the data store.
• Delete On Completion: check this if you want to delete the entry from the data store after retrieving.

Delete Operation: To delete messages from the data store.

• Data Store Name:  The name of the data store from where an entry needs to be deleted.
• Visibility: indicates if the data store is visible only to this integration flow or to all the integration flows on the tenant.
• Entry ID: the unique identifier of the entry you want to delete from the data store. If this is left empty, the last entry that was added to the data store is retrieved. It can also be picked from a header or from the message using XPath.

Notes:

• Never use the name sap_global_store for your data store, because it is used by the system to store variables created by the Write Variable step.
• If the message processing fails, the transaction is rolled back and the entry added to the data store will be deleted.
• If you try to Write an entry to the data store with an entry ID that already exists and “Overwrites Existing Message” is not checked, the message processing will fail.

Thanks for reading :-)