Delta Loader Scenarios¶
The following scenarios are examples to help understanding how the Delta Loader operates in different situations.
Table Schema Assumptions
Throughout these scenarios, we use a consistent source table schema for clarity:
Source Table Schema
| Column | Description | Constraints |
|---|---|---|
id_col |
Primary key column | Not null, unique |
partition_col |
Partitioning column | Not null |
value_col |
Data values column | No constraints |
For simplicity, we assume the target table always has the same schema as the source table.
Delta Load Scenarios¶
Case 1: Create Fresh Delta Transaction
Scenario Setup
The source table is created and initial data is inserted.
Table History:
| Version | Action |
|---|---|
| 0 | CREATE TABLE |
| 1 | INSERT INTO |
Initial Metadata State:
The Delta CDF Loader metadata table is empty for the target table identifier.
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| empty |
Expected Result
- Data is read from version 0 to version 1
- Metadata table is updated with the transaction record
Final Metadata State:
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| 0 | 1 | true | false |
Case 2: Delta Load Without Changes
Scenario Setup
Following Case 1, no new changes have been made to the source table.
Table History:
| Version | Action |
|---|---|
| 0 | CREATE TABLE |
| 1 | INSERT INTO |
Initial Metadata State:
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| 0 | 1 | true | false |
Expected Result
- No new data is read (table unchanged)
- New metadata entry tracks the "no-change" state
Final Metadata State:
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| 0 | 1 | true | false |
| 1 | 1 | true | false |
Case 3: Merge with Deduplication
Scenario Setup
Initial data is inserted, then some rows are updated via MERGE operation.
Table History:
| Version | Action |
|---|---|
| 0 | CREATE TABLE |
| 1 | INSERT INTO |
| 2 | MERGE INTO |
Initial Metadata State:
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| empty |
Expected Result
- Inserted and updated rows are read from the source
- Data is deduplicated (latest version per
id_col) - Deduplicated data is merged into target table
Final Metadata State:
| start_commit_ver | end_commit_ver | is_processed | is_stale |
|---|---|---|---|
| 0 | 2 | true | false |
Delete Operations Limitation
The Delta CDF Loader currently does not support deletes on the target table. Any rows deleted in the source table will still be written to the target table or remain there.
Case 1: Create Fresh Timestamp Transaction
Scenario Setup
The source table is created and initial data is inserted.
Table History:
| Version | Action |
|---|---|
| 0 | CREATE TABLE |
| 1 | INSERT INTO |
Initial Metadata State:
The Delta Loader metadata table is empty for the target table identifier.
| last_read_timestamp | is_processed | is_stale |
|---|---|---|
| empty |
Expected Result
- Data is read from 2025-01-01 to 2025-01-02
- Metadata table is updated with the last read timestamp
Final Metadata State:
| last_read_timestamp | is_processed | is_stale |
|---|---|---|
| 2025-01-02 | true | false |
Case 2: Timestamp Load Without Changes
Scenario Setup
Following Case 1, no new data has been added to the source table within the timestamp range.
Table History:
| Version | Action |
|---|---|
| 0 | CREATE TABLE |
| 1 | INSERT INTO |
Initial Metadata State:
| last_read_timestamp | is_processed | is_stale |
|---|---|---|
| 2025-01-02 | true | false |
Expected Result
- No new data is read (no changes in timestamp range)
- New metadata entry tracks the "no-change" state
Final Metadata State:
| last_read_timestamp | is_processed | is_stale |
|---|---|---|
| 2025-01-02 | true | false |
| 2025-01-02 | true | false |