Files
MustansirandAmaan Ullah b6389e2419 [INS-104] Support units in S3 source (#4560)
* implemented Source unit for S3.
Implemented. integration test

* use bucket as source unit

* remove code duplication, reuse from Chunks

* remove unnecessary change

* remove unused functions

* revisit tests

* revert unnecessary change

* change SourceUnitKind to s3_bucket

* handle nil objectCount inside scanBucket

* handle nil objectCount outside loop

* add bucket to resume log

* add bucket and role to error log, remove enumerating log

* implement sub unit resumption

* add comment to checkpointer for unit scans

* implement SourceUnitUnmarshaller on source with the new S3SourceUnit, add test to test resumption on multiple buckets with concurrent ChunkUnit processing

* add role to SourceUnitID

* Revert "add role to SourceUnitID"

This reverts commit 549e6bede9fe4ed276f7a23f5a5b2bf037133264.

* add role to source unit ID, keep track of resumption using source unit ID instead of just bucket name

* rename bucket -> unitID in UnmarshalSourceUnit

---------

Co-authored-by: Amaan Ullah <[email protected]>
2025-12-04 12:54:28 +05:00
..