trufflehog/pkg/sources/s3

S3 source

Scans objects in S3 buckets, or in any S3-compatible service.

Scan paths

The source implements both the plain Source interface and SourceUnitEnumChunker, where one unit is one bucket (plus the role used to reach it).

• Chunks walks every bucket in sequence.
• Enumerate reports each bucket as a unit, and ChunkUnit scans one bucket.

Both end up in scanBucket -> pageChunker, so per-object behavior is identical either way. The CLI always takes the unit path, because main.go enables source units and the source supports them.

Credentials

Set by which flags are present:

Flags	Credentials used
none	anonymous, for public buckets only
--cloud-environment	the AWS SDK's normal chain: env vars, ~/.aws/credentials, instance metadata
--key and --secret	those static credentials
--key, --secret, --session-token	those temporary credentials

The default is anonymous, not your local AWS profile. Scanning a private bucket without --cloud-environment or explicit keys gives 403 AccessDenied.

--role-arn assumes a role before scanning, and can be repeated. With roles and no --bucket, the source tries every bucket each role can list, so AccessDenied on buckets a role cannot reach is expected and does not fail the scan.

Choosing buckets

--bucket scans only the named buckets. --ignore-bucket scans everything else. They cannot be combined; doing so fails at startup. Bucket names are matched exactly, not as globs or prefixes.

Choosing objects within a bucket

Four flags, all repeatable, all matched against the object key before the object is downloaded. A skipped object costs no GetObject request.

• --include-prefix / --exclude-prefix match a literal key prefix. They can be used together to scan a subtree while leaving part of it out, and an exclude match always wins.
• --include-extension / --exclude-extension match the file extension, written without a leading dot. They cannot be used together, since listing what to scan already excludes everything else. Matching is case insensitive.

Prefixes are literal, so --include-prefix=log matches logs/app.txt and logs-archive/app.txt alike.

The two kinds are combined with AND: an object has to pass both to be scanned. Entries are trimmed, and blank ones are ignored.

Filtering happens after listing, so it saves GetObject requests but not ListObjectsV2 requests. A bucket is still paginated in full even when a prefix covers a small part of it. Pushing prefixes into ListObjectsV2Input.Prefix would cut that too, and has not been done.

Three edge cases worth knowing:

• Only the last extension counts. path.Ext("backup.tar.gz") is .gz, so --exclude-extension=tar.gz never matches, while --exclude-extension=gz does.
• A key with no extension, like Makefile, matches no extension entry, so --include-extension skips it and --exclude-extension keeps it.
• A dotfile is treated as all extension. .env has extension env, so --exclude-extension=env skips it, and any --include-extension list that omits env skips it too. That is easy to do by accident, and .env files are a common source of secrets.

Filters apply to object keys only, not to files inside an archive. --exclude-extension=mp4 skips clip.mp4 in the bucket but not a clip.mp4 packed inside media.zip, since archive contents are filtered by the shared deny list instead. Excluding zip avoids expanding the archive at all.

Prefixes are matched with strings.HasPrefix, so an exclude prefix naming one key also excludes anything that key is a prefix of: --exclude-prefix=foo/bar.txt excludes foo/bar.txt.bak as well. There is no exact-key match.

The same prefixes and extensions apply to every bucket in a scan. There is no way to scope a prefix to one bucket.

If a filter excludes every object in a bucket, the scan logs Scanned no objects in bucket with the count, so a mistyped prefix does not look like a clean scan of an empty bucket.

Objects skipped regardless of configuration

• Glacier and Glacier Instant Retrieval storage classes
• Larger than --max-object-size, which defaults to 250 MiB and cannot be raised above it
• Empty
• Extensions in the shared deny list in pkg/common/vars.go, which already covers images, audio, and video
• Keys ending in /, which are directory markers

Custom endpoints

--endpoint points the scan at an S3-compatible service instead of AWS S3. Setting it changes three things:

• The client uses path-style addressing, since these services rarely publish the wildcard DNS that virtual-hosted addressing needs.
• Per-bucket region discovery is skipped. GetBucketRegion is AWS-only, and a custom endpoint serves its own buckets, so requests sign with --region instead.
• Object links in finding metadata point at the endpoint rather than amazonaws.com, so they resolve the way the scan read them.

The endpoint is parsed in Init, so a malformed one fails at startup rather than as an SDK error mid-scan. A missing scheme is assumed to be https.

--region sets the region used to sign requests and defaults to us-east-1. On AWS each bucket's real region is discovered automatically, so it mostly matters with a custom endpoint.

Resumption

Checkpointer tracks which objects in the current page of up to 1000 have finished, and records the highest consecutively completed key as StartAfter in Progress.EncodedResumeInfo, along with the bucket and role. An interrupted scan resumes from there.

Only consecutive completions are checkpointed. If objects 0 to 5 and 7 to 8 are done but 6 is not, the checkpoint stops at 5, so resuming may rescan a few objects but never misses one.

This matters when adding a new skip to pageChunker: a skipped object still has to call UpdateObjectCompletion. Skipping without it stalls the checkpoint at the first skipped object, and a resumed scan then redoes everything after it.

The OSS CLI does not persist progress between runs, so resumption only takes effect where something stores and returns EncodedResumeInfo.