diff --git a/pkg/sources/circleci/README.md b/pkg/sources/circleci/README.md new file mode 100644 index 000000000..9f63901f3 --- /dev/null +++ b/pkg/sources/circleci/README.md @@ -0,0 +1,124 @@ +# CircleCI Source + +## Overview + +The CircleCI source enables TruffleHog to scan CircleCI build data for secrets, credentials, and sensitive data. It scans every project accessible by the provided API token, walking each project's build history, job steps, and build configuration. + +## CircleCI Fundamentals + +### What is CircleCI? + +CircleCI is a continuous integration and delivery (CI/CD) platform that automates building, testing, and deploying software. Pipelines are triggered by VCS events (e.g., a GitHub or Bitbucket push) and run a sequence of jobs and steps defined in a project's `config.yml`. + +### Key CircleCI Terminology + +| Term | Description | +|------|-------------| +| **Project** | A repository connected to CircleCI, identified by VCS type, org/username, and repo name | +| **Build** | A single execution of a project's pipeline, identified by an incrementing build number | +| **Job** | A defined unit of work within a build (e.g., `build`, `test`, `deploy`) | +| **Step** | An individual command or action executed within a job | +| **Action** | The underlying execution unit CircleCI's v1.1 API returns per step, with a link to its output | +| **circle.yml** | The build configuration used for a given build (legacy v1.1 naming for `config.yml`) | +| **API Token** | A personal or project token used to authenticate against the CircleCI API | + +## Features + +- **Full Project Discovery**: Automatically lists every project visible to the provided token +- **Build History Scanning**: Iterates through all builds for each discovered project +- **Job Step Output Scanning**: Fetches and scans the output of every action within every build step +- **Build Configuration Scanning**: Scans the `circle.yml`/`config.yml` used for each build +- **Concurrent Processing**: Projects are scanned in parallel using a bounded worker pool +- **CIRCLE_SHA1 Redaction**: Strips the `CIRCLE_SHA1=` environment line from scanned output to avoid noisy commit-SHA false positives + +## Configuration + +### Authentication + +CircleCI only supports token-based authentication. + +**CLI Usage:** +```bash +trufflehog circleci --token cci_xxxxxxxxxxxxxxxxxxxx +``` + +The token can also be provided via the `CIRCLECI_TOKEN` environment variable: + +```bash +export CIRCLECI_TOKEN=cci_xxxxxxxxxxxxxxxxxxxx +trufflehog circleci +``` + +**YAML Configuration:** + +This is the correct shape for a CircleCI entry in a `--config` file: + +```yaml +sources: +- connection: + '@type': type.googleapis.com/sources.CircleCI + token: "cci_xxxxxxxxxxxxxxxxxxxx" + name: circleci-scan + type: SOURCE_TYPE_CIRCLECI + verify: true +``` + +Note: as of this writing, `pkg/config/config.go` does not have a case for `SOURCE_TYPE_CIRCLECI` in its source loader, so a config file entry like the one above will fail to load with `got unexpected source type`. Use the CLI flags or environment variable above until that is added. + +## How Scanning Works + +### Scanning Process + +1. **Project Listing**: Fetches all projects the token can access via `GET /projects` +2. **Build Listing**: For each project, lists all builds via `GET /project/:vcs/:org/:repo` +3. **Job/Step Retrieval**: For each build, fetches step and action details via `GET /project/:vcs/:org/:repo/:build_num` +4. **Action Output Scanning**: Downloads and scans the output of every action's `output_url` +5. **Build Config Scanning**: Scans the build's `circle.yml` content +6. **Chunk Generation**: Emits chunks of data to the detection engine, tagged with VCS type, username, repository, build number, and step name + +### What Gets Scanned + +- **Job Step Output**: stdout/stderr captured for each executed step +- **Build Configuration**: The `circle.yml`/`config.yml` content used for each build + +### What Doesn't Get Scanned + +- Artifacts, checkout keys, and environment variables (not currently implemented; see `TODO` in source) +- Lines containing `CIRCLE_SHA1=` (explicitly stripped before scanning) + +## Usage Examples + +### Scanning All Accessible Projects + +```bash +trufflehog circleci --token cci_xxxxxxxxxxxxxxxxxxxx +``` + +### Scanning Using an Environment Variable Token + +```bash +export CIRCLECI_TOKEN=cci_xxxxxxxxxxxxxxxxxxxx +trufflehog circleci +``` + +## Troubleshooting + +### Common Issues + +**Issue**: `invalid credentials, status 401` when listing projects +**Solution**: Verify the token is correct and has not been revoked. Regenerate a personal API token from CircleCI account settings if needed. + +--- + +**Issue**: `no builds found for project` errors for some projects +**Solution**: This is expected for projects with no build history; scanning continues for other projects. + +--- + +**Issue**: Slow scanning with many projects +**Solution**: Increase concurrency via the `concurrency` setting used when initializing the source; scanning is parallelized per project. + +--- + +**Issue**: Missing artifacts, checkout keys, or environment variables in scan results +**Solution**: These are not yet scanned by this source (tracked as a `TODO`); only build step output and build configuration are covered today. diff --git a/pkg/sources/circleci/circleci_test.go b/pkg/sources/circleci/circleci_test.go index 25becd8f6..e8dc92f5a 100644 --- a/pkg/sources/circleci/circleci_test.go +++ b/pkg/sources/circleci/circleci_test.go @@ -108,6 +108,42 @@ func TestSource_Scan(t *testing.T) { } } +func TestRemoveCircleSha1Line(t *testing.T) { + tests := []struct { + name string + input string + want string + }{ + { + name: "removes line containing CIRCLE_SHA1", + input: "export PATH=/usr/bin\nexport CIRCLE_SHA1=abcdef123456\necho done", + want: "export PATH=/usr/bin\necho done", + }, + { + name: "no CIRCLE_SHA1 line present", + input: "line one\nline two", + want: "line one\nline two", + }, + { + name: "removes multiple CIRCLE_SHA1 lines", + input: "a\nCIRCLE_SHA1=111\nb\nCIRCLE_SHA1=222\nc", + want: "a\nb\nc", + }, + { + name: "empty input", + input: "", + want: "", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := removeCircleSha1Line([]byte(tt.input)) + assert.Equal(t, tt.want, string(got)) + }) + } +} + // additional test for edge cases func TestSource_EdgeCases(t *testing.T) { tests := []struct { diff --git a/pkg/sources/docker/README.md b/pkg/sources/docker/README.md index 862a93fe5..79845ae22 100644 --- a/pkg/sources/docker/README.md +++ b/pkg/sources/docker/README.md @@ -59,88 +59,82 @@ The Docker source supports several image reference formats: For public images that don't require authentication: -**YAML Configuration:** -```yaml -sources: - - type: docker - name: public-images - docker: - unauthenticated: {} - images: - - nginx:latest - - alpine:3.18 -``` - **CLI Usage:** ```bash trufflehog docker --image nginx:latest ``` +**YAML Configuration:** +```yaml +sources: +- connection: + '@type': type.googleapis.com/sources.Docker + unauthenticated: {} + images: + - nginx:latest + - alpine:3.18 + name: public-images + type: SOURCE_TYPE_DOCKER + verify: true +``` + --- #### 2. Basic Authentication For private registries requiring username and password: -**YAML Configuration:** -```yaml -sources: - - type: docker - name: private-registry - docker: - basic_auth: - username: myuser - password: mypassword - images: - - myregistry.com/private-image:latest - - myregistry.com/another-image:v1.0.0 -``` - **CLI Usage:** Trufflehog does not provide basic authentication using username and password through CLI at the moment. +**YAML Configuration:** +```yaml +sources: +- connection: + '@type': type.googleapis.com/sources.Docker + basic_auth: + username: myuser + password: mypassword + images: + - myregistry.com/private-image:latest + - myregistry.com/another-image:v1.0.0 + name: private-registry + type: SOURCE_TYPE_DOCKER + verify: true +``` + --- #### 3. Bearer Token For registries using token-based authentication (e.g., Dockerhub registry): -**YAML Configuration:** -```yaml -sources: - - type: docker - name: truffle-packages - docker: - bearer_token: "ghp_xxxxxxxxxxxxxxxxxxxx" - images: - - myorg/myapp:latest - - myorg/frontend:v2.1.0 -``` - **CLI Usage:** ```bash trufflehog docker --image myorg/myapp:latest --bearer-token eyJ_xxxxxxxxxxxxxxxxxxxx ``` +**YAML Configuration:** +```yaml +sources: +- connection: + '@type': type.googleapis.com/sources.Docker + bearer_token: "ghp_xxxxxxxxxxxxxxxxxxxx" + images: + - myorg/myapp:latest + - myorg/frontend:v2.1.0 + name: truffle-packages + type: SOURCE_TYPE_DOCKER + verify: true +``` + --- #### 4. Docker Keychain Uses credentials from your local Docker configuration (`~/.docker/config.json`): -**YAML Configuration:** -```yaml -sources: - - type: docker - name: local-docker-creds - docker: - docker_keychain: true - images: - - myregistry.com/private-image:latest - - docker.io/myorg/app:latest -``` - **CLI Usage:** ```bash # First, authenticate with Docker @@ -161,6 +155,20 @@ docker login quay.io cat ~/.docker/config.json ``` +**YAML Configuration:** +```yaml +sources: +- connection: + '@type': type.googleapis.com/sources.Docker + docker_keychain: true + images: + - myregistry.com/private-image:latest + - docker.io/myorg/app:latest + name: local-docker-creds + type: SOURCE_TYPE_DOCKER + verify: true +``` + --- ### Namespace Scanning (This feature is currently in beta version and under testing) @@ -184,11 +192,13 @@ trufflehog docker --namespace myorg --registry-token **YAML Configuration:** ```yaml sources: - - type: docker - name: org-scan - docker: - namespace: myorg - registry_token: "ghp_xxxxxxxxxxxxxxxxxxxx" +- connection: + '@type': type.googleapis.com/sources.Docker + namespace: myorg + registry_token: "ghp_xxxxxxxxxxxxxxxxxxxx" + name: org-scan + type: SOURCE_TYPE_DOCKER + verify: true ``` Supported registries: @@ -281,19 +291,6 @@ docker login my-registry.io trufflehog docker --image my-registry.io/private-app:v1.0.0 ``` -## Testing Results - -| Test Case | Status | Command/Configuration | Registry URL | Notes | -|-----------|--------|----------------------|--------------|-------| -| Scan remote image on DockerHub | ✅ Success | `--image ` | https://hub.docker.com/ | Public images work without authentication | -| Scan specific tag of image on DockerHub | ✅ Success | `--image :` | https://hub.docker.com/ | Tag specification working correctly | -| Scan all images under namespace | In Progress | `--namespace ` | DockerHub, Quay, GHCR | Automatically discovers all public images | -| Scan remote image on Quay.io | ✅ Success | `--image quay.io/prometheus/prometheus` | https://quay.io/search | Public Quay.io registry supported | -| Scan multiple images | ✅ Success | `--image --image ` | Multiple registries | Sequential scanning of multiple images | -| Scan remote image on DockerHub with token | ✅ Success | `--token `(Generate token using username and password) | https://hub.docker.com/ | Authenticated scanning for private repos | -| Scan private image on Quay | ⏸️ Halted | N/A | https://quay.io/ | RedHat requires paid account for private repos | -| Scan private image on GHCR | ✅ Success | `--image ghcr.io/` | https://github.com/packages | GitHub Container Registry | - ## Troubleshooting ### Common Issues @@ -315,3 +312,8 @@ trufflehog docker --image my-registry.io/private-app:v1.0.0 **Issue**: Files not being scanned **Solution**: Check exclude patterns and file size limits. Verify files are under 50MB. + +--- + +**Issue**: Cannot scan private image on Quay.io +**Note**: Quay.io requires a paid account to access private repositories. (Not supported yet)