trufflesecurity/trufflehog
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add documentation for CircleCI source (#5268)

Browse Source 
* Add documentation for CircleCI source

* updated yaml config
This commit is contained in:
Kashif Khan
2026-09-16 12:10:06 +05:00
committed by GitHub
parent b1d7dae8db
commit b8a71ee3fe
3 changed files with 230 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/sources/circleci/README.md
+124
View File
@@ -0,0 +1,124 @@
# CircleCI Source
## Overview
The CircleCI source enables TruffleHog to scan CircleCI build data for secrets, credentials, and sensitive data. It scans every project accessible by the provided API token, walking each project's build history, job steps, and build configuration.
## CircleCI Fundamentals
### What is CircleCI?
CircleCI is a continuous integration and delivery (CI/CD) platform that automates building, testing, and deploying software. Pipelines are triggered by VCS events (e.g., a GitHub or Bitbucket push) and run a sequence of jobs and steps defined in a project's `config.yml`.
### Key CircleCI Terminology
| Term | Description |
|------|-------------|
| **Project** | A repository connected to CircleCI, identified by VCS type, org/username, and repo name |
| **Build** | A single execution of a project's pipeline, identified by an incrementing build number |
| **Job** | A defined unit of work within a build (e.g., `build`, `test`, `deploy`) |
| **Step** | An individual command or action executed within a job |
| **Action** | The underlying execution unit CircleCI's v1.1 API returns per step, with a link to its output |
| **circle.yml** | The build configuration used for a given build (legacy v1.1 naming for `config.yml`) |
| **API Token** | A personal or project token used to authenticate against the CircleCI API |
## Features
- **Full Project Discovery**: Automatically lists every project visible to the provided token
- **Build History Scanning**: Iterates through all builds for each discovered project
- **Job Step Output Scanning**: Fetches and scans the output of every action within every build step
- **Build Configuration Scanning**: Scans the `circle.yml`/`config.yml` used for each build
- **Concurrent Processing**: Projects are scanned in parallel using a bounded worker pool
- **CIRCLE_SHA1 Redaction**: Strips the `CIRCLE_SHA1=` environment line from scanned output to avoid noisy commit-SHA false positives
## Configuration
### Authentication
CircleCI only supports token-based authentication.
**CLI Usage:**
```bash
trufflehog circleci --token cci_xxxxxxxxxxxxxxxxxxxx
```
The token can also be provided via the `CIRCLECI_TOKEN` environment variable:
```bash
export CIRCLECI_TOKEN=cci_xxxxxxxxxxxxxxxxxxxx
trufflehog circleci
```
**YAML Configuration:**
This is the correct shape for a CircleCI entry in a `--config` file:
```yaml
sources:
- connection:
'@type': type.googleapis.com/sources.CircleCI
token: "cci_xxxxxxxxxxxxxxxxxxxx"
name: circleci-scan
type: SOURCE_TYPE_CIRCLECI
verify: true
```
Note: as of this writing, `pkg/config/config.go` does not have a case for `SOURCE_TYPE_CIRCLECI` in its source loader, so a config file entry like the one above will fail to load with `got unexpected source type`. Use the CLI flags or environment variable above until that is added.
## How Scanning Works
### Scanning Process
1. **Project Listing**: Fetches all projects the token can access via `GET /projects`
2. **Build Listing**: For each project, lists all builds via `GET /project/:vcs/:org/:repo`
3. **Job/Step Retrieval**: For each build, fetches step and action details via `GET /project/:vcs/:org/:repo/:build_num`
4. **Action Output Scanning**: Downloads and scans the output of every action's `output_url`
5. **Build Config Scanning**: Scans the build's `circle.yml` content
6. **Chunk Generation**: Emits chunks of data to the detection engine, tagged with VCS type, username, repository, build number, and step name
### What Gets Scanned
- **Job Step Output**: stdout/stderr captured for each executed step
- **Build Configuration**: The `circle.yml`/`config.yml` content used for each build
### What Doesn't Get Scanned
- Artifacts, checkout keys, and environment variables (not currently implemented; see `TODO` in source)
- Lines containing `CIRCLE_SHA1=` (explicitly stripped before scanning)
## Usage Examples
### Scanning All Accessible Projects
```bash
trufflehog circleci --token cci_xxxxxxxxxxxxxxxxxxxx
```
### Scanning Using an Environment Variable Token
```bash
export CIRCLECI_TOKEN=cci_xxxxxxxxxxxxxxxxxxxx
trufflehog circleci
```
## Troubleshooting
### Common Issues
**Issue**: `invalid credentials, status 401` when listing projects
**Solution**: Verify the token is correct and has not been revoked. Regenerate a personal API token from CircleCI account settings if needed.
---
**Issue**: `no builds found for project` errors for some projects
**Solution**: This is expected for projects with no build history; scanning continues for other projects.
---
**Issue**: Slow scanning with many projects
**Solution**: Increase concurrency via the `concurrency` setting used when initializing the source; scanning is parallelized per project.
---
**Issue**: Missing artifacts, checkout keys, or environment variables in scan results
**Solution**: These are not yet scanned by this source (tracked as a `TODO`); only build step output and build configuration are covered today.
pkg/sources/circleci/circleci_test.go
+36
View File
@@ -108,6 +108,42 @@ func TestSource_Scan(t *testing.T) {
}
}
func TestRemoveCircleSha1Line(t *testing.T) {
tests := []struct {
name string
input string
want string
}{
{
name: "removes line containing CIRCLE_SHA1",
input: "export PATH=/usr/bin\nexport CIRCLE_SHA1=abcdef123456\necho done",
want: "export PATH=/usr/bin\necho done",
},
{
name: "no CIRCLE_SHA1 line present",
input: "line one\nline two",
want: "line one\nline two",
},
{
name: "removes multiple CIRCLE_SHA1 lines",
input: "a\nCIRCLE_SHA1=111\nb\nCIRCLE_SHA1=222\nc",
want: "a\nb\nc",
},
{
name: "empty input",
input: "",
want: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := removeCircleSha1Line([]byte(tt.input))
assert.Equal(t, tt.want, string(got))
})
}
}
// additional test for edge cases
func TestSource_EdgeCases(t *testing.T) {
tests := []struct {
pkg/sources/docker/README.md
+70 -68
View File
@@ -59,88 +59,82 @@ The Docker source supports several image reference formats:
For public images that don't require authentication:
**YAML Configuration:**
```yaml
sources:
- type: docker
name: public-images
docker:
unauthenticated: {}
images:
- nginx:latest
- alpine:3.18
```
**CLI Usage:**
```bash
trufflehog docker --image nginx:latest
```
**YAML Configuration:**
```yaml
sources:
- connection:
'@type': type.googleapis.com/sources.Docker
unauthenticated: {}
images:
- nginx:latest
- alpine:3.18
name: public-images
type: SOURCE_TYPE_DOCKER
verify: true
```
---
#### 2. Basic Authentication
For private registries requiring username and password:
**YAML Configuration:**
```yaml
sources:
- type: docker
name: private-registry
docker:
basic_auth:
username: myuser
password: mypassword
images:
- myregistry.com/private-image:latest
- myregistry.com/another-image:v1.0.0
```
**CLI Usage:**
Trufflehog does not provide basic authentication using username and password through CLI at the moment.
**YAML Configuration:**
```yaml
sources:
- connection:
'@type': type.googleapis.com/sources.Docker
basic_auth:
username: myuser
password: mypassword
images:
- myregistry.com/private-image:latest
- myregistry.com/another-image:v1.0.0
name: private-registry
type: SOURCE_TYPE_DOCKER
verify: true
```
---
#### 3. Bearer Token
For registries using token-based authentication (e.g., Dockerhub registry):
**YAML Configuration:**
```yaml
sources:
- type: docker
name: truffle-packages
docker:
bearer_token: "ghp_xxxxxxxxxxxxxxxxxxxx"
images:
- myorg/myapp:latest
- myorg/frontend:v2.1.0
```
**CLI Usage:**
```bash
trufflehog docker --image myorg/myapp:latest --bearer-token eyJ_xxxxxxxxxxxxxxxxxxxx
```
**YAML Configuration:**
```yaml
sources:
- connection:
'@type': type.googleapis.com/sources.Docker
bearer_token: "ghp_xxxxxxxxxxxxxxxxxxxx"
images:
- myorg/myapp:latest
- myorg/frontend:v2.1.0
name: truffle-packages
type: SOURCE_TYPE_DOCKER
verify: true
```
---
#### 4. Docker Keychain
Uses credentials from your local Docker configuration (`~/.docker/config.json`):
**YAML Configuration:**
```yaml
sources:
- type: docker
name: local-docker-creds
docker:
docker_keychain: true
images:
- myregistry.com/private-image:latest
- docker.io/myorg/app:latest
```
**CLI Usage:**
```bash
# First, authenticate with Docker
@@ -161,6 +155,20 @@ docker login quay.io
cat ~/.docker/config.json
```
**YAML Configuration:**
```yaml
sources:
- connection:
'@type': type.googleapis.com/sources.Docker
docker_keychain: true
images:
- myregistry.com/private-image:latest
- docker.io/myorg/app:latest
name: local-docker-creds
type: SOURCE_TYPE_DOCKER
verify: true
```
---
### Namespace Scanning (This feature is currently in beta version and under testing)
@@ -184,11 +192,13 @@ trufflehog docker --namespace myorg --registry-token <access_token>
**YAML Configuration:**
```yaml
sources:
- type: docker
name: org-scan
docker:
namespace: myorg
registry_token: "ghp_xxxxxxxxxxxxxxxxxxxx"
- connection:
'@type': type.googleapis.com/sources.Docker
namespace: myorg
registry_token: "ghp_xxxxxxxxxxxxxxxxxxxx"
name: org-scan
type: SOURCE_TYPE_DOCKER
verify: true
```
Supported registries:
@@ -281,19 +291,6 @@ docker login my-registry.io
trufflehog docker --image my-registry.io/private-app:v1.0.0
```
## Testing Results
| Test Case | Status | Command/Configuration | Registry URL | Notes |
|-----------|--------|----------------------|--------------|-------|
| Scan remote image on DockerHub | ✅ Success | `--image <image_name>` | https://hub.docker.com/ | Public images work without authentication |
| Scan specific tag of image on DockerHub | ✅ Success | `--image <image_name>:<tag_name>` | https://hub.docker.com/ | Tag specification working correctly |
| Scan all images under namespace | In Progress | `--namespace <namespace>` | DockerHub, Quay, GHCR | Automatically discovers all public images |
| Scan remote image on Quay.io | ✅ Success | `--image quay.io/prometheus/prometheus` | https://quay.io/search | Public Quay.io registry supported |
| Scan multiple images | ✅ Success | `--image <image_name> --image <image_name>` | Multiple registries | Sequential scanning of multiple images |
| Scan remote image on DockerHub with token | ✅ Success | `--token <token>`(Generate token using username and password) | https://hub.docker.com/ | Authenticated scanning for private repos |
| Scan private image on Quay | ⏸️ Halted | N/A | https://quay.io/ | RedHat requires paid account for private repos |
| Scan private image on GHCR | ✅ Success | `--image ghcr.io/<image_name>` | https://github.com/packages | GitHub Container Registry |
## Troubleshooting
### Common Issues
@@ -315,3 +312,8 @@ trufflehog docker --image my-registry.io/private-app:v1.0.0
**Issue**: Files not being scanned
**Solution**: Check exclude patterns and file size limits. Verify files are under 50MB.
---
**Issue**: Cannot scan private image on Quay.io
**Note**: Quay.io requires a paid account to access private repositories. (Not supported yet)