diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 786b4a806..2badad299 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,12 @@ Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/tru # Resources +## How things work + +It can be a bit daunting diving into the code and wrapping your head around the project from a high level. The following two docs help give that high level overview: +* [Process Flow](docs/process_flow.md) +* [Concurrency Overview](docs/concurrency.md) + ## Adding new secret detectors We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together! diff --git a/docs/concurrency.md b/docs/concurrency.md new file mode 100644 index 000000000..15edf73fa --- /dev/null +++ b/docs/concurrency.md @@ -0,0 +1,43 @@ + + +## Concurrency + +```mermaid +sequenceDiagram + %% Setup the workers + participant Main + Note over Main: e.startWorkers()
kicks off some number
of threads per worker type + create participant ScannerWorkers + Main->>ScannerWorkers: e.startScannerWorkers() + Note over ScannerWorkers: ScannerWorkers are primarily
responsible for enumerating
and chunking a source + create participant VerificationOverlapWorkers + Main->>VerificationOverlapWorkers: e.startVerificationOverlapWorkers() + Note over VerificationOverlapWorkers: VerificationOverlapWorkers
handles chunks
matched to multiple
detectors + create participant DetectorWorkers + Main->>DetectorWorkers: e.startDetectorWorkers() + Note over DetectorWorkers: DetectorWorkers are primarily
responsible for running
detectors on chunks + create participant NotifierWorkers + Main->>NotifierWorkers: e.startNotifierWorkers() + Note over NotifierWorkers: Primarily responsible for reporting
results (typically to the cmd line) + + %% Set up the parallelism + par + Note over Main,ScannerWorkers: Depending on the type of
scan requested, calls one of
engine.(ScanGit|ScanGitHub|ScanFileSystem|etc) + Main->>ScannerWorkers: e.ChunksChan()
<- chunk + and + Note over ScannerWorkers: Decode chunks and find matching detectors + ScannerWorkers->>DetectorWorkers: e.detectableChunksChan
<- detectableChunk + Note over ScannerWorkers: When multiple detectors match on the
same chunk we have to decided _which_
detector will verify found secrets + ScannerWorkers->>VerificationOverlapWorkers: e.verificationOverlapChunksChan
<- verificationOverlapChunk + and + Note over VerificationOverlapWorkers: Decide which detectors to run on that chunk + VerificationOverlapWorkers->>DetectorWorkers: e.detectableChunksChan
<- detectableChunk + and + Note over DetectorWorkers: Run detection (finding secrets),
optionally verify them
do filtering and enrichment + DetectorWorkers->>NotifierWorkers: e.ResultsChan()|e.results
<-detectors.ResultWithMetadata + and + Note over NotifierWorkers: Write results to output + end + + +``` \ No newline at end of file diff --git a/docs/process_flow.md b/docs/process_flow.md new file mode 100644 index 000000000..1936df69c --- /dev/null +++ b/docs/process_flow.md @@ -0,0 +1,138 @@ +# TruffleHog Process Flows + +## Scans + +## Data Flow + +```mermaid +flowchart LR + SourceDecomposition["`**Source Decomposition** + +Breaking up the locations that we are looking _for_ secrets into small chunks`"] + + DetectorMatching{Chunk
to
Detector
Matching} + + SecretDetection["`**Secret Detection** + +Finding secrets in these chunks and (optionally) verifying whether they are live`"] + + ResultNotification["`**Result Notification** + +Enriching results with metadata and (usually) printing to console`"] + + SourceDecomposition -- chunks --> DetectorMatching + DetectorMatching -- matched chunks --> SecretDetection + SecretDetection -- results --> ResultNotification +``` + +#### Source Decomposition + +```mermaid +flowchart TD + subgraph Source + direction TB + SourceDescription("`**(1)** Sources are top level places we find data/files/text to _scan_`") + GitSource["git Source"] + GitHubSource["GitHub Source"] + FilesystemSource["File System Source"] + PostmanSource["Postman Source"] + end + + subgraph Unit + direction TB + UnitDescription("`**(2)** Units are natural subdivisions of Sources, but still quite large`") + FilesystemUnit[Directory] + GitUnit[Git Repository] + end + + subgraph Chunk + direction TB + ChunkDescription("`**(3)** Chunks are the smallest units that we decompose our chunks into, and are subsequent passed on to detection`") + FilesystemChunk[file contents] + GitRepositoryChunk["`git log diff hunks`"] + PostmanChunk[data chunk] + end + + + SourceDescription -- decomposed into --> UnitDescription + UnitDescription -- further decomposed into --> ChunkDescription + + + GitSource -- cloned locally
if not already local --> GitUnit + GitHubSource -- cloned locally --> GitUnit + PostmanSource -- Most sources\ndon't use units --> PostmanChunk + FilesystemSource --> FilesystemUnit + + GitUnit -- git log -p --> GitRepositoryChunk + FilesystemUnit --> FilesystemChunk + + style SourceDescription fill:#89553e + style UnitDescription fill:#89553e + style ChunkDescription fill:#89553e +``` + +#### Chunk to Detector Matching + +```mermaid +flowchart LR + + + KeywordMatching["`**Keyword Matching** +_(Aho-Corsick)_ + +Match chunks to detectors based on the presence of specific keywords in the chunk`"] + + chunks --> KeywordMatching --> detectors +``` + +#### Secret Detection + +```mermaid +flowchart LR + +subgraph Detector + direction RL + subgraph DetectorDescription[" "] + DetectorDescriptionText["`Detectors are the bits that actually check for the existence of a secret in a chunk, and (optionally) verify it`"] + ExampleDetectors["`Example Detectors: + * AWS + * Azure + * Twilio`"] + end + + subgraph DetectorResponsibility[" "] + direction LR + + De-Dupe-Detectors["`**De-Dupe-Detectors** + +If multiple detectors keyword-match on the same chunk, we have some logic that chooses which detector will verify found secret (so we don't duplicate verification requests to externa APIs)`"] + + CollectMatches["`**Collect Matches** + +Detector specific regexes are run against the matched chunks, resulting in unverified secrets`"] + VerifyMatches["`**Verify Matches** + +Optionally, observed unverified secrets are verified by attempting to use them against live services`"] + + De-Dupe-Detectors -- deduped detectors --> CollectMatches + CollectMatches -- regex matched chunks --> VerifyMatches + end + + style DetectorDescription fill:#89553e + style DetectorDescriptionText fill:#89553e +end +``` + +#### Result Notification + +```mermaid +flowchart LR + + Dispatcher["`**Dispatcher** + +Results, verified or otherwise, are sent to a dispatcher to be sent to whichever place we're updating about the +results -- usually the command line.`"] + + results --> Dispatcher --> output +``` + diff --git a/pkg/engine/engine.go b/pkg/engine/engine.go index eb29b4ab7..e33018b67 100644 --- a/pkg/engine/engine.go +++ b/pkg/engine/engine.go @@ -1,3 +1,6 @@ +// Check the [process flow](docs/process_flow.md) and [concurrency](docs/concurrency.md) docs for +// something of a structural overview + package engine import (