Martin Locklear
35e23ee4e2
As I was wrapping my head around the project it wasn't very obvious to me what the high level structures of the project where (for example - what are the most important words?) These two docs try to provide what I think I probably would have appreciated when I started digging in. The two docs are: * docs/process_flow.md - Gives a high level overview of how trufflehog injests and processes sources * docs/concurrency.md - Tries to give a big picture overview of the concurrency structure that trufflehog uses, including the primary channels Also added a couple quick links from likely jumping off points in existing docs/code
43 lines
2.2 KiB
Markdown
43 lines
2.2 KiB
Markdown
|
|
|
|
## Concurrency
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
%% Setup the workers
|
|
participant Main
|
|
Note over Main: e.startWorkers()<br />kicks off some number<br />of threads per worker type
|
|
create participant ScannerWorkers
|
|
Main->>ScannerWorkers: e.startScannerWorkers()
|
|
Note over ScannerWorkers: ScannerWorkers are primarily<br />responsible for enumerating<br />and chunking a source
|
|
create participant VerificationOverlapWorkers
|
|
Main->>VerificationOverlapWorkers: e.startVerificationOverlapWorkers()
|
|
Note over VerificationOverlapWorkers: VerificationOverlapWorkers<br />handles chunks<br />matched to multiple<br />detectors
|
|
create participant DetectorWorkers
|
|
Main->>DetectorWorkers: e.startDetectorWorkers()
|
|
Note over DetectorWorkers: DetectorWorkers are primarily<br />responsible for running<br />detectors on chunks
|
|
create participant NotifierWorkers
|
|
Main->>NotifierWorkers: e.startNotifierWorkers()
|
|
Note over NotifierWorkers: Primarily responsible for reporting<br />results (typically to the cmd line)
|
|
|
|
%% Set up the parallelism
|
|
par
|
|
Note over Main,ScannerWorkers: Depending on the type of<br />scan requested, calls one of<br />engine.(ScanGit|ScanGitHub|ScanFileSystem|etc)
|
|
Main->>ScannerWorkers: e.ChunksChan()<br /><- chunk
|
|
and
|
|
Note over ScannerWorkers: Decode chunks and find matching detectors
|
|
ScannerWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk
|
|
Note over ScannerWorkers: When multiple detectors match on the<br />same chunk we have to decided _which_<br />detector will verify found secrets
|
|
ScannerWorkers->>VerificationOverlapWorkers: e.verificationOverlapChunksChan<br /><- verificationOverlapChunk
|
|
and
|
|
Note over VerificationOverlapWorkers: Decide which detectors to run on that chunk
|
|
VerificationOverlapWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk
|
|
and
|
|
Note over DetectorWorkers: Run detection (finding secrets),<br />optionally verify them<br />do filtering and enrichment
|
|
DetectorWorkers->>NotifierWorkers: e.ResultsChan()|e.results<br /><-detectors.ResultWithMetadata
|
|
and
|
|
Note over NotifierWorkers: Write results to output
|
|
end
|
|
|
|
|
|
``` |