Make a first pass at some structural introduction docs (#4076)

As I was wrapping my head around the project it wasn't very obvious to me what the high level structures of the project where (for example - what are the most important words?) These two docs try to provide what I think I probably would have appreciated when I started digging in. The two docs are: * docs/process_flow.md - Gives a high level overview of how trufflehog injests and processes sources * docs/concurrency.md - Tries to give a big picture overview of the concurrency structure that trufflehog uses, including the primary channels Also added a couple quick links from likely jumping off points in existing docs/code
2025-05-01 10:56:20 -05:00
parent 1637f5de5b
commit 35e23ee4e2
4 changed files with 190 additions and 0 deletions
@@ -8,6 +8,12 @@ Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/tru

 # Resources

+## How things work
+
+It can be a bit daunting diving into the code and wrapping your head around the project from a high level.  The following two docs help give that high level overview:
+* [Process Flow](docs/process_flow.md)
+* [Concurrency Overview](docs/concurrency.md)
+
 ## Adding new secret detectors

 We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
@@ -0,0 +1,43 @@
+
+
+## Concurrency
+
+```mermaid
+sequenceDiagram
+    %% Setup the workers
+    participant Main
+    Note over Main: e.startWorkers()<br />kicks off some number<br />of threads per worker type
+    create participant ScannerWorkers
+    Main->>ScannerWorkers: e.startScannerWorkers()
+    Note over ScannerWorkers: ScannerWorkers are primarily<br />responsible for enumerating<br />and chunking a source
+    create participant VerificationOverlapWorkers
+    Main->>VerificationOverlapWorkers: e.startVerificationOverlapWorkers()
+    Note over VerificationOverlapWorkers: VerificationOverlapWorkers<br />handles chunks<br />matched to multiple<br />detectors
+    create participant DetectorWorkers
+    Main->>DetectorWorkers: e.startDetectorWorkers()
+    Note over DetectorWorkers: DetectorWorkers are primarily<br />responsible for running<br />detectors on chunks
+    create participant NotifierWorkers
+    Main->>NotifierWorkers: e.startNotifierWorkers()
+    Note over NotifierWorkers: Primarily responsible for reporting<br />results (typically to the cmd line)
+    
+    %% Set up the parallelism
+    par
+        Note over Main,ScannerWorkers: Depending on the type of<br />scan requested, calls one of<br />engine.(ScanGit|ScanGitHub|ScanFileSystem|etc)
+        Main->>ScannerWorkers:  e.ChunksChan()<br /><- chunk
+    and 
+        Note over ScannerWorkers: Decode chunks and find matching detectors
+        ScannerWorkers->>DetectorWorkers: e.detectableChunksChan<br /><- detectableChunk
+        Note over ScannerWorkers: When multiple detectors match on the<br />same chunk we have to decided _which_<br />detector will verify found secrets
+        ScannerWorkers->>VerificationOverlapWorkers: e.verificationOverlapChunksChan<br /><- verificationOverlapChunk
+    and
+        Note over VerificationOverlapWorkers: Decide which detectors to run on that chunk
+        VerificationOverlapWorkers->>DetectorWorkers:  e.detectableChunksChan<br /><- detectableChunk
+    and
+        Note over DetectorWorkers: Run detection (finding secrets),<br />optionally verify them<br />do filtering and enrichment
+        DetectorWorkers->>NotifierWorkers: e.ResultsChan()|e.results<br /><-detectors.ResultWithMetadata
+    and
+        Note over NotifierWorkers: Write results to output
+    end
+        
+    
+```
@@ -0,0 +1,138 @@
+# TruffleHog Process Flows
+
+## Scans
+
+## Data Flow
+
+```mermaid
+flowchart LR
+    SourceDecomposition["`**Source Decomposition**
+
+Breaking up the locations that we are looking _for_ secrets into small chunks`"]
+
+    DetectorMatching{Chunk<br/>to<br/>Detector<br/>Matching}
+    
+    SecretDetection["`**Secret Detection**
+
+Finding secrets in these chunks and (optionally) verifying whether they are live`"]
+
+    ResultNotification["`**Result Notification**
+
+Enriching results with metadata and (usually) printing to console`"]
+    
+    SourceDecomposition -- chunks --> DetectorMatching
+    DetectorMatching -- matched chunks --> SecretDetection
+    SecretDetection -- results --> ResultNotification
+```
+
+#### Source Decomposition
+
+```mermaid
+flowchart TD
+    subgraph Source
+        direction TB
+        SourceDescription("`**(1)** Sources are top level places we find data/files/text to _scan_`")
+        GitSource["git Source"]
+        GitHubSource["GitHub Source"]
+        FilesystemSource["File System Source"]
+        PostmanSource["Postman Source"]
+    end
+
+    subgraph Unit
+        direction TB
+        UnitDescription("`**(2)** Units are natural subdivisions of Sources, but still quite large`")
+        FilesystemUnit[Directory]
+        GitUnit[Git Repository]
+    end
+
+    subgraph Chunk
+        direction TB
+        ChunkDescription("`**(3)** Chunks are the smallest units that we decompose our chunks into, and are subsequent passed on to detection`")
+        FilesystemChunk[file contents]
+        GitRepositoryChunk["`git log diff hunks`"]
+        PostmanChunk[data chunk]
+    end
+
+
+    SourceDescription -- decomposed into --> UnitDescription
+    UnitDescription -- further decomposed into --> ChunkDescription
+
+
+    GitSource -- cloned locally<br />if not already local --> GitUnit
+    GitHubSource -- cloned locally --> GitUnit
+    PostmanSource -- Most sources\ndon't use units --> PostmanChunk
+    FilesystemSource --> FilesystemUnit
+
+    GitUnit -- git log -p --> GitRepositoryChunk
+    FilesystemUnit --> FilesystemChunk
+
+    style SourceDescription fill:#89553e
+    style UnitDescription fill:#89553e
+    style ChunkDescription fill:#89553e
+```
+
+#### Chunk to Detector Matching
+
+```mermaid
+flowchart LR
+
+
+    KeywordMatching["`**Keyword Matching**
+_(Aho-Corsick)_
+
+Match chunks to detectors based on the presence of specific keywords in the chunk`"]
+    
+    chunks --> KeywordMatching --> detectors
+```
+
+#### Secret Detection
+
+```mermaid
+flowchart LR
+
+subgraph Detector
+    direction RL
+    subgraph DetectorDescription["  "]
+        DetectorDescriptionText["`Detectors are the bits that actually check for the existence of a secret in a chunk, and (optionally) verify it`"]
+        ExampleDetectors["`Example Detectors:
+                    * AWS
+                    * Azure
+                    * Twilio`"]
+    end
+    
+    subgraph DetectorResponsibility[" "]
+        direction LR
+
+        De-Dupe-Detectors["`**De-Dupe-Detectors**
+
+If multiple detectors keyword-match on the same chunk, we have some logic that chooses which detector will verify found secret (so we don't duplicate verification requests to externa APIs)`"]
+
+        CollectMatches["`**Collect Matches**
+
+Detector specific regexes are run against the matched chunks, resulting in unverified secrets`"]
+        VerifyMatches["`**Verify Matches**
+
+Optionally, observed unverified secrets are verified by attempting to use them against live services`"]
+        
+        De-Dupe-Detectors -- deduped detectors --> CollectMatches
+        CollectMatches -- regex matched chunks --> VerifyMatches
+    end
+    
+    style DetectorDescription fill:#89553e
+    style DetectorDescriptionText fill:#89553e
+end
+```
+
+#### Result Notification
+
+```mermaid
+flowchart LR
+
+    Dispatcher["`**Dispatcher**
+
+Results, verified or otherwise, are sent to a dispatcher to be sent to whichever place we're updating about the 
+results -- usually the command line.`"]
+    
+    results --> Dispatcher --> output
+```
+
@@ -1,3 +1,6 @@
+// Check the [process flow](docs/process_flow.md) and [concurrency](docs/concurrency.md) docs for
+// something of a structural overview
+
 package engine

 import (