Martin Locklear
35e23ee4e2
As I was wrapping my head around the project it wasn't very obvious to me what the high level structures of the project where (for example - what are the most important words?) These two docs try to provide what I think I probably would have appreciated when I started digging in. The two docs are: * docs/process_flow.md - Gives a high level overview of how trufflehog injests and processes sources * docs/concurrency.md - Tries to give a big picture overview of the concurrency structure that trufflehog uses, including the primary channels Also added a couple quick links from likely jumping off points in existing docs/code
3.9 KiB
3.9 KiB
TruffleHog Process Flows
Scans
Data Flow
flowchart LR
SourceDecomposition["`**Source Decomposition**
Breaking up the locations that we are looking _for_ secrets into small chunks`"]
DetectorMatching{Chunk<br/>to<br/>Detector<br/>Matching}
SecretDetection["`**Secret Detection**
Finding secrets in these chunks and (optionally) verifying whether they are live`"]
ResultNotification["`**Result Notification**
Enriching results with metadata and (usually) printing to console`"]
SourceDecomposition -- chunks --> DetectorMatching
DetectorMatching -- matched chunks --> SecretDetection
SecretDetection -- results --> ResultNotification
Source Decomposition
flowchart TD
subgraph Source
direction TB
SourceDescription("`**(1)** Sources are top level places we find data/files/text to _scan_`")
GitSource["git Source"]
GitHubSource["GitHub Source"]
FilesystemSource["File System Source"]
PostmanSource["Postman Source"]
end
subgraph Unit
direction TB
UnitDescription("`**(2)** Units are natural subdivisions of Sources, but still quite large`")
FilesystemUnit[Directory]
GitUnit[Git Repository]
end
subgraph Chunk
direction TB
ChunkDescription("`**(3)** Chunks are the smallest units that we decompose our chunks into, and are subsequent passed on to detection`")
FilesystemChunk[file contents]
GitRepositoryChunk["`git log diff hunks`"]
PostmanChunk[data chunk]
end
SourceDescription -- decomposed into --> UnitDescription
UnitDescription -- further decomposed into --> ChunkDescription
GitSource -- cloned locally<br />if not already local --> GitUnit
GitHubSource -- cloned locally --> GitUnit
PostmanSource -- Most sources\ndon't use units --> PostmanChunk
FilesystemSource --> FilesystemUnit
GitUnit -- git log -p --> GitRepositoryChunk
FilesystemUnit --> FilesystemChunk
style SourceDescription fill:#89553e
style UnitDescription fill:#89553e
style ChunkDescription fill:#89553e
Chunk to Detector Matching
flowchart LR
KeywordMatching["`**Keyword Matching**
_(Aho-Corsick)_
Match chunks to detectors based on the presence of specific keywords in the chunk`"]
chunks --> KeywordMatching --> detectors
Secret Detection
flowchart LR
subgraph Detector
direction RL
subgraph DetectorDescription[" "]
DetectorDescriptionText["`Detectors are the bits that actually check for the existence of a secret in a chunk, and (optionally) verify it`"]
ExampleDetectors["`Example Detectors:
* AWS
* Azure
* Twilio`"]
end
subgraph DetectorResponsibility[" "]
direction LR
De-Dupe-Detectors["`**De-Dupe-Detectors**
If multiple detectors keyword-match on the same chunk, we have some logic that chooses which detector will verify found secret (so we don't duplicate verification requests to externa APIs)`"]
CollectMatches["`**Collect Matches**
Detector specific regexes are run against the matched chunks, resulting in unverified secrets`"]
VerifyMatches["`**Verify Matches**
Optionally, observed unverified secrets are verified by attempting to use them against live services`"]
De-Dupe-Detectors -- deduped detectors --> CollectMatches
CollectMatches -- regex matched chunks --> VerifyMatches
end
style DetectorDescription fill:#89553e
style DetectorDescriptionText fill:#89553e
end
Result Notification
flowchart LR
Dispatcher["`**Dispatcher**
Results, verified or otherwise, are sent to a dispatcher to be sent to whichever place we're updating about the
results -- usually the command line.`"]
results --> Dispatcher --> output