Update Adding_Detectors_external.md (#1822)
* Update Adding_Detectors_external.md * Update Adding_Detectors_external.md * Update Adding_Detectors_external.md * Lil more
This commit is contained in:
Zachary Rice
@@ -15,9 +15,9 @@ The purpose of Secret Detectors is to discover secrets with exceptionally high s
|
||||
+ [Sourcing Guidelines](#sourcing-guidelines)
|
||||
+ [Development Guidelines](#development-guidelines)
|
||||
+ [Development Dependencies](#development-dependencies)
|
||||
+ [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
|
||||
+ [Creating a new Secret Detector](#creating-a-new-secret-detector)
|
||||
+ [Testing the Detector](#testing-the-detector)
|
||||
* [Addendum](#addendum)
|
||||
+ [Using a test secret file](#using-a-test-secret-file)
|
||||
+ [Adding Protos in Windows](#adding-protos-in-windows)
|
||||
|
||||
## Getting Started
|
||||
@@ -52,7 +52,7 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or
|
||||
|
||||
3. Proceed from step 3 of [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
|
||||
|
||||
### Creating a new Secret Scanner
|
||||
### Creating a new Secret Detector
|
||||
|
||||
1. Add a new Secret Detector enum to the [`DetectorType` list here](/proto/detectors.proto).
|
||||
|
||||
@@ -69,23 +69,60 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or
|
||||
Add the secret scanner to the [`pkg/engine/defaults.go`](https://github.com/trufflesecurity/trufflehog/blob/main/pkg/engine/defaults.go) file like [`github.com/trufflesecurity/trufflehog/v3/pkg/detectors/<detector_name>`](https://github.com/trufflesecurity/trufflehog/blob/b71ea27a696bdf1c3141f637fda4ee4936c2f2d6/pkg/engine/defaults.go#L9) and
|
||||
[`<detector_name>.Scanner{}`](https://github.com/trufflesecurity/trufflehog/blob/b71ea27a696bdf1c3141f637fda4ee4936c2f2d6/pkg/engine/defaults.go#L1546)
|
||||
|
||||
5. Complete the secret detector.
|
||||
5. Complete the Secret Detector.
|
||||
|
||||
The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
|
||||
The secret detector can be completed with these general steps:
|
||||
The Secret Detector can be completed with these general steps:
|
||||
|
||||
1. Create a [test secrets file, and export the variable](#using-a-test-secret-file)
|
||||
2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
|
||||
3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
|
||||
1. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
|
||||
2. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
|
||||
* Make sure you understand [verification indeterminacy](#verification-indeterminacy).
|
||||
4. Update the tests with these test cases at minimum:
|
||||
1. Found and verified (using a credential loaded from GCP Secrets)
|
||||
2. Found and unverified (determinately, i.e. the secret is invalid)
|
||||
3. Found and unverified (indeterminately due to timeout)
|
||||
4. Found and unverified (indeterminately due to an unexpected API response)
|
||||
5. Not found
|
||||
6. Any false positive cases that you come across
|
||||
5. Create a pull request for review.
|
||||
3. Create a [test for the detector](#testing-the-detector)
|
||||
4. Create a pull request for review.
|
||||
|
||||
### Testing the Detector
|
||||
To ensure the quality of your PR, make sure your tests are passing with verified credentials.
|
||||
|
||||
1. Create a file called `.env` with this env file format:
|
||||
|
||||
```bash
|
||||
SECRET_TYPE_ONE=value
|
||||
SECRET_TYPE_ONE_INACTIVE=v@lue
|
||||
```
|
||||
|
||||
2. Export the `TEST_SECRET_FILE` variable, pointing to the env file:
|
||||
|
||||
```bash
|
||||
export TEST_SECRET_FILE=".env"
|
||||
```
|
||||
The `.env` file should be in the new detector's directory like this:
|
||||
```
|
||||
├── tailscale
|
||||
│ ├── .env
|
||||
│ ├── tailscale.go
|
||||
│ └── tailscale_test.go
|
||||
```
|
||||
Now that a `.env` file is present, the test file can load secrets locally.
|
||||
|
||||
3. Next, update the tests as necessary. A test file has already been generated by the `go run hack/generate/generate.go` command from earlier. There are 5 cases that have been generated:
|
||||
1. Found and verified (using a credential loaded from the .env file)
|
||||
2. Found and unverified (determinately, i.e. the secret is invalid)
|
||||
3. Found and unverified (indeterminately due to timeout)
|
||||
4. Found and unverified (indeterminately due to an unexpected API response)
|
||||
5. Not found
|
||||
|
||||
Make any necessary updates to the tests. Note there might not be any changes required as the tests generated by the `go run hack/generate/generate.go` command are pretty good.
|
||||
[Here is an exemplary test file for a detector which covers all 5 test cases](https://github.com/trufflesecurity/trufflehog/blob/6f9065b0aae981133a7fa3431c17a5c6213be226/pkg/detectors/browserstack/browserstack_test.go).
|
||||
|
||||
4. Now run the tests and check to make sure they are passing ✔️!
|
||||
```bash
|
||||
go test ./pkg/detectors/<detector> -tags=detectors
|
||||
```
|
||||
|
||||
If the tests are passing, feel free to open a PR!
|
||||
|
||||
|
||||
|
||||
|
||||
## Addendum
|
||||
|
||||
@@ -102,23 +139,6 @@ For example, consider a hypothetical authentication endpoint that returns `200 O
|
||||
* A `403` response would indicate that verification failed **determinately** and no error object should be returned.
|
||||
* Any other response would indicate that verification failed **indeterminately** and an error object should be returned.
|
||||
|
||||
### Using a test secret file
|
||||
|
||||
1. Create a file called `.env` with this env file format:
|
||||
|
||||
```bash
|
||||
SECRET_TYPE_ONE=value
|
||||
SECRET_TYPE_ONE_INACTIVE=v@lue
|
||||
```
|
||||
|
||||
2. Export the `TEST_SECRET_FILE` variable, pointing to the env file:
|
||||
|
||||
```bash
|
||||
export TEST_SECRET_FILE=".env"
|
||||
```
|
||||
|
||||
Now, the detector test should attempt to load the given env key from that file.
|
||||
|
||||
### Adding Protos in Windows
|
||||
|
||||
1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
|
||||
@@ -136,7 +156,3 @@ Now, the detector test should attempt to load the given env key from that file.
|
||||
make protos
|
||||
```
|
||||
|
||||
### Testing a detector
|
||||
```bash
|
||||
go test ./pkg/detectors/<detector> -tags=detectors
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user