From f2d0f809a840e57be02e0b951c88c69c4dbdd7f6 Mon Sep 17 00:00:00 2001 From: Zachary Rice Date: Tue, 26 Sep 2023 17:18:58 -0500 Subject: [PATCH] Update Adding_Detectors_external.md (#1822) * Update Adding_Detectors_external.md * Update Adding_Detectors_external.md * Update Adding_Detectors_external.md * Lil more --- hack/docs/Adding_Detectors_external.md | 90 +++++++++++++++----------- 1 file changed, 53 insertions(+), 37 deletions(-) diff --git a/hack/docs/Adding_Detectors_external.md b/hack/docs/Adding_Detectors_external.md index a40de07fb..c97369f71 100644 --- a/hack/docs/Adding_Detectors_external.md +++ b/hack/docs/Adding_Detectors_external.md @@ -15,9 +15,9 @@ The purpose of Secret Detectors is to discover secrets with exceptionally high s + [Sourcing Guidelines](#sourcing-guidelines) + [Development Guidelines](#development-guidelines) + [Development Dependencies](#development-dependencies) - + [Creating a new Secret Scanner](#creating-a-new-secret-scanner) + + [Creating a new Secret Detector](#creating-a-new-secret-detector) + + [Testing the Detector](#testing-the-detector) * [Addendum](#addendum) - + [Using a test secret file](#using-a-test-secret-file) + [Adding Protos in Windows](#adding-protos-in-windows) ## Getting Started @@ -52,7 +52,7 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or 3. Proceed from step 3 of [Creating a new Secret Scanner](#creating-a-new-secret-scanner) -### Creating a new Secret Scanner +### Creating a new Secret Detector 1. Add a new Secret Detector enum to the [`DetectorType` list here](/proto/detectors.proto). @@ -69,23 +69,60 @@ Note: Be sure to update the tests to reference the new secret values in GSM, or Add the secret scanner to the [`pkg/engine/defaults.go`](https://github.com/trufflesecurity/trufflehog/blob/main/pkg/engine/defaults.go) file like [`github.com/trufflesecurity/trufflehog/v3/pkg/detectors/`](https://github.com/trufflesecurity/trufflehog/blob/b71ea27a696bdf1c3141f637fda4ee4936c2f2d6/pkg/engine/defaults.go#L9) and [`.Scanner{}`](https://github.com/trufflesecurity/trufflehog/blob/b71ea27a696bdf1c3141f637fda4ee4936c2f2d6/pkg/engine/defaults.go#L1546) -5. Complete the secret detector. +5. Complete the Secret Detector. The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on. - The secret detector can be completed with these general steps: + The Secret Detector can be completed with these general steps: - 1. Create a [test secrets file, and export the variable](#using-a-test-secret-file) - 2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/). - 3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not. + 1. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/). + 2. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not. * Make sure you understand [verification indeterminacy](#verification-indeterminacy). - 4. Update the tests with these test cases at minimum: - 1. Found and verified (using a credential loaded from GCP Secrets) - 2. Found and unverified (determinately, i.e. the secret is invalid) - 3. Found and unverified (indeterminately due to timeout) - 4. Found and unverified (indeterminately due to an unexpected API response) - 5. Not found - 6. Any false positive cases that you come across - 5. Create a pull request for review. + 3. Create a [test for the detector](#testing-the-detector) + 4. Create a pull request for review. + +### Testing the Detector +To ensure the quality of your PR, make sure your tests are passing with verified credentials. + +1. Create a file called `.env` with this env file format: + + ```bash + SECRET_TYPE_ONE=value + SECRET_TYPE_ONE_INACTIVE=v@lue + ``` + +2. Export the `TEST_SECRET_FILE` variable, pointing to the env file: + + ```bash + export TEST_SECRET_FILE=".env" + ``` + The `.env` file should be in the new detector's directory like this: + ``` + ├── tailscale + │   ├── .env + │   ├── tailscale.go + │   └── tailscale_test.go + ``` + Now that a `.env` file is present, the test file can load secrets locally. + +3. Next, update the tests as necessary. A test file has already been generated by the `go run hack/generate/generate.go` command from earlier. There are 5 cases that have been generated: + 1. Found and verified (using a credential loaded from the .env file) + 2. Found and unverified (determinately, i.e. the secret is invalid) + 3. Found and unverified (indeterminately due to timeout) + 4. Found and unverified (indeterminately due to an unexpected API response) + 5. Not found + + Make any necessary updates to the tests. Note there might not be any changes required as the tests generated by the `go run hack/generate/generate.go` command are pretty good. + [Here is an exemplary test file for a detector which covers all 5 test cases](https://github.com/trufflesecurity/trufflehog/blob/6f9065b0aae981133a7fa3431c17a5c6213be226/pkg/detectors/browserstack/browserstack_test.go). + +4. Now run the tests and check to make sure they are passing ✔️! +```bash + go test ./pkg/detectors/ -tags=detectors + ``` + +If the tests are passing, feel free to open a PR! + + + ## Addendum @@ -102,23 +139,6 @@ For example, consider a hypothetical authentication endpoint that returns `200 O * A `403` response would indicate that verification failed **determinately** and no error object should be returned. * Any other response would indicate that verification failed **indeterminately** and an error object should be returned. -### Using a test secret file - -1. Create a file called `.env` with this env file format: - - ```bash - SECRET_TYPE_ONE=value - SECRET_TYPE_ONE_INACTIVE=v@lue - ``` - -2. Export the `TEST_SECRET_FILE` variable, pointing to the env file: - - ```bash - export TEST_SECRET_FILE=".env" - ``` - -Now, the detector test should attempt to load the given env key from that file. - ### Adding Protos in Windows 1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6. @@ -136,7 +156,3 @@ Now, the detector test should attempt to load the given env key from that file. make protos ``` -### Testing a detector -```bash - go test ./pkg/detectors/ -tags=detectors - ```