126 lines
4.7 KiB
Markdown
126 lines
4.7 KiB
Markdown
# Secret Detectors
|
|
|
|
Secret Detectors have these two major functions:
|
|
|
|
1. Given some bytes, extract possible secrets, typically using a regex.
|
|
2. Validate the secrets against the target API, typically using a HTTP client.
|
|
|
|
The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
|
|
|
|
## Table of Contents
|
|
|
|
- [Secret Detectors](#secret-detectors)
|
|
- [Table of Contents](#table-of-contents)
|
|
- [Getting Started](#getting-started)
|
|
- [Sourcing Guidelines](#sourcing-guidelines)
|
|
- [Development Guidelines](#development-guidelines)
|
|
- [Development Dependencies](#development-dependencies)
|
|
- [Creating a new Secret Scanner](#creating-a-new-secret-detector)
|
|
- [Addendum](#addendum)
|
|
- [Managing Test Secrets](#managing-test-secrets)
|
|
- [Setting up Google Cloud SDK](#setting-up-google-cloud-sdk)
|
|
|
|
## Getting Started
|
|
|
|
### Sourcing Guidelines
|
|
|
|
We are interested in detectors for services that meet at least one of these criteria
|
|
- host data (they store any sort of data provided)
|
|
- have paid services (having a free or trial tier is okay though)
|
|
|
|
If you think that something should be included outside of these guidelines, please let us know.
|
|
|
|
### Development Guidelines
|
|
|
|
- When reasonable, favor using the `net/http` library to make requests instead of bringing in another library.
|
|
- Use the [`common.SaneHttpClient`](pkg/common/http.go) for the `http.Client` whenever possible.
|
|
- We recommend an editor with gopls integration (such as Vscode with Go plugin) for benefits like easily running tests, autocompletion, linting, type checking, etc.
|
|
|
|
### Development Dependencies
|
|
|
|
- A GitLab account
|
|
- A Google account
|
|
- [Google Cloud SDK installed](#setting-up-google-cloud-sdk)
|
|
- Go 1.17+
|
|
- Make
|
|
|
|
### Creating a new Secret Scanner
|
|
|
|
1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum.
|
|
|
|
2. Generate the Secret Detector
|
|
|
|
```bash
|
|
go run hack/generate/generate.go detector <DetectorType enum name>
|
|
```
|
|
|
|
3. Complete the secret detector.
|
|
|
|
The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
|
|
The secret detector can be completed with these general steps:
|
|
|
|
1. Add the test secret to GCP Secrets. See [managing test secrets](#managing-test-secrets)
|
|
2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
|
|
3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
|
|
4. Update the tests with these test cases at minimum:
|
|
1. Found and verified (using a credential loaded from GCP Secrets)
|
|
2. Found and unverified
|
|
3. Not found
|
|
4. Any false positive cases that you come across
|
|
5. Create a merge request for review. CI tests must be passing.
|
|
|
|
## Addendum
|
|
|
|
### Managing Test Secrets
|
|
|
|
Do not embed test credentials in the test code. Instead, use GCP Secrets Manager.
|
|
|
|
1. Access the latest secret version for modification.
|
|
|
|
Note: `/tmp/s` is a valid path on Linux. You will need to change that for Windows or OSX, otherwise you will see an error. On Windows you will also need to install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
|
|
|
|
```bash
|
|
gcloud secrets versions access --project trufflehog-testing --secret detectors3 latest > /tmp/s
|
|
```
|
|
|
|
2. Add the secret that you need for testing.
|
|
|
|
The command above saved it to `/tmp/s`.
|
|
|
|
The format is standard env file format,
|
|
|
|
```bash
|
|
SECRET_TYPE_ONE=value
|
|
SECRET_TYPE_ONE_INACTIVE=v@lue
|
|
```
|
|
|
|
3. Update the secret version with your modification.
|
|
|
|
```bash
|
|
gcloud secrets versions add --project trufflehog-testing detectors3 --data-file /tmp/s
|
|
```
|
|
|
|
4. Access the secret value as shown in the [example code](pkg/detectors/heroku/heroku_test.go).
|
|
|
|
### Setting up Google Cloud SDK
|
|
|
|
1. Install the Google Cloud SDK: https://cloud.google.com/sdk/docs/install
|
|
2. Authenticate with `gcloud auth login --update-adc` using your Google account
|
|
|
|
### Adding Protos in Windows
|
|
|
|
1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
|
|
2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
|
|
3. Open Ubuntu cli and install `dos2unix`.
|
|
```bash
|
|
sudo apt install dos2unix
|
|
```
|
|
4. Identify the `trufflehog` local directory and convert `scripts/gen_proto.sh` file in Unix format.
|
|
```bash
|
|
dos2unix ./scripts/gen_proto.sh
|
|
```
|
|
5. Open [/proto/detectors.proto](/proto/detectors.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
|
|
```bash
|
|
make protos
|
|
```
|