Scan for secrets
Perform an endorctl scan using the following methods to detect secrets:
-
Scan a specific code reference - Scan for secrets only on a defined path in the context of a checked-out branch, commit SHA or tag to identify secrets and raise findings. This helps you to identify secrets that are leaked in the context of what you are working on right now.
-
Scan complete history - Scan for secrets in all existing branches or tags to identify if a secret has ever been leaked in the history of the project and raise findings. This helps you to identify if any secret has ever been leaked even if it was not leaked in the context of what you are working on right now.
-
Scan pre-commits - Scan for secrets in the code before committing the code to your repository during the automated pre-commit checks. This helps you identify and remove sensitive information from your code files early in the development life cycle.
Scan a specific code reference
When starting a secrets scan, this default choice utilizes specified rules to search for patterns on the files located in the path where the scan is initiated.
To initiate a scan use the following command:
endorctl scan --secrets
Scan complete history
Users can scan the Git logs by using the complete history scan. The repository should be present in the scanned path. endorctl examines the entire repository history to search for secrets.
To perform a complete scan, include the --git-logs
option in the command line.
endorctl scan --secrets --git-logs
Scan pre-commits
You can check for secrets before committing the code to the repository as part of pre-commit hooks.
Pre-requisites:
You must install and initialize endorctl before scanning the precommits.
- Create a
.git/hooks/pre-commit
file at the root of your Git repository to configure the pre-commit hook. It runs automatically when you make a commit and looks for secrets in your commit.
cd .git/hooks
touch pre-commit
- Edit the
.git/hooks/pre-commit
and include:
#!/bin/bash
#
# Script invoked on git commit.
#
if ! endorctl scan --pre-commit-checks --secrets; then
echo "Pre-commit checks failed"
exit 1
fi
echo "No secrets found: Pre-commit checks succeeded"
--pre-commit-checks
performs a pre-commit scan and will scan only the current changes that you are committing to the repository.
- Set the file permissions to make it executable.
chmod +x .git/hooks/pre-commit
Note
You can’t push the.git/hooks/
folder to the Git repository because it’s only recognized locally on your system.
To include the pre-commit code in the Git repository, save it in a different location, like a hooks/ directory
, and then copy it into .git/hooks/
. This way, you can easily push the hook code to your Git repository.
- You can set up this hook on other systems in your organization by creating a script and running it on each system.
sh setup-hooks.sh
#!/bin/sh
# Copy all hooks to .git/hooks cp hooks/* .git/hooks/
chmod +x .git/hooks/*
Note
Endor Labs secret rules come packaged with theendorctl
binary, so a local secrets scan using the --pre-commit
flag does not need to connect to Endor Labs services over the internet, making the scan extremely fast. However, this also means the pre-commit scan does not include any custom secret rules added to your namespace.
Here’s an example output when no secrets are found.
Here’s an example when secrets are detected and the commit fails.
Exclude false positives from secret scan
There could be cases where certain lines of code or specific patterns are mistakenly flagged as potential secrets but are safe to include such as test values or non-sensitive information.
To handle such false positives, you can annotate the non-sensitive lines with endorctl:allow
.
# These are test credentials, safe to commit
username = "test_user" # endorctl:allow
password = "test_password" # endorctl:allow
Scan for secrets using regular expression
Endor Labs scans for secrets based on regular expressions that are designed to detect the presence of a secret. It then validates the discovered secrets against external APIs to identify if they are valid. Valid secrets actively provide access to a service or an application and can be used to gain unauthorized access.
Regular expressions are customized to match specific types of secrets, such as GitHub personal access tokens, OAuth access tokens, AWS access tokens, OpenAPI keys, Client IDs, Client Secrets, and more.
For example, a GitHub Personal Access Token can be described with the following regular expression:
github_pat_[0-9a-zA-Z_]{82}
Users can use the following rules to scan their codebase and detect secrets:
System rules - Endor Labs provides users with out-of-the-box rules for secret patterns for many public services like GitHub, GitLab, AWS, Bitbucket, Dropbox, and more.
Custom rules - If you are using a service that is not included in the out-of-the-box list of secret patterns provided by Endor Labs, you can build your own custom rule to scan and detect the secrets for any service.
System rules
You can view the out-of-the-box Endor Labs secret rules in the user interface or on the command line. The pre-configured secret rules are listed for several popular services such as GitHub, GitLab, AWS, Dropbox, Adobe, Atlassian, Bitbucket, Coinbase, Databricks, and more.
The following table lists the most important fields of the rule definition:
Field name | Description |
---|---|
meta.name | The name of the rule |
spec.rule_id | The rule identifier must be unique across all rules, both the system and the ones created in your namespace |
spec.regex | This field contains the pattern that the secrets scanner will try to match against to create a result of this type |
spec.keywords | The keywords are used for an initial check of a pattern before the full regex expression gets evaluated |
spec.validation | The details about how a secret can be validated |
spec.disabled | Always enabled for system rules |
Fetch system rules from the user interface
Use the following instructions to view the out-of-the-box Endor Labs secret rules in the user interface:
- From the sidebar, click Settings and select SECRETS.
- To view the complete details of a rule, click the three vertical dots on the right side and click View Rule.
Fetch system rules from the CLI
To fetch the Endor Labs secret scanning rules from the command line type the following commands:
endorctl api list -r SecretRule -n <your-namespace>
For example, to see the rule for the GitHub Personal Access Token, you could search by the name GitHub Personal Access Token
or by the rule-id github-pat
:
endorctl api get -r SecretRule -n <your-namespace> --name "GitHub Personal Access Token"
endorctl api list -r SecretRule -n <your-namespace> --filter=spec.rule_id==github-pat
Validator
You can use a validator to check if a discovered secret is valid or not. The Endor Labs system rules for secrets include the necessary validator. When you validate a secret, the finding for that secret is categorized as critical, ensuring it receives higher priority compared to others.
When defining a custom rule, you can add your own validator from the command line or from the user interface. The system uses this information to send an HTTP request such as a GET or POST to the address specified by the public service for the detected secret.
For example, when a GitHub Personal Access Token named “ghp_endor123” is detected, the system sends the following HTTP request to GitHub’s address:
curl -H "Authorization: Token "ghp_endor123" https://api.github.com/user
The authentication codes defined by the service are used to mark the secrets as valid or invalid.
The validation portion of the secret rule contains the following fields:
Field | Description |
---|---|
name | The name of the validator |
http_request.uri | The address where the HTTP request should be sent |
http_request.method | The HTTP method to be used (GET or POST) |
http_request.header.(key, val) | A set of key/value pairs that are added to the HTTP header. See HTTP Request Header |
http_response.successful_auth_codes | The set of HTTP response codes that should be used to tag a secret as valid. For example, http.StatusOK (200) |
http_response.failed_auth_codes | The set of HTTP response codes that should be used to tag a secret as invalid. For example, http.StatusUnauthorized (401) |
HTTP Request Header
As shown above, this is a set of key/value pairs that should be added to the header. In its simplest form, it contains simple strings, for example:
{
"key": "Content-Type",
"value": "application/json"
}
There are cases where one needs to use a value on runtime and substitute a pattern. For example, the secret itself that needs to be substituted is one such case. This is achieved by declaring a value using the {{.Value}} pattern.
For the HTTP header section that includes the secret, the block looks like this:
{
"key": "Token",
"value": "{{.AuthzValue}}",
"authz": true,
}
In this case, the scanner will replace the candidate secret that was detected and add it to the HTTP request header in place of {{.AuthzValue}}.
This describes a special case where this key/value pair is marked with the authz
flag and is used to craft the “Authorization” part of the header, where three options are supported:
Key | Header |
---|---|
Basic | Authorization: Basic "hash{{.AuthzValue}}” |
Bearer | Authorization: Bearer {{.AuthzValue}} |
Token | Authorization: Token {{.AuthzValue}} |
Custom rules
Endor Labs provides a way to build your own custom rule to scan and detect secrets of any service. You can also add validation rules to identify if the detected secrets are active. Endor Labs scans and validates the secrets and raises findings with severity as defined in the finding policy that includes this rule.
Create custom secret rules from the user interface
To create custom secret rules from the Endor Labs user interface, use the following procedure.
- Click Settings and select Secrets.
- Click Add Custom Detector.
- In the DECTECTOR NAME, enter a unique name to identify the service or the secret you want to scan.
- Enter a DESCRIPTION to include any additional details to easily identify your custom secret rule.
- In the DETECTION RULE, enter a Regex to validate this specific type of secret.
- To optionally configure a validation rule, in VALIDATION URL, enter the URL of your service that Endor Labs can use to verify the validity of this secret.
- In SUCCESS RESPONSE CODES, enter the set of HTTP response codes to tag a secret as valid. For example, 200 indicates the HTTP Status OK.
- In FAILURE RESPONSE CODES, enter the set of HTTP response codes to tag a secret as invalid. For example, 401 indicates the HTTP Status Unauthorized.
- From the Authorization Header, select the authorization method used by the secrets provider.
Create custom secret rules from the command line
For example, consider a token “demo_value123” can be described using a regular expression. Here is an example of the rule specification:
"meta": {
"name": "Demo Token"
},
"spec": {
"disabled": false,
"keywords": [
"demo_"
],
"regex": "demo_[0-9a-zA-Z]{20}",
"rule_id": "demo-rule"
}
Use the following command from the CLI to create this custom rule.
$ endorctl api create -r SecretRule -n demo \
> --data '{
> "meta": {
> "name": "Demo Token"
> },
> "spec": {
> "disabled": false,
> "keywords": [
> "demo_"
> ],
> "regex": "demo_[0-9a-zA-Z]{20}",
> "rule_id": "demo-rule"
> }
> }'
INFO: Initiating host-check ...
INFO: Host-check complete
{
"meta": {
"create_time": "2023-09-27T17:08:18.436936Z",
"kind": "SecretRule",
"name": "Demo Token",
"update_time": "2023-09-27T17:08:18.436936Z",
"upsert_time": "2023-09-27T17:08:18.436936Z",
"version": "v1"
},
"spec": {
"disabled": false,
"keywords": [
"demo_"
],
"regex": "demo_[0-9a-zA-Z]{20}",
"rule_id": "demo-rule"
},
"tenant_meta": {
"namespace": "demo"
},
"uuid": "65146182aaeeffbaf5b6b553"
}
After the rule is created, the system uses this rule to detect this category of secrets.
If you can validate the secret using an HTTP request, then you can also add validation to this rule. See the following example for creating a validation rule for a demo_test123 token.
curl -H "Authorization: Bearer "demo_test123" https://api.testserver.com/user
Then the validation specification can be:
"validation": {
"name": "Demo secrets validator",
"http_request": {
"header": [
{
"key": "Bearer",
"value": "{{.AuthzValue}}",
"authz": true
}
],
"method": "GET",
"uri": "https://api.testserver.com/user"
},
"http_response": {
"failed_auth_codes": [
401
],
"successful_auth_codes": [
200
]
}
}
View secret findings
Users can view the findings generated out of secrets, prioritize them and take corrective action.
- From the sidebar, select Findings.
- Click Secrets to view secret findings and their severities.
Feedback
Was this page helpful?
Thanks for the feedback. Write to us at support@endor.ai to tell us more.
Thanks for the feedback. Write to us at support@endor.ai to tell us more.