Secrets import

Distributing sensitive information across multiple external systems creates several challenges, including:

• Increased operational overhead.
• Increased exposure risk from data sprawl.
• Increased risk of outdated and out-of-sync information.

Using Vault as a single source of truth for sensitive data increases security and reduces management overhead, but migrating preexisting data from multiple and/or varied sources can be complex and costly.

The secrets import process helps you automate and streamline your sensitive data migration with codified import plans as HCL files. Import plans tell Vault which KVv2 secrets engine instance to store the expected secret data in, the source system for which data will be read from, and how to filter this data. Three HCL blocks make this possible:

• The destination block defines target KVv2 mounts.
• The source block provides credentials for connecting to the external system.
• The mapping block defines how Vault should decide which data gets imported (and possibly transformed) before writing the information to KVv2.

Destinations

Vault stores imported secrets in a Vault KVv2 secrets engine mount. Destination blocks start with destination_vault and define the desired KVv2 mount path and an optional namespace. The combination of these represent the exact location in your Vault instance you want the information stored.

HCL syntax

destination_vault {
  name      = "my-dest-1"
  namespace = "ns-1"
  mount     = "mount-1"
}

• name (string: <required>) - A unique name for the destination block that can be referenced in subsequent mapping blocks.

• mount (string: <required>) - The mount path for the target KVv2 instance.

• address (string) - Optional network address of the Vault server with the KVv2 secrets engine enabled. By default, the Vault client's address will be used.

• credentials_file (string) - Optional path to authentication token for the Vault server at the specified address. By default, the Vault client's token will be used.

• namespace (string) - Optional namespace path containing the specified KVv2 mount. By default, Vault looks for the KVv2 mount under the root namespace.

Sources

Vault can import secrets from the following sources:

• GCP Secret Manager
• AWS Secret Manager
• Azure KeyVault
• HashiCorp Vault

To pull data from a source during import, Vault needs credentials for the external system. You can provide credentials directly as part of the import plan, or use Vault to automatically generate dynamic credentials if you already have the corresponding secrets engine configured. Otherwise credentials will be fetched from the default locations appropriate to the external system.

HCL syntax

Source blocks start with source_<external_system> and include any connection information required by the target system or the secrets engine to leverage. For example:

source_gcp {
  name             = "my-gcp-source-1"
  credentials_file = "/path/to/service-account-key.json"
}

• name (string: <required>) - A unique name for the source block that can be referenced in subsequent mapping blocks.

• credentials_file (string: <required>) - Path to a credential file with read permissions for the target system.

Depending on the source system, additional information may be required. Refer to the connection documentation for your source system to determine the full set of required fields for that system type.

Mappings

Mappings glue the source and destination together and filter the migrated data, to determine what is imported and what is ignored.

HCL syntax

Mapping blocks require a name, a source name, a destination name, and any corresponding transformations or filters that apply for each mapping type. Example:

mapping {
  name        = "my-map-1"
  source      = "my-gcp-source-1"
  destination = "my-dest-1"
  filter      = "Secret.Name matches `(foo|bar)`"
  transform "exact" {
    from = "foo"
    to = "foosball"
  }
  transform "regexp" {
    from = "foo(.*)"
    to = "bar$1"
  }
}

• name (string: <required>) - A unique name for the mapping block.

• source (string: <required>) - The name of a previously-defined source block from which the data should be read.

• destination (string: <required>) - The name of a previously defined destination block to which the data should be written.

• filter (string: "") - Optional string containing a (bexpr style boolean expression)[https://github.com/hashicorp/go-bexpr] that limits which secrets Vault imports from the source.

Filters

Filters describes conditions for secret fields that a source secret must meet to be imported. You can filter against the following secret fields:

• Secret.Name - The name of the source secret
• Secret.Tags - A map of key-value user-defined metadata associated with the source secret

You can also apply simple binary conditions to input values:

• Secret.Name != "foo"
• Secret.Tags.Team matches "my-dept.*"
• Secret.Tags contains "my-required-tag-name"

These can be combined using and, or, and not:

• (Secret.Name == "foo" or Secret.Tags.Team == "my-team") and Secret.Tags.Category = "my-cat"

Transformers

Transform stanzas come in two forms: exact and regexp.

An exact transform allows renaming a secret during import, so that the from secret name is imported into Vault as a secret named to. In the following example it, takes a source secret named foo and transforms it to foosball during import.

  transform "exact" {
    from = "foo"
    to = "foosball"
  }

regexp transforms rename secrets during import using Go regular expression syntax. For example, the transform below imports any secret in the source whose name starts with foo and replaces the foo prefix with bar.

  transform "regexp" {
    from = "foo(.*)"
    to = "bar$1"
  }
  transform "regexp" {
    from = "foo(?P<suffix>.*)"
    to = "bar$suffix"
  }

The from value supports parentheses to bookend capture groups and named capture groups using the syntax (?P<name>re). When you use named capture groups, you can reference the named group in the to value. For example, $name instead of $1.