Hoplite

Hoplite is a Kotlin library for loading configuration files into typesafe classes in a boilerplate-free way. Define your config using Kotlin data classes, and at startup Hoplite will read from one or more config files, mapping the values in those files into your config classes. Any missing values, or values that cannot be converted into the required type will cause the config to fail with detailed error messages.

Features

Multiple formats: Write your configuration in several formats: Yaml, JSON, Toml, Hocon, or Java .props files or even mix and match formats in the same system.
Property Sources: Per-system overrides are possible from JVM system properties, environment variables, JNDI or a per-user local config file.
Batteries included: Support for many standard types such as primitives, enums, dates, collection types, inline classes, uuids, nullable types, as well as popular Kotlin third party library types such as NonEmptyList, Option and TupleX from Arrow.
Custom Data Types: The Decoder interface makes it easy to add support for your custom domain types or standard library types not covered out of the box.
Cascading: Config files can be stacked. Start with a default file and then layer new configurations on top. When resolving config, lookup of values falls through to the first file that contains a definition. Can be used to have a default config file and then an environment specific file.
Beautiful errors: Fail fast at runtime, with beautiful errors showing exactly what went wrong and where.
Preprocessors: Support for several preprocessors that will replace placeholders with values resolved from external configs, such as AWS Secrets Manager, Azure KeyVault and so on.
Reloadable config: Trigger config reloads on a fixed interval or in response to external events such as consul value changes.

Changelog

See the list of changes in each release here.

Getting Started

Add Hoplite to your build:

implementation 'com.sksamuel.hoplite:hoplite-core:<version>'

You will also need to include a module for the format(s) you to use.

Next define the data classes that are going to contain the config. You should create a top level class which can be named simply Config, or ProjectNameConfig. This class then defines a field for each config value you need. It can include nested data classes for grouping together related configs.

For example, if we had a project that needed database config, config for an embedded HTTP server, and a field which contained which environment we were running in (staging, QA, production etc), then we may define our classes like this:

data class Database(val host: String, val port: Int, val user: String, val pass: String)
data class Server(val port: Int, val redirectUrl: String)
data class Config(val env: String, val database: Database, val server: Server)

For our staging environment, we may create a YAML (or Json, etc) file called application-staging.yaml. The name doesn't matter, you can use any convention you wish.

env: staging

database:
  host: staging.wibble.com
  port: 3306
  user: theboss
  pass: 0123abcd

server:
  port: 8080
  redirectUrl: /404.html

Finally, to build an instance of Config from this file, and assuming the config file was on the classpath, we can simply execute:

val config = ConfigLoaderBuilder.default()
               .addResourceSource("/application-staging.yml")
               .build()
               .loadConfigOrThrow<Config>()

If the values in the config file are compatible, then an instance of Config will be returned. Otherwise, an exception will be thrown containing details of the errors.

Config Loader

As you have seen from the getting started guide, ConfigLoader is the entry point to using Hoplite. We create an instance of this loader class through the ConfigLoaderBuilder builder. To this builder we add sources, configuration, enable reports, add preprocessors and more.

To create a default builder, use ConfigLoaderBuilder.default() and after adding your sources, call build. Here is an example:

ConfigLoaderBuilder.default()
  .addResourceSource("/application-prod.yml")
  .addResourceSource("/reference.json")
  .build()
  .loadConfigOrThrow<MyConfig>()

The default method on ConfigLoaderBuilder sets up recommended defaults. If you wish to start with a completely empty config builder, then use ConfigLoaderBuilder.empty().

There are two ways to retrieve a populated data class from config. The first is to throw an exception if the config could not be resolved. We do this via the loadConfigOrThrow<T> function. Another is to return a ConfigResult validation monad via the loadConfig<T> function if you want to handle errors manually.

For most cases, when you are resolving config at application startup, the exception based approach is better. This is because you typically want any errors in config to abort application bootstrapping, dumping errors immediately to the console.

Beautiful Errors

When an error does occur, if you choose to throw an exception, the errors will be formatted in a human-readable way along with as much location information as possible. No more trying to track down a NumberFormatException in a 400 line config file.

Here is an example of the error formatting for a test file used by the unit tests. Notice that the errors indicate which file the value was pulled from.

Error loading config because:

    - Could not instantiate 'com.sksamuel.hoplite.json.Foo' because:

        - 'bar': Required type Boolean could not be decoded from a Long (classpath:/error1.json:2:19)

        - 'baz': Missing from config

        - 'hostname': Type defined as not-null but null was loaded from config (classpath:/error1.json:6:18)

        - 'season': Required a value for the Enum type com.sksamuel.hoplite.json.Season but given value was Fun (/home/user/default.json:8:18)

        - 'users': Defined as a List but a Boolean cannot be converted to a collection (classpath:/error1.json:3:19)

        - 'interval': Required type java.time.Duration could not be decoded from a String (classpath:/error1.json:7:26)

        - 'nested': - Could not instantiate 'com.sksamuel.hoplite.json.Wibble' because:

            - 'a': Required type java.time.LocalDateTime could not be decoded from a String (classpath:/error1.json:10:17)

            - 'b': Unable to locate a decoder for java.time.LocalTime

Supported Formats

Hoplite supports config files in several formats. You can mix and match formats if you really want to. For each format you wish to use, you must include the appropriate hoplite module on your classpath. The format that hoplite uses to parse a file is determined by the file extension.

Format	Module	File Extensions
Json	`hoplite-json`	.json
Yaml Note: Yaml files are limited 3mb in size.	`hoplite-yaml`	.yml, .yaml
Toml	`hoplite-toml`	.toml
Hocon	`hoplite-hocon`	.conf
Java Properties files	built-in	.props, .properties

If you wish to add another format you can extend Parser and provide an instance of that implementation to the ConfigLoaderBuilder via addParser.

That same function can be used to map non-default file extensions to an existing parser. For example, if you wish to have your config in files called application.data but in yaml format, then you can register .data with the Yaml parser like this:

ConfigLoaderBuilder.default().addParser("data", YamlParser).build()

Note: fatJar/shadowJar

If attempting to build a "fat Jar" while using multiple file type modules, it is essential to use the shadowJar plugin and to add the directive mergeServiceFiles() in the shadowJar Gradle task. More info

Property Sources

The PropertySource interface is how Hoplite reads configuration values.

Hoplite supports several built in property source implementations, and you can write your own if required.

The EnvironmentVariableOverridePropertySource, SystemPropertiesPropertySource and UserSettingsPropertySource sources are automatically registered, with precedence in that order. Other property sources can be passed to the config loader builder as required.

EnvironmentVariablesPropertySource

The EnvironmentVariablesPropertySource reads config from environment variables. It does not map cases. So, HOSTNAME does not provide a value for a field with the name hostname.

For nested config, use a period to separate keys, for example topic.name would override name located in a topic parent. Alternatively, in some environments a . is not supported in ENV names, so you can also use double underscore __. Eg topic__name would be translated to topic.name.

Optionally you can also create a EnvironmentVariablesPropertySource with allowUppercaseNames set to true to allow for uppercase-only names.

EnvironmentVariableOverridePropertySource

The EnvironmentVariableOverridePropertySource reads config from environment variables like the EnvironmentVariablesPropertySource. However, unlike that latter source, it is registered by default and only looks for env vars with a special config.override. prefix. This prefix is stripped from the variable before being applied. This can be useful to apply changes at runtime without requiring a build.

For example, given a config key of database.host, if an env variable exists with the key config.override.database.host, then the value in the env var would override.

In some environments a . is not supported in ENV names, so you can also use double underscore __. Eg topic__name would be translated to topic.name.

SystemPropertiesPropertySource

The SystemPropertiesPropertySource provides config through system properties that are prefixed with config.override.. For example, starting your JVM with -Dconfig.override.database.name would override a config key of database.name residing in a file.

UserSettingsPropertySource

The UserSettingsPropertySource provides config through a config file defined at ~/.userconfig.[ext] where ext is one of the supported formats.

InputStreamPropertySource

The InputStreamPropertySource provides config from an input stream. This source requires a parameter that indicates what the format is. For example, InputStreamPropertySource(input, "yml")

ConfigFilePropertySource

Config from files or resources are retrieved via instances of ConfigFilePropertySource. This property source is added automatically when we pass strings to the loadConfigOrThrow or loadConfig functions.

There are convenience methods on ConfigLoaderBuilder to construct ConfigFilePropertySources from resources on the classpath or files.

For example, the following are equivalent:

ConfigLoader().loadConfigOrThrow<MyConfig>("/config.json")

and

ConfigLoaderBuilder.default()
   .addResourceSource("/config.json")
   .build()
   .loadConfigOrThrow<MyConfig>()

The advantage of the second approach is that we can specify a file can be optional, for example:

ConfigLoaderBuilder.default()
  .addResourceSource("/missing.yml", optional = true)
  .addResourceSource("/config.json")
  .build()
  .loadConfigOrThrow<MyConfig>()

JsonPropertySource

To use a JSON string as a property source, we can use the JsonPropertySource implementation. For example,

ConfigLoaderBuilder.default()
   .addSource(JsonPropertySource(""" { "database": "localhost", "port": 1234 } """))
   .build()
   .loadConfigOrThrow<MyConfig>()

YamlPropertySource

To use a Yaml string as a property source, we can use the YamlPropertySource implementation.

ConfigLoaderBuilder.default()
   .addSource(YamlPropertySource(
     """
        database: "localhost"
        port: 1234
     """))
   .build()
   .loadConfigOrThrow<MyConfig>()

TomlPropertySource

To use a Toml string as a property source, we can use the TomlPropertySource implementation.

ConfigLoaderBuilder.default()
  .addSource(TomlPropertySource(
    """
        database = "localhost"
        port = 1234
     """))
  .build()
  .loadConfigOrThrow<MyConfig>()

PropsPropertySource

To use a java.util.Properties object as property source, we can use the PropsPropertySource implementation.

ConfigLoaderBuilder.default()
  .addSource(PropsPropertySource(myProps))
  .build()
  .loadConfigOrThrow<MyConfig>()

Cascading Config

Hoplite has the concept of cascading or layered or fallback config. This means you can pass more than one config file to the ConfigLoader. When the config is resolved into Kotlin classes, a lookup will cascade or fall through one file to another in the order they were passed to the loader, until the first file that defines that key.

For example, if you had the following two files in yaml:

application.yaml:

elasticsearch:
  port: 9200
  clusterName: product-search

application-prod.yaml:

elasticsearch:
  host: prd-elasticsearch.scv
  port: 8200

And both were passed to the ConfigLoader like this: ConfigLoader().loadConfigOrThrow<Config>("/application-prod.yaml", "/application.yaml"), then lookups will be attempted in the order the files were declared. So in this case, the config would be resolved like this:

elasticsearch.port = 8200 // the value in application-prod.yaml takes priority
elasticsearch.host = prd-elasticsearch.scv // only defined in application-prod.yaml
elasitcsearch.clusterName = product-search // only defined in application.yaml

Let's see a more complicated example. In JSON this time.

default.json

{
  "a": "alice",
  "b": {
    "c": true,
    "d": 123
  },
  "e": [
    {
      "x": 1,
      "y": true
    },
    {
      "x": 2,
      "y": false
    }
  ],
  "f": "Fall"
}

prod.json

{
  "a": "bob",
  "b": {
    "d": 999
  },
  "e": [
    {
      "y": true
    }
  ]
}

And we will parse the above config files into these data classes:

enum class Season { Fall, Winter, Spring, Summer }
data class Foo(val c: Boolean, val d: Int)
data class Bar(val x: Int?, val y: Boolean)
data class Config(val a: String, val b: Foo, val e: List<Bar>, val f: Season)

val config = ConfigLoader.load("prod.json", "default.json")
println(config)

The resolution rules are as follows:

"a" is present in both files and so is resolved from the first file - which was "prod.json"
"b" is present in both files and therefore resolved from the file as well
"c" is a nested value of "b" and is not present in the first file so is resolved from the second file "default.json"
"d" is a nested value of "b" present in both files and therefore resolved from the first file
"e" is present in both files and so the entire list is resolved from the first file. This means that the list only contains a single element, and x is null despite being present in the list in the first file. List's cannot be merged.
"f" is only present in the second file and so is resolved from the second file.

Strict Mode

Hoplite can be configured to throw an error if a config value is not used. This is useful to detect stale configs.

To enable this setting, use .strict() on the config builder. For example:

ConfigLoaderBuilder.default()
  .addResourceSource("/config-prd.yml", true)
  .addResourceSource("/config.yml")
  .strict()
  .build()
  .loadConfig<MyConfig>()

An example