Configuration management with omegaconf

A simple showcase of why I use omegaconf in my projects to manage configurations.

Configuration management is a crucial aspect of software development that often doesn’t get the attention it deserves. In this post, I’ll explore why configuration management is important and dive into why omegaconf has become my go-to solution for Python projects.

Why Configuration Management Matters

In modern software development, applications often need to be configurable across different environments (development, staging, production) and may require various settings like database connections, API keys, and feature flags. Managing these configurations effectively is essential for:

• Maintaining different environments without code changes
• Securing sensitive information
• Making applications more flexible and maintainable
• Managing feature/settings toggles
• Supporting different deployment scenarios

Enter OmegaConf

OmegaConf is a YAML-based hierarchical configuration system for Python applications that provides everything I need in one package.

Here’s what makes it special:

• Type-safe configuration with optional runtime type checking
• Config composition and merging
• Variable interpolation (including nested interpolation)
• Environment variable support via the ${oc.env:} resolver
• Custom resolvers for dynamic values based on Python functions
• Structured configuration with validation
• Support for both YAML and Python dictionary inputs

Features in Action

Let’s explore some of OmegaConf’s key features with practical examples.

1. Templating, Placeholders, and Hierarchical Access

First, let’s create a configuration that demonstrates these features:

from pprint import pprint
from omegaconf import OmegaConf

conf = OmegaConf.create('''
domain: company.com
users: 
    sinan:
        alias: sinanpl
        email: ${.alias}@${domain}
'''
)

# Resolve all interpolations and convert to a regular Python dict
OmegaConf.to_container(conf, resolve=True)

{'domain': 'company.com',
 'users': {'sinan': {'alias': 'sinanpl', 'email': 'sinanpl@company.com'}}}

Notice how email is interpolated with the alias and domain variables in the main and user entry.

We can easily access nested values using dot notation:

# Access nested values using dot notation
print(conf.users.sinan.email)

sinanpl@company.com

2. Environment Variables

OmegaConf makes it easy to integrate environment variables using the oc.env resolver. When the configuration is loaded, it will automatically replace the placeholder with the actual environment variable value:

from omegaconf import OmegaConf

conf = OmegaConf.create('''
database_password: ${oc.env:DATABASE_PASSWORD}
env: ${oc.env:ENVIRONMENT_NOT_SET,development_as_default} # env variable default
'''
)

# Resolve all interpolations and convert to a regular Python dict
pprint(OmegaConf.to_container(conf, resolve=True))

{'database_password': 'my_very_secret_db_password',
 'env': 'development_as_default'}

3. Custom Resolvers

OmegaConf allows you to create custom resolvers for dynamic values. This is particularly useful for generating timestamps, file paths, or any other runtime values requiring Python processing:

from omegaconf import OmegaConf
from datetime import datetime

# Register a custom resolver that returns today's date
OmegaConf.register_new_resolver(
    "py_today", 
    lambda: datetime.now().date().isoformat(), 
    replace=True
)

conf = OmegaConf.create('''
reporting_date: ${py_today:}
reporting_filename: ${reporting_date}-report.csv
''')

pprint(OmegaConf.to_container(conf, resolve=True))

{'reporting_date': '2025-03-22', 'reporting_filename': '2025-03-22-report.csv'}

Config Merging & Putting It All Together

There are some config files in my directory. In a real project, this is easily more extensive and complicated.

Click to see the config files in my directory

# conf/globals.yml
app:
  name: MyApp
  version: 1.0.0
  debug: false
  report_date: ${py_today:}

database:
  host: localhost
  port: 5432
  password: ${oc.env:DATABASE_PASSWORD}
  name: mydb
  full_url: ${database.host}:${database.port}/${database.name}

# conf/data_science.yml
ds:
  models: 
    rf:
      n_estimators: 100
      max_depth: 10
      random_state: 42
      input_features:
        - feature_1
        - feature_2
        - feature_3
    xgboost:
      n_estimators: 100
      max_depth: 10
      random_state: 42 
      input_features:
        - feature_1
        - feature_2
        - feature_3

Here’s a complete example that combines all these features into a practical configuration:

from omegaconf import OmegaConf
from pathlib import Path
from datetime import datetime
from functools import reduce

# Register today's date resolver
OmegaConf.register_new_resolver("py_today", lambda: datetime.now().strftime("%Y-%m-%d"), replace=True)

configs = [OmegaConf.load(p) for p in Path("conf").glob("*.yml")]
all_conf = reduce(OmegaConf.merge, configs)

pprint(OmegaConf.to_container(all_conf, resolve=True))

{'app': {'debug': False,
         'name': 'MyApp',
         'report_date': '2025-03-22',
         'version': '1.0.0'},
 'database': {'full_url': 'localhost:5432/mydb',
              'host': 'localhost',
              'name': 'mydb',
              'password': 'my_very_secret_db_password',
              'port': 5432},
 'ds': {'models': {'rf': {'input_features': ['feature_1',
                                             'feature_2',
                                             'feature_3'],
                          'max_depth': 10,
                          'n_estimators': 100,
                          'random_state': 42},
                   'xgboost': {'input_features': ['feature_1',
                                                  'feature_2',
                                                  'feature_3'],
                               'max_depth': 10,
                               'n_estimators': 100,
                               'random_state': 42}}}}

Conclusion

OmegaConf provides a powerful and flexible way to manage configurations in Python applications. Its features like hierarchical access, environment variable integration, and custom resolvers make it an excellent choice for both small and large projects. The ability to combine static configuration with dynamic values and environment-specific settings makes it particularly useful in modern application development.

Whether you’re building a small script or a large-scale application, OmegaConf can help you manage your configurations more effectively and securely.

For more information, check out the OmegaConf documentation. If you’re working on more complex applications, consider using Hydra, a more extensive framework built on top of OmegaConf.