Background

We live in an age of Drupal complexity. In the early days of Drupal, many developers would have a single Drupal instance/environment (aka copy) that was their production site, where they would test out new modules and develop new functionality. Developing on the live website however sometimes met with disastrous consequences when things went wrong! Over time, technology on the web grew, and nowadays it's fairly standard to have a Drupal project running on multiple environments to allow site development to be run in parallel to a live website without causing disruptions. New functionality is developed first in isolated private copies of the website, put into a testing environment where it is approved by clients, and eventually merged into the live production site.

While multiple environments allow for site development without causing disruptions on the live production website, it introduces a new problem; how to ensure consistency between site copies so that they are all working with the correct code.

This series of articles will explore the Configuration API, how it enables functionality to be migrated between multiple environments (sites), and ways of using the Configuration API with contributed modules to effectively manage the configuration of a project. This series will consist of the following posts:

• Part 1: The Configuration API
• Part 2: How the API works (coming soon)
• Part 3: Using the API (coming soon)
• Part 4: Extending the API with contributed modules (coming soon)
• Part 5: Module developers and the API (coming soon)

Terminology

Before we get started, let's review some terminology. A Drupal project is the overall combination of the core codebase and all of the environments that are used for development of the project. An environment is a copy/clone of the website that is accessible at a unique domain name (URL). When most users on the web think of a website, they think of the environment accessed at a single URL, like google.com or facebook.com. These URLs are the entry to the live production environments for these websites. However, in addition to the production environment, large projects will have additional environments, where code is developed and tested before being deployed to the production environment. The combination of these environments make up the project. Drupal 8 is built to enable developers to migrate functionality for a project between environments, using the Configuration API.

Components of a Drupal environment

What makes up an environment of a Drupal project? In other words, when making a 'copy' of a Drupal site, what are the elements that need to be migrated/copied to create that copy? A Drupal environment consists of the following elements:

1. Codebase: At the core of any Drupal system is codebase that makes up the 'engine' that runs Drupal. The codebase of any Drupal site consists of Drupal core, modules (both contributed and custom), and themes (again, both contributed and custom). The codebase is a series of files, and these files provide the functionality of a Drupal site.
2. Configuration: Configuration is the collection of settings that define how the project will implement the functionality provided by the codebase. The codebase provides the abstract functionality to create things like content types, fields and taxonomies. Developers configure a Drupal site through the admin interface, to create the actual content types, fields, and taxonomies used in the project, that will allow end-users to do things like create pages or make comments. As developers build out the functionality of the project through the admin interface, the settings they choose are saved as the configuration for that environment. Configuration is environment-specific, and the Configuration API provides a means of migrating configuration between environments.
3. Content: Content is the data on a site that is specific to the given environment. Content may be created by end-users, or by content admins. While content, like configuration, sometimes needs to be migrated between environments, content should be considered environment-specific. The Migrate API provides a means of migrating content between environments.
4. Uploaded files: Most Drupal projects will have files uploaded as content. These are not part of the codebase that provides the functionality of the system, rather they are files that are to be provided to end users. Examples are images, audio/video files and PDFs.
5. Third party technologies: Drupal 8 is a hub framework, built to integrate with 3rd party libraries, APIs, softwares, and other technologies. For example, a project may use a 3rd party CSS/JS framework, or swap out the default database caching backend to a Redis caching backend, which provides significant performance improvements. While these third party technologies are not part of Drupal, often the technologies will be required in each environment for the project to run.

The above five elements together make up a Drupal environment. The environment is a fully-functioning Drupal copy (instance), with its own configuration and content. Each environment will have its own domain (URL) at which the environment can be accessed.

Types of environments

Typical projects these days will have a minimum of three environments:

1. Production environment: The main live environment that end users will use.
2. Staging environments: One or more environments on which site owners can test new functionality or bug fixes before they are deployed to the production environment.
3. Development environments, where new functionality can be developed in isolation. Development environments may be publicly accessible, or may only exist on a developers computer, not accessible to the outside internet.

Content and configuration - both stored in the database

A major issue that existed with all versions of Drupal core up to and including Drupal 7, was that content and configuration were both stored in the database, and there was no way to separate them. This made it difficult to manage configuration changes, such as adding a field to a content type, between multiple environments. There was no way to export the configuration for that single field, so to ensure consistency full databases were migrated. The problem is that database migrations are an all-or-nothing venture; when migrating from one environment to another, the entire database of the source environment overwrote the entire database on the target environment, meaning all content and configuration was overridden on the target environment.

This opens up a problem though. Imagine this scenario:

1. A developer copies the production database to their local computer.
2. Someone creates new content on the production environment, that becomes part of the database.
3. The developer adds a new field on their local environment.
4. The developer migrates their local database to the production environment, overwriting all configuration and content on the production environment.

With the above scenario, the new field created on the developer's local environment is now on the production environment, however the content created in step 2 is overwritten and lost. As such, the above process is untenable. Databases cannot be migrated 'up' (to production) after a site has gone live. They can only be migrated 'down' (from production).

To overcome this problem, a common way to make changes to configuration before Drupal 8 was to make configuration changes on the production environment, then copy the production environment's database down to the other environments. For example, if a developer was writing code that depended on a new field, rather than create the field on their local development environment, they would create the field on the production environment, then copy the production environment database down to their local development environment. Now the field exists on both the production environment as well as their development environment. This solution works, but is akin to killing a mosquito with a sledgehammer - it's overkill. The developer did not need the whole database, only the configuration for that single field. And any changes they had already made in their own environment, such as content for testing, is now wiped out with data from the production environment.

While this process worked (and still does continue to work for many projects), it was not ideal. It led to issues where database migrations had to be coordinated with clients and other developers to ensure that it was safe to wipe out content or configuration someone else may still be working with.

The Configuration API

The Drupal 8 Configuration API was created to overcome the above issue, so that developers could migrate configuration - such as Field API fields, content types, and taxonomies - between environments, without having to overwrite the entire database. It decoupled configuration from content. To understand how it makes things better, let's again review the components of a Drupal site, and the storage mechanisms behind them:

• Codebase (files)
• Configuration (database)
• Content (database)
• Uploaded files (files)

With the Configuration API, configuration can be exported into files, and these files can be migrated between environments and imported into the new environment. This means that we can now migrate individual items of configuration without having to overwrite the entire database. The Configuration API decouples configuration from content. In the next parts of this series, we will explore how the API work

Note 1: the Drupal 7 Features module essentially does the same thing, and though the methodology is different, many of the concepts in the following articles will be relevant to that module as well.

Note 2: the Drupal 8 Migrate API was developed in parallel to the Configuration API, allowing for content also to be migrated, again without overwriting the entire database.

Summary

In this article, we looked at an overview of the Configuration API to understand what it is, and why it was created. After reading this article, you should have an understanding of what a Drupal environment is, and what it consists of. You should understand that in Drupal 8, configuration is stored in the database, but can be exported to files, which can then be used to migrate content between environments. In Part 2 - How the API works, we will take a closer, more technical look at how the API works, and in Part 3 - Using the API, we will discuss how to use the API, and some contributed modules that extend its functionality.