TL&DR: Use drupal.org's issue forks to make Drupal 9 compatibility fixes work with Composer.

While most software developers are in agreement on the two hardest things in software development – cache invalidation, naming things, and off-by-one errors – not everyone is in agreement on how to rank the rest of the top ten challenges. One that must surely rank high is dependency management algorithms and dependency management tools. These make sure that different libraries and additions to a codebase are kept up to date and APIs are kept compatible. For example, supposing a codebase has SquareWidget v2 and CircleWidget v2; if SquareWidget v3 comes out but is incompatible with CircleWidget v2, the codebase's dependency management tool would prevent updating to SquareWidget v3 until a compatible version of CircleWidget was available.

In the Drupal world we've historically avoided formal dependency management as we could just download a package from drupal.org and get running, occasionally realizing "Oh I needed CTools too" and grabbing it. Along the way some folks built the Drush tool which, amongst other things, could download these dependencies automatically. It wasn't until Drupal 8 came along that more formal dependency management became a thing because of its heavy use of 3rd party libraries, in large part thanks to the use of the Composer system. This tool came out of the wider PHP community's need for a generic, reliable dependency management system, and in the Drupal maintainers' drive to adopt more 3rd party libraries and tools it was the obvious choice. After an initial bumpy learning phase, almost all contemporary Drupal 8 and 9 website projects today are managed using Composer.

The drupal.org infrastructure provides a custom wrapper around Drupal core, module, and theme meta data so that it can be loaded using Composer using its metadata platform Packagist. Modules and themes which already include a composer.json file will have that made available as-is. However, a large portion of Drupal 8 and 9 contrib projects don't have this, so the drupal.org infrastructure maps the info.yml files into a format Composer can understand. This way a module that was initially ported to Drupal 8 a few years ago can still be added to a contemporary D8 project managed with Composer, even if the module hasn't been touched in years. It's awesome.

The World of Drupal 9 Updates

Back in October 2019, a new feature was added to core 8.7.8 which allowed modules to specify the versions they were compatible with by using a new line in their info files. This new line became a requirement on Drupal 9 as there needed to be an indication in modules & themes to indicate what APIs they were compatible with. For most projects the new line makes the info file look like this – note how the old "core" value is now replaced with a "core_version_requirement" value:

name: Commerce Migrate
type: module
description: Provides various Commerce-specific Migrate handlers.
core_version_requirement: ^8.8 || ^9
package: Commerce (contrib)
dependencies:
 - drupal:migrate
 - drupal:telephone
 - migrate_plus:migrate_plus

A module (or theme) could use the new line to indicate they were compatible with both Drupal 8 and 9 simultaneously, and the majority of the most popular modules have made the necessary changes.

Drupal 9 presented the first major release of Drupal core that was such a low impact update it was possible to run many websites on either 8.9 or 9.0 just by swapping the core files out (and updating the dependencies). Gone are the days of having to rebuild a site from scratch for each major upgrade, instead we just have to keep our codebases fully up to date and swap to the new core release pretty quickly.

Except for that one line.

That one new line has to be in each info.yml file in the codebase (except for test files, but that's a different matter), so any under-maintained or unmaintained module or theme will need to be updated. Thanks to the wonders of modern development tools, it was estimated that almost a quarter of all Drupal 8 modules & themes could be upgraded to be compatible with Drupal 9 by just changing their info file! Over the course of 2020, thanks to contributions and collaborations from folks all over the world, a huge number of modules and themes were updated to be fully compatible with Drupal 9, and a large portion of those that don't have releases have patches available.

The Catch-22

The fact that there are patches to make Drupal 8 modules & themes compatible with Drupal 9 is great for maintainers or would-be maintainers - they don't need to go through the efforts of making all of the changes themselves, they can just review what has been provided and, hopefully, commit it. Normally patches are great for end-users too, because again they don't have to take the time to make the change themselves, someone has made it available for them.

Here is what happens when you apply a patch using Composer:

1. Composer downloads the project's listing from Drupal's custom Packagist system (see above).
2. It compares the dependencies from the listing against what's currently in the codebase.
3. It deletes the existing copy of the underlying module or theme, if present.
4. It downloads a fresh copy of the module/theme that matches the dependencies.
5. It applies the patch.

Normally this patch process works great - you find a patch, add to your codebase, and away you go, remembering to leave a comment to the patch creators how well it works for you. However, there's a major limitation here - even if the patch contains changes to the composer.json dependencies, it's applied after Composer has decided whether or not to install it.

In short, you can't use a patch to tell Composer that a Drupal 8 module is compatible with Drupal 9.

Improved Code Collaboration Workflows

Drupal originally used CVS to manage the codebase. This was very limited by today's standards, but it was reasonable back in the early 2000s and provided a means to centrally manage the large codebases of both core and the ever expanding array of contributed modules & themes. Proposed changes to these codebases were handled using patch files, which are simply text files which indicate a file to be modified, which lines are to be removed and which are to be added. It worked well enough, but there was a large learning curve for beginners to get past.

Over the years more flexible & functional replacements for CVS became common, including centralized systems like Subversion and decentralized systems like Mercurial, Perforce or git. Rather than take the short jump to another centralized system, the effort was taken to build out a replacement code management platform using git, under the umbrella project name of "The Great Git Migration". Completed in 2011, the effort was lead by Sam Boyer, and the community has been all the better for it.

However, after the git migration was completed the community was still stuck with patch files. While github had its pull request workflow, the Drupal community was screaming at the need for somewhat archaic collaboration workflows.

Skip ahead nine years and an awful lot of discussion and research, in 2020 the community finally had a replacement code collaboration workflow in the form of merge requests via the Gitlab system. This new workflow allows anyone to create a fork of a project, make changes, and then create a gihub-pull-request -like change request, dubbed a "merge request", for others to review. Unlike github's pull request system, it's also really easy for anyone to collaborate on the same merge request, which lends itself really nicely to collaboration with others rather than solo development. After some opt-in beta testing, the new system was launched community-wide for every single code project on drupal.org to use.

The new issue fork and merge request system is based upon the simple concept that each individual issue on drupal.org can have its own copy of that project's codebase, an issue fork, and that codebase can be downloaded individually using git. With an issue fork anyone can make whatever changes they need directly with git and others can then download those changes directly using git - no additional tools needed, no patch files flowing around.

This also means that it's possible to tell Composer to download the codebase from an issue fork instead of from the main repository.

This means that an issue fork can be used to get around Composer's patch-vs-dependencies catch-22!

Putting it All Together

First off, it should be noted that issue forks are, to all intents and purposes, a separate physical repository than the parent project they fork from. This means that you cannot just download the issue fork by telling Composer to use a specific branch of the main project, Composer has to be told to use a completely different repository.

It's also worth bearing in mind that, for a given Drupal project (module or theme), only one issue fork can be used at a time. Because an issue fork is a separate repository, it isn't possible to download two different versions of the same module/theme and magically have them squish together. Therefore, if multiple merge requests / issue forks are needed for a given project then the others have to be applied as patches; alternatively, a separate "meta" issue could be created that combines multiple changes into one merge request, but at that point it might be easier to just become a maintainer and commit the changes.

In this example, I'm going to use the merge request created for the Drupal 9 compatibility issue for the Commerce Migrate module.

• First off, find the issue fork portion of the d.o issue, which should be right underneath the list of attachments & patches, which is underneath the issue summary.
  
• Click the "Show commands" button to expand out the example git commands.
  
• In both the "Add & fetch" sections there will be a "git remote add" line. Included in this is a URL that's needed to download the codebase from the issue fork - one starts with "git@git.drupal.org" while the other starts with "https://git.drupalcode.org". Copy the full line (click the "copy" icon to copy it to the clipboard) and extract the URL, e.g. git@git.drupal.org:issue/commerce_migrate-3150733.git.
• Click the "commands" button again to hide them, and then get the branch name, which in the example above is "3150733-drupal-9-compatibility".
• In the site's composer.json file, in the "repositories" section add a new item with two values: {"type": "git", "url": "URLFROMABOVE"} e.g.:

         {
             "type": "git",
             "url": "git@git.drupal.org:issue/commerce_migrate-3150733.git"
         }

• Look for the item in the "repositories" section that has "type" set to "composer". If it doesn't exist already, add an item called "exclude" and make it a list. Add the Composer name of the module/theme you want to use, e.g. "drupal/commerce_migrate", so that it looks like this:

         {
             "type": "composer",
             "url": "https://packages.drupal.org/8",
             "exclude": ["drupal/commerce_migrate"]
         },

• Change the listing for the project in the "require" (or "require-dev") section to point to the branch name identified above, e.g. "drupal/commerce_migrate": "dev-3150733-drupal-9-compatibility",
• Save the changes to the file.
• Update the project in composer, e.g. composer update drupal/commerce_migrate.

The last command will now download that project from the issue fork instead of the main codebase.

Note: these should only be used as a short term solution, the goal should always be to collaborate to get changes committed so that these steps aren't needed.

(there might be other ways of doing this using repository priorities, but this method works)

But it Didn't Work?

One problem that can arise is that Composer can't process the project, which it will tell you with this error message:

  [Composer\Repository\InvalidRepositoryException]
  No valid composer.json was found in any branch or tag of git@git.drupal.org:issue/commerce_migrate-3150733.git, could not load a package from it.

This simply means that the project doesn't have a "composer.json" file in it, so you can fix that by adding a composer.json file to the repository. Once that is created (make sure to run "composer validate" before saving it!) and uploaded to the issue fork, it'll be possible to download it to a site's codebase again.

Put That Thing Back Where it Came From or So Help Me

Because they don't keep current with upstream changes and can fall out of date quickly, issue forks should be used sparingly in website projects. As it happens, the patch for Commerce Migrate I wrote this blog post around was committed between the time I started the blog post and it was published – "The Drop Is Always Moving", as they say.

When the day arrives and the project has its Drupal 9 fixes committed, there are a few steps to remove the issue hackery and make the website's codebase happy again.

1. Remove the extra "repositories" item.
2. Remove the "exclude" line from the "type":"composer" repository; if there aren't any remaining items in the "exclude" section it can be removed entirely.
3. Change the "require" line (or "require-dev" line) back to point to the appropriate release that includes the Drupal 9 fixes.
4. Run "composer validate" to make sure the compost.json file is correct.
5. Run "composer update drupal/PROJECTNAME" to get the new, cleaner version of the project.
6. Commit the changes.
7. Celebrate.

That Was a Lot of Words, Do You Have a Picture?

This topic was covered in a recent Contrib Half Hour. Because I forgot to record that meeting (I'm a professional, honest) I repeated the steps the following week, so now there's a recording of me stepping through the process to create an issue fork to make a Drupal 9 fix for a Drupal 8 module work in Composer: