by David Snopek on February 12, 2019 - 2:26pm

One the most popular articles on our blog is an article I wrote a year and half ago about how to install CiviCRM on Drupal 8.

The method described there worked (and still more-or-less works), but it's... a mess.

It involves running a dozen or so commands, and is pretty easy to get wrong. All of this is just to get code assembled in such a way that you CAN install it.

I'm happy to announce that you can now do this in just a single command!

There's still some little issues and bugs with running CiviCRM on Drupal 8 that need to be manually worked around, but getting over that first hurdle of simply allowing you to install it in the first place should be significantly easier with this new method.

Read the full article to find out how!

Background: Why is this hard?

The biggest problem is that both Drupal 8 and CiviCRM depend on some of the same PHP libraries, for example, Symfony.

In PHP, you can't define the same class twice (this would be a fatal error) and you certainly can't define the same class twice at two different versions (for example, one version for Drupal and one for CiviCRM).

So, you couldn't just copy CiviCRM with all its dependencies into an instance of Drupal, because some of those dependencies would conflict with what's already in Drupal.

And, unfortunately, you couldn't just make a special CiviCRM bundle that's "optimized" for a particular version of Drupal 8, because each Drupal 8 site is potentially unique: you can update the PHP libraries used by a Drupal 8 site (for example, upgrading Symfony to a newer version) or add new PHP libraries that could conflict.

The Magic of Composer

Composer is a tool that's used by PHP applications and libraries to find and download a compatible set of dependencies.

For example, CiviCRM needs Symfony 2 (version 2.8.44 or greater) or any version of Symfony 3. Drupal 8.6.9 needs Symfony 3.4 (version 3.4.14 or newer), although, soon it will be possible to use Drupal 8 with Symfony 4 as well.

Using composer, you can say, "I need Drupal 8.6.9 and CiviCRM 5.10.0" and it can pull in a version of Symfony that works for both.

If your Drupal site used Symfony 4 (because some Drupal module needed it), it would error out and say you need to either remove that module (so Symfony can be downgraded to Symfony 3) or remove CiviCRM.

That's why composer is so heavily involved in getting Drupal and CiviCRM to work together!

The peculiarities of CiviCRM

Of course, that's not the whole problem, because otherwise it would have been possible to solve this with a single composer command years ago.

CiviCRM has been around for over a decade (which is a lifetime ago in the PHP ecosystem), and still has some legacy pecularilies that need to be accounted for..

Namely, CiviCRM depends on going through a non-composer-y build process to generate a "release" that is actually usable.

Some of those things that need a build process could be reworked in a way so that they didn't need it, and others could be done in a composer-compatible way, such that CiviCRM would work like any composer library. Work is being done on that, but those are hard problems which will take time to solve.

TL;DR - What are the commands?

Heh, alright! Time for the actual actionable steps. :-)

First, you need to have the following installed:

And, make sure that you have a recent version of Composer! A couple of people have tried to use this process with older versions and have experienced issues. (In general, you shouldn't be using a composer that's older that 90 days - the Composer eco-system evolves quickly!)

Then, to create a new Drupal 8 site with CiviCRM:

# replace 'some-dir' with the directory to create
composer create-project roundearth/drupal-civicrm-project:8.x-dev some-dir --no-interaction

Or, to add to an existing Drupal 8 site (assuming you used the best practice method of starting from the drupal-project composer template):

composer require roundearth/civicrm-composer-plugin civicrm/civicrm-drupal-8

If you have a Drupal 8 site that isn't based on the drupal-project composer template, well, you're going to need to convert your site.

I'd recommend creating a new codebase (using the 'composer create-project' command above), adding all the modules/themes/libraries from the old codebase, and then switching the site to the new codebase. Assuming you didn't forget to copy anything over, that should work, but definitely keep a backup.

Installing CiviCRM

Once you have the code in place using the commands above, you'll actually need to install CiviCRM. Due to some bugs in CiviCRM for Drupal 8, there's a bunch of caveats and extra steps associated with that.

I've documented some of the steps on the project page for 'roundearth/drupal-civicrm-project':

https://gitlab.com/roundearth/drupal-civicrm-project#installing-civicrm

Hopefully, all these bugs will be fixed in time.

The good news is that the composer project template will always pull in the latest versions of CiviCRM and its Drupal 8 integration, so these steps should be able to remain the same, and will just start working better as those bugs get fixed. :-)

How does it work?

There are two parts to this:

It does the following:

  1. Run's Bower to pull in Javascript and CSS dependencies
  2. Downloads a release of CiviCRM and copies in the missing files generated by the usual build process
  3. Downloads any requested CiviCRM extensions
  4. Copies any web assets from CiviCRM (which is installed in the vendor directory, which isn't web accessible) into the web root

In case you're curious, or want to help improve it, here's where all the most interesting code lives:

https://gitlab.com/roundearth/civicrm-composer-plugin/blob/master/src/Handler.php

Conclusion

One of the main challenges in improving CiviCRM on Drupal 8, is just getting it installed for developers who might want to help.

We've been involved in the effort to port the Webform CiviCRM module to Drupal 8, which has also faced this same challenge.

That's why a small part of the development of this new method was funded via the donations that were made to that effort. The rest was just part of our efforts in maintaining Roundearth, a Drupal 8 and CiviCRM platform for non-profits.

We're hoping this will help, not only with getting CiviCRM installed on Drupal 8, but also help to accelerate development. :-)

Want to read more articles like this?

myDropWizard.com blog Subscribe to the myDropWizard.com blog and recieve e-mail updates when new articles are published!

Comments

Thank you so much ;-)
This is great.

Thanks David. Your work on this will certainly make it easier to get started with CiviCRM on Drupal 8 until official support is finished. Doing a fresh install I did have two problems though:

1) When I run the first command it fails: composer create-project roundearth/drupal-civicrm-project:8.x-dev some-dir --no-interaction
The error is: Problem 1 ... - civicrm/civicrm-core 5.10.2 requires zetacomponents/mail dev-1.7-civi -> no matching package found.

After going in to the subdirectory created by this command and running this command from the old instructions:
composer config repositories.zetacomponents-mail vcs https://github.com/civicrm/zetacomponents-mail.git
I could see that the problem was that Github's API limit was reached. Creating a new personal token was the answer. This does not happen if zetacomponents is already in composer's cache or composer has already been configured with a Github personal token. This might be normal but I couldn't see it in the docs so I'm posting it for others to see.

2) I get this error when trying to navigate to CiviCRM's dashboard: 'DB Constraint Violation - contact_id should possibly be marked as mandatory for DashboardContact,create API. If so, please raise a bug report.'
No CiviCRM contact is created. The workaround of logging out of Drupal and logging in doesn't work for me.

Thanks :-)

Regarding problem nr 1: It's pretty inevitable when using composer that you'll need to configure a GitHub token, however, the error message from composer is usually better. :-( The last time I saw it, I remember it explicitly telling me to create a GitHub token and to copy-paste it in.

Regarding problem nr 2: Hm, that's strange. That's exactly the error message that is supposed to be fixed by logging out and logging back in again! Maybe try deleting all cookies and all sessions from the session table?

Thanks David. I have got around problem 2. I think the extra step that I had to do was to edit & save the Drupal profile again, before logging out and in again.

Ah, ok, great! Glad you got it working :-)

Thanks for all the hard work you and your team have done making it possible for Drupal 8 and CiviCRM to work together. And sharing it. And documenting it. David's team also created the foundation of the Drupal 8 version of CiviCRM Entity, https://www.drupal.org/project/civicrm_entity , All very huge for the CiviCRM community!

First of all, thank you for the effort you have already put in to getting this install process working...

I am trying to add CiviCRM to an existing drupal 8.6.7 test site that was installed using composer, before installing it into similarly configured production site. I of course skipped to the TL;DR section, installed bower and issued the composer requires command as listed.

That appeared to work just fine, however, when I enabled the module it causes the ubiqitous "Website has encountered a problem" page. I looked in the logs, and the problem was that I am missing the "civicrm.settings.php" file. No problem says I, and I created an empty one, half hoping it would realize that the settings weren't right, and go into a configuration mode like Drupal does on a fresh install. It didn't, but now it complains that "civicrm_root" is misconfigured.

I can't seem to find any relevant information regarding what I should set in that file to resolve this problem. I found "vendor/civicrm/civicrm-core/templates/CRM/common/civicrm.settings.php.template" which appears to be a template file, and I tried by hand to fill it out, but it was still upset, and filling templates by hand feels "wrong".

I strongly suspect that I have missed a step in the installation somehow, but I cannot find any thing that shines light on what I may have done wrong, except for the requirements page in the CiviCRM "System Administrator Guide" which advises that Drupal 8 is unsupported, and to use Drupal 7.

Could you provide any hints on how to resolve the problem I am facing?

Thanks again!

I haven't ever seen this issue but I have a theory: installing the 'civicrm' module is supposed create the civicrm.settings.php, but the /sites/default directory is usually not writeable on a live site. So, this is what I'd try:

  1. Disable the 'civicrm' module
  2. Run: chmod a+w sites/default/
  3. Enalbe the 'civicrm' module
  4. Run: chmod a-w sites/default/

Please let me know if that works!

Thanks for getting back to me

I had to use drush pm:uninstall to uninstall as the site is WSOD with CiviCRM enabled. Reenabling the module either with drush or throught the admin pages does not work, this showed up in the relevent nginx error log...
2019/03/01 13:35:36 [error] 3835#3835: *302156 FastCGI sent in stderr: "PHP message: PHP Fatal error: require_once(): Failed opening required '/www/drupal/drupal-8.x/d8_root/vendor/civicrm/civicrm-core/packages/DB.php' and there is no packages directory.

In the meantime, I was working on getting the site working using "rinse and repeat" with the submit button and the logfile, the significant majority of the errors were similar to
2019/03/01 13:16:55 [error] 3835#3835: *301920 FastCGI sent in stderr: "PHP message: Error: Class 'CRM_Utils_File' not found in /www/drupal/drupal-8.x/d8_root/vendor/civicrm/civicrm-core/CRM/Core/Config/Runtime.php on line 103
and were fixed by adding something like
require_once 'CRM/Utils/File.php';
to the file listed.

There are _many_ of these which leads me to believe that something is wrong in my installation, although using composer to remove and reinstall the components did not resolve anything. I am tempted to start a new site using the from scratch instructions above, just to see what happens. Unfortunatly, this would cause an impact should I need to do it on the target site.

This first one:

"PHP message: PHP Fatal error: require_once(): Failed opening required '/www/drupal/drupal-8.x/d8_root/vendor/civicrm/civicrm-core/packages/DB.php' and there is no packages directory.

It sounds like the Composer plugin didn't put the vendor/civicrm/civicrm-core/packages directory in place. When you ran composer, did you see the output from '[civicrm-composer-plugin]' saying it was copying missing files from the CiviCRM release?

And this one:

2019/03/01 13:16:55 [error] 3835#3835: *301920 FastCGI sent in stderr: "PHP message: Error: Class 'CRM_Utils_File' not found in /www/drupal/drupal-8.x/d8_root/vendor/civicrm/civicrm-core/CRM/Core/Config/Runtime.php on line 103

Sounds even worse: like vendor/civicrm/civicrm-core is missing altogether. Maybe you need to try running 'composer install' or 'composer update' again?

Also, just to confirm, you're Drupal 8 was created using the drupal-project composer template? If not, all this composer stuff probably won't work right, but Drupal 8 from the tarball isn't 100% composer-friendly.

I gave the create-project install a try, and that has exhibited the same issues with no civicrm.settings.php file created, and when I do create one, the civicrm_root variable is ignored. It seems that the code tests it by trying to include_once 'CRM/Core/Config.php', but as $civicrm_root is never added to the path so of course the include_once fails to find the class and modules/contrib/civicrm/src/Civicrm.php throws an exception. Unless the path manipulation is in the bottom of the settings file, this would still fail and cause the WSOD.

Yes, the original drupal install was performed using composer, I don't remember the project name, but 'drupal-project' sounds familiar. I just completed a composer remove/composer require cycle and did not see anything related to copying files, just a slew of statements starting with "Installing", followed by some "suggests" statements for other packages.

vendor/civicrm/civicrm-core does exist with vendor being at the same level as the web directory, however I cannot find DB.php anywhere in the installed files.

[email protected]:/www/drupal/drupal-8.x/d8_root# find . -name DB.php
[email protected]:/www/drupal/drupal-8.x/d8_root#

I have it partially working, what I ended up doing was
1. issue the composer command above
2. from an installed drupal 7 tree, copy the the packages directory into the appropriate drupal 8 folder
3. ensure the settings folder is writable
4. drush en this will generate the settings file.
5. from the drupal 7 tree, copy the necessary .mysql files and apply them to the database your drupal 8 database is using

This will result with some things working, however, you will notice that civicrm will loose any form of theming, as well as what appears to be the majority of the CSS. The one thing I am interested in is memberships and that does not seem to write data to the database. So you can't set up memberships.

Unfortunately this appears to be another case where drupal 8 is not ready and my only path forward is to recreate the existing site on drupal 7 and install civicrm there. Hopefully there will be an upgrade path in the future.

Thanky ou for your work and help with this, it's a pity it didn't work out for this organization.

Something weird is happening in your environment.

2. from an installed drupal 7 tree, copy the the packages directory into the appropriate drupal 8 folder

The composer plugin should definitely copy the packages directory over, and if it fails to, it should error the whole process out. So, it really seems like the composer plugin isn't running at all. Do you see any lines starting with "> [civicrm-composer-plugin]" in your composer output? If the plugin is installed and working you should see 5 such lines. If not, what version of composer are you using? Do you see the plugin in your composer.json?

If the composer plugin isn't copying the packages over, then it's probably not doing the dozen other things the plugin is supposed to be doing...

The one thing I am interested in is memberships and that does not seem to write data to the database. So you can't set up memberships.

We have a half dozen Drupal 8 & CiviCRM sites running live for customers.

Not all of them use memberships, but some of them do. So, I know for a fact that membership do work with Drupal 8 & CiviCRM once you manage to get it installed and working correctly!

I don't know if your problem with memberships is related to your installation problems, or if it's another problem, but it should work. :-)

My current version of composer is 1.2.2...

I did a remove and require, in the require the only thing I saw referring to civicrm-composer-plugin was
- Installing roundearth/civicrm-composer-plugin (dev-master d0d8075)
Cloning d0d8075ae11df0a5581c38991bb9783afe1f9616 from cache
and in composer.json this is in the require section
"roundearth/civicrm-composer-plugin": "dev-master",

I updated my version of composer to 1.8.4, and tried over, this time I got
> [civicrm-composer-plugin] Running bower for CiviCRM...
followed by
[ErrorException]
Undefined variable: civicrm_package

Which looks like definite progress!

I am pretty sure all of the issues I am seeing are related to the bad install, as you mention, there is no telling what it hasn't done on my behalf :)

Aha! Composer will yell at you if you're using a version that's older than 90 days, and 1.2.2 is from November 8, 2016. :-) I'll add a note to the article about making sure to use a recent composer, though.

> [civicrm-composer-plugin] Running bower for CiviCRM...
followed by
[ErrorException]
Undefined variable: civicrm_package

Ah, thanks! That actually points to a bug the plugin. I've just pushed a commit to try and fix that:

https://gitlab.com/roundearth/civicrm-composer-plugin/commit/f6a4ab1d79a... 

Hopefully, it's not too much further until you have a working install :-)

It's strange because composer was not complaining at all.

That change did fix that issue, however when trying to install into the existing site, I got

[RuntimeException]
Unable to determine CiviCRM release version from dev-master

I tried to fix this by specifying a "real" version for civicrm-core and psr/log (it required a downgrade from 1.1.0) with

sudo -u www-data composer require civicrm/civicrm-drupal-8 civicrm/civicrm-core:5.10.4 psr/log:1.0.2 roundearth/civicrm-composer-plugin

This did not result in a working version because I think there were some leftovers from previous installs. So I tried a brand new site with a completely empty databse, and it at least installed without issue. Running through the pages makes me think that there is some missing javascript and/or css as the "configure dashboard" link doesn't work, nor do the pages with automatically updating fieldsets and tabs (ie memberships, contribution pages or payment processors) there also seems to be some "white-on-white" text. I am using a base bootstrap theme as it is representative to that which the organisation is using and I tried turning off the "aggregate XXX" performance settings and clearing the caches, but that does not seem to have resolved the problem.

Thank you for your help, I am almost close enough to taste it :)

The organisation uses a theme derived from bootstrap

Thanks once again, I think I am just about good to go!

I went nosing around with "Inspect Element" in my browser to try and figure out what was missing, and discovered that I had made a typo in the CiviCRM Resource URL setting... Fixing that has cleared up all the formatting issues I was seeing...

Awesome! Reading through your most recent comments, it doesn't sound like there's any more remaining problems? At least none I can find mentioned in the comments. :-)

No, there are no outstanding technical problems at the moment.

I just need to get the demo site finalized so the board can take it for a spin, and if they like it, I will install it into their existing site. I don't expect a significant problem there, as it appears the underlying cause was an old version of composer...

Thanks for the help, it's very much appreciated.

Anthony, I'm interested in knowing whether your New Mailing function was working after all this. I'm finding that post-install, the wysiwyg editor is always broken due to a bug in Civi's URI processing. It seems to require a direct hack on core in order to fix it. If I'm right, that feature will never work on a Drupal8 install. If I'm wrong, I'm not following the directions correctly somehow.

Unfortunatly I haven't looked that far into it yet (paid work has gotten in the way)

I did go down a rabbit hole trying to get Drupal 8 to create a linked user for a CiviCRM contact. It doesn't seem that functionality is quite there in CiviCRM Entity yet. The closest I got was to poll some views via the rest api with a python script, and manipulating the user list according to the contact list.

I still want to use CiviCRM, and if I find some spare time, I'll look into the issues I have and contribute the changes back.

Using this method, will CiviCRM update with a simple composer outdated/composer update command? How up to date will the repository be kept? I'm particularly concerned when security updates are released.

Thanks as always for your work on the project!

This is using the CiviCRM repositories & releases directly for the actual CiviCRM code. What we're adding here is a Composer plugin (which should work for any CiviCRM version) and a Composer project template to help start a new site with Drupal 8 and the plugin. So, we don't have to update anything when a new CiviCRM release comes out - as soon as Packagist sees the new release you should be able to update to it with 'composer update'. :-)

Hello David,
I would like to point out in some Drupal 8 configuration as Thunder the main directory is "docroot" instead of the "web: directory.
When I get the update that run Bower I noticed that web assets goes to the web directory and not in the right one for Thunder :

[civicrm-composer-plugin] Running bower for CiviCRM...
> [civicrm-composer-plugin] Syncing CiviCRM web assets to /web/libraries/civicrm...

Then I made it working well (web -> docroot/) with updates creating an alias with command : ln -s docroot web

Ah, yes. My plan was that that would be configurable in the composer.json, but I hadn't gotten around to that yet. We could maybe even read the configuration for 'installer-paths' to figure out where the site usually puts it's libraries? I just opened an issue on GitLab for this:

https://gitlab.com/roundearth/civicrm-composer-plugin/issues/1

Thanks so much for writing all this up. I updated to latest Composer then followed your instruction (command line to install whole new package). Everything ran fine for a long time, then came this warning:
Package phpunit/phpunit-mock-objects is abandoned, you should avoid using it. No replacement was suggested
Then the whole thing aborted
Generating autoload files
- .csslintrc (https://cgit.drupalcode.org/drupal/plain/.csslintrc?h=8.6.12): Downloading (failed)

[Composer\Downloader\TransportException]
The "https://cgit.drupalcode.org/drupal/plain/.csslintrc?h=8.6.12" file could not be downloaded (HTTP/1.1 503 Backend unavailable, connection time
out)
Any advice>

Hi! I think both of those things are upstream Drupal problems.

The phpunit/phpunit-mock-objects thing should be harmless: support was dropped for that project and Drupal hasn't updated/replaced it yet - that should happen in a later Drupal version.

The problem that I think broke the composer install is the 503 error, which I suspect is a temporary issue with Drupal.org. Trying that link now works for me, so perhaps give it another try and it'll work for you too?

Thanks!

David, thanks so much for your reply. I tried again, twice, and at the second time of asking it worked! So I am in.
Thanks for your excellent write ups.

Thanks so much for your work. There is very little information available to suggest that CiviCRM works with Drupal 8, which I think is having a chilling effect on recommending CiviCRM. It certainly has raised questions in our organization about whether we should recommend CiviCRM to our clients.

So, I appreciate that you are helping to clarify the status of CiviCRM and Drupal 8.

Do you think we are any closer to being able to update the official documentation with respect to D8?
https://docs.civicrm.org/sysadmin/en/latest/requirements/

And I noticed that the README on this repo is downright scary - no substantial updates in 3 years - even though the repo has had a continuous stream of commits. I suspect that the integration is in fact at least beta quality at this point, and works with current Civi/D8 versions. There is no issue queue there, but I noticed you are a contributor, so I'm reaching out here in the hopes that you could update it or poke someone.

Oops, the repo I was referring to with the out-of-date README is:
https://github.com/civicrm/civicrm-drupal-8

Thanks!

Yeah, updating those docs would be great. I'm not totally sure how to update the official documentation (probably a Git repo somewhere we can make a PR to) but I just made a PR to update that README file:

https://github.com/civicrm/civicrm-drupal-8/pull/22

Hopefully, it (or some variation) will get merged soon. :-)

EDIT: Here's a PR for the official docs too: https://github.com/civicrm/civicrm-sysadmin-guide/pull/160

Wow, thanks for taking such quick action on this!

When I use the method of starting from the drupal-project composer template (https://github.com/drupal-composer/drupal-project):
and run the command,
composer require roundearth/civicrm-composer-plugin civicrm/civicrm-drupal-8,
I get the following Error:
[InvalidArgumentException]
Could not find a version of package roundearth/civicrm-composer-plugin matching your minimum-stability (stable). Require it with an explicit version constraint allowing its desired stability.

Where I'm wrong? What should i do to resolve this?
Thanks

Hi!

I think you need to change your minimum-stability to 'dev' because there isn't a stable release of roundearth/civicrm-composer-plugin:

https://getcomposer.org/doc/04-schema.md#minimum-stability

I hope that helps!

Been using D8 since last year, including CiviCRM since start of this year, and yeah it was hard to install!

With some CiviCRM updates I have to manually go into the civicrm-core directory and do a bower install. Looks like this civicrm-composer-plugin would take care of it for me. But with the build system we have (Jenkins), everything is supposed to be made an artifact, then deployed, so I think it doesn't quite work in that context?

Hi!

With some CiviCRM updates I have to manually go into the civicrm-core directory and do a bower install. Looks like this civicrm-composer-plugin would take care of it for me.

Yep! It'll run bower, install your extensions, sync web assets to the web root, etc.

But with the build system we have (Jenkins), everything is supposed to be made an artifact, then deployed, so I think it doesn't quite work in that context?

It could certainly work with Jenkins! If you're running 'composer install' as part of your Jenkins build, and the Jenkins "agent" that's doing the building has access to all the necessary tools (bower, etc) then it should be able to build and deploy the artifact just fine.

I hope that helps!

Great! I'll try it, thanks.

David, this has been such a helpful article. But there is one further issue which I cannot find any answer to on the web: if I add CiviCRM extensions via the CiviCRM dashboard (as seems to be recommended) how do I get them into the composer and GIT workflow? Any advice? Thanks so much!

If you want the extensions to be installed by composer, this plugin lets you put them in the 'extra' sections of your composer.json. For example:

{
    "extra": {
        "civicrm": {
            "extensions": {
                "org.civicrm.shoreditch": "https://github.com/civicrm/org.civicrm.shoreditch/archive/v0.1-alpha30.zip"
            }
        }
    }
}

This is kind of a workaround, because it'd be better if CiviCRM extensions were normal packages on Packagist, and had add their own "type" registered, similar to how Drupal has "drupal-module" (I guess it'd be "civicrm-extension"). But it works if you want to include an extension in your code base for a Git workflow, rather than have it download in the site files (which is what the dashboard does by default).

This bypasses the downloading of the extension that the dashboard uses, but you'll still have to go to the dashboard to enable the module.

Hello David, on a fresh install of D8+Civi, there several hick-up : : [ErrorException] unlink(): No such file or directory This give a broken Civi site.
Edit : This occurs either on a combo install or in 2 steps (on a drupal install).

This should be fixed now!

I tried updating Drupal 8.6 to the latest Drupal 8.7, and the composer update fails on the new version of CiviCRM (5.13.1). When I run a clean install as in this article (which I have done successfully before) it also now fails on the latest CiviCRM release. Could you please offer any advice on how to deal with this?

This could be the issue that the previous commenter was having that's fixed now. Can you try it again?

Add comment

o