Issue #506: Add support for composer based site builds #694

oxyc · 2016-06-03T17:43:22Z

Issue #506. This is still WIP, just checking travis' opinion.

oxyc · 2016-06-03T17:45:51Z

default.config.yml

 drupal_major_version: 8
-drupal_core_path: "/var/www/drupalvm/drupal"
+drupal_core_path: "{{ drupal_composer_path }}/web"


Out of the box this should still work for makefiles. It's a bit weird having the otherwise empty drupal/ directory though. Not sure we even need to suggest changing it though. The recommended way is composer and if it still works for makefiles, that's good enough in my opinion.

This is a rather significant change though...

As everyone has their config.yml configured already, it's technically not a breaking change though.

I generally see projects use docroot rather than web for the actual drupal directory... could we do that if we use composer.json and install it in the proper directory?

oxyc · 2016-06-03T17:52:15Z

@geerlingguy if you like this approach and travis passes, tell me and I'll start working on updating the docs (I also need to manually test that everything works with a custom composer.json).

geerlingguy · 2016-06-03T17:56:15Z

default.config.yml

+# used instead of the 'drupal_composer_package'.
+build_composer: true
+drupal_composer_path: "/var/www/drupalvm/drupal"
+drupal_composer_package: "drupal-composer/drupal-project:8.x-dev"


I was thinking we could have an example.composer.json and build from that, just like using drupal.make.yml; this approach isn't necessarily wrong, but I feel like it would be more intuitive to have a composer.json and build from that rather than building the composer install using a CLI command.

You might be correct, drupal-project adds some complexity. However, with this approach we don't need to maintain a composer.json at all, and it also removes some steps from the quick start guide.

But if you prefer, I can add a basic example.composer.json and maybe add the create-project command installation as an option. build_composer_project: true or something.

joestewart · 2016-06-03T18:28:36Z

Some context to drupal-project and web/ vs docroot/ - drupal-composer/drupal-project#64

There is a little work being done there to make specifying the docroot easier.

oxyc · 2016-06-03T18:48:54Z

In case we want to support create-project based site builds I added it to a separate commit.

All 3 setups are being tested on travis. Note: docs are entirely out of sync.

oxyc · 2016-06-03T18:50:43Z

The CI fail is unrelated, I'm assuming it works but will test it locally as well.

@geerlingguy want me to keep support for composer create-project or should we go simple and just support composer.json?

oxyc · 2016-06-03T18:54:33Z

Erm just realized it's a bit chaotic with example.composer.json next to Drupal VM's own composer.json hehe... I'm currently using the same setup as with makefiles (copying it over to the VM with ansible so it works when provisioning production environments as well). I'm also looking for it in config_dir to be consistent with the makefile. This won't work with the default setup where config_dir is Drupal VM's own directory.

I guess we need to commit ~~the docroot/~~ a drupal/ directory and place it (and look for it) there.

oxyc · 2016-06-03T19:15:49Z

I don't really like placing the example.composer.json in a separate subdir like that, but don't see any other option.

@geerlingguy thoughts? Personally I prefer the first approach more to be honest. Just default to drupal-project (I'm biased as I use that project though), and let the file be generated with optionally required dependencies in drupal_composer_dependencies. On top of that add support for skipping the project creation, and instead specifying a custom composer.json, making the user aware of Drupal VM's own composer.json.

geerlingguy · 2016-06-03T19:22:44Z

@oxyc - Let's do the first and use drupal-project. We can pivot at some point if needed, and it makes more sense to do that and simplify overall setup, I think. People who are deep enough into the composer workflow should/will use Drupal VM as a dependency anyways, so no need to support only composer.json builds out of the box.

geerlingguy · 2016-06-03T19:23:20Z

Sorry for creating the bit of rework, but I'm coming around to what you originally proposed. Will need to poke and prod locally in a bit.

oxyc · 2016-06-03T20:08:11Z

provisioning/tasks/build-composer.yml

+  copy:
+    src: "{{ drupal_composer_json }}"
+    dest: "{{ drupal_composer_install_dir }}/composer.json"
+  when: drupal_composer_json is defined and not drupal_site.stat.exists


This isn't needed for the usual setups (BLT or projects where Drupal VM is in vendor), but I added it as it could be useful in some scenarios. Eg. provisioning production environments (same logic as for makefiles).

oxyc · 2016-06-03T20:12:21Z

If everything looks okay, I'll get started on adjusting the docs and running some more tests in a bit.

geerlingguy · 2016-06-03T20:16:34Z

Testing locally.

geerlingguy · 2016-06-03T20:21:47Z

@oxyc - CentOS 7 build failed with:

TASK [Check if site is already installed.] *************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 2] No such file or directory: '/var/www/drupalvm/drupal/web'

geerlingguy · 2016-06-03T20:22:17Z

We'll see how test run 777 goes.

geerlingguy · 2016-06-03T20:28:50Z

@oxyc is the Generate Drupal project with composer package. supposed to take forever? My CPU's been spinning for a while at 200%, for I think 10 minutes or so. Is there a way to get that process to speed up without killing my CPU, or is this just the way it is with drupal-project?

oxyc · 2016-06-03T20:30:49Z

provisioning/tasks/build-makefile.yml

-  file:
-    path: "{{ drupal_core_path }}"
-    state: directory
-    recurse: yes


Hmm, what was the reason behind this being recursive?

oxyc · 2016-06-03T20:32:18Z

@oxyc is the Generate Drupal project with composer package. supposed to take forever? My CPU's been spinning for a while at 200%, for I think 10 minutes or so. Is there a way to get that process to speed up without killing my CPU, or is this just the way it is with drupal-project?

Hmm not sure... I'll see if I can reach them on twitter.

geerlingguy · 2016-06-03T20:35:39Z

See also: drupal-composer/drupal-project#137 (comment) — but yeah, this is kind of a deal breaker imo, as it's been about 10 minutes (still killing my CPU currently), and I'm on a 1.7 Ghz i7 on VMware with an SSD, and a 100 Mbps internet connection.

I can't even begin to imagine how many hours it will take a poor soul stuck on an old PC with an Atom processor and a 2 Mbps internet connection :P

geerlingguy · 2016-06-03T20:37:17Z

Everything installed successfully, so there's that :)

oxyc · 2016-06-03T20:37:58Z

This run had all 3 installs:

With a bare minimum composer.json – 12min
With the old Makefile – 8min
With drupal-project – 16min

Yeah that's really bad. I'm on a 1.1ghz Macbook and it only takes a few minutes for me though. Haven't even noticed it's slower than the makefiles.

geerlingguy · 2016-06-03T20:38:51Z

Maybe we could still default to the makefile but support composer.json, then... I'm going to test again on a rebuild and see if it's any quicker.

geerlingguy · 2016-06-03T20:39:29Z

Some stats from my end:

drupal-project: 22 minutes on first run
drupal-project: 11 minutes on second run
make: 6 minutes on first run

geerlingguy · 2016-06-03T20:48:03Z

@oxyc could we use https://github.com/hirak/prestissimo somehow?

oxyc · 2016-06-03T20:59:24Z

Interesting! I'll give it a try once I get these silly tests to pass.

geerlingguy · 2016-06-03T21:05:03Z

Updated my stats—6 min vs 22 minutes is a definite deal-breaker :(

We could default to makefile since it's waaaay faster, and then allow people to use composer if they'd like. Unless we can figure out a way to get it at least less than 2x slower than makefiles. Maybe prestissimo will work.

geerlingguy · 2016-06-03T22:31:01Z

Grr... this thread has some great data in it though—we can do a follow-up issue to look at how to make things faster. Maybe there's a trick someone has up their sleeve to supercharge the install.

oxyc · 2016-06-03T22:32:09Z

We could look for composer.json in the Vagrantfile and run composer there instead :D That way the users cache would be re-used across all projects. Or run it on the host with ansible and then sync it? Either way, that's another task for sure. A lot of fiddling for different setup environments.

thom8 · 2016-06-04T06:22:46Z

@oxyc you could try syncing the host ~/.composer/cache to the VM but no help when you don't have an existing cache.

vagrant_synced_folders:
  - local_path: ~/.composer/cache
    destination: /root/.composer/cache
    type: nfs
    create: true

I assume the local cache is why it's not such a major issue for most ppl not to mention pulling projects with such a huge history like drupal..

Shallow clones would be a big help! -- composer/composer#4961

joestewart · 2016-06-04T11:52:04Z

For comparison, try vagrant-cachier. It supports composer - http://fgrehm.viewdocs.io/vagrant-cachier/buckets/composer/

oxyc · 2016-06-04T14:05:32Z

@oxyc you could try syncing the host ~/.composer/cache to the VM but no help when you don't have an existing cache.

@thom8 yeah we should probably add a section on how to improve performance. That would be one way, and vagrant-cachier another as @joestewart suggested. Maybe that can be a separate docs task though.

A lot of back and forth here. Now I'm back to drush makefile being the default, and having build_composer and build_composer_project as separate tasks. build_composer_project is not tested in travis but I'm testing it locally now.

oxyc · 2016-06-04T14:26:10Z

Tested locally:

default makefile approach
drupal.composer.json
drupal-project
drupal_composer_path: false + drupal_composer_install_dir: "/var/www/drupalvm"

oxyc · 2016-06-04T15:02:32Z

default.config.yml

 drupal_major_version: 8
-drupal_core_path: "/var/www/drupalvm/drupal"
+drupal_core_path: "{{ drupal_composer_install_dir }}/docroot"


I can also keep the old behaviour as makefiles are still the default. Just need to mention that this needs to be changed in the composer docs.

Yeah, let's stick to the old for now, since we're not defaulting to composer.

oxyc · 2016-06-04T15:44:14Z

vagrant-cachier doesn't work as it requires PHP to exist before provisioning the VM.

I'll see if it works using the generic cache bucket.

oxyc · 2016-06-04T16:25:47Z

Not sure why, but vagrant-cachier isn't caching anything for me... Seems like it's not even invoked.

For reference I was trying a Vagrantfile.local with:

config.cache.auto_detect = false
config.cache.enable :generic, { :cache_dir => '/home/vagrant/.composer/cache' }

as per http://fgrehm.viewdocs.io/vagrant-cachier/buckets/generic/

Even with auto_detect on, nothing gets cached though.

Not sure if this is just happening on my laptop. I have some WIP docs but as I can't actually get the shared cache to work I wont add it. https://gist.github.com/oxyc/7c768feaa3c16a9f52c1196689f3a0a7

(edit: I missed that the scope setting was required, it works now so I'm running a benchmark).

geerlingguy · 2016-06-04T20:21:34Z

I think if we get in the other two changes mentioned above, I'm happy to merge this and get the 3.1.0 release tagged. Hopefully we can find a way to get performance to the point where composer is the default by .2 or .3!

geerlingguy · 2016-06-05T02:04:00Z

default.config.yml

 drupal_major_version: 8
-drupal_core_path: "/var/www/drupalvm/drupal"
+drupal_core_path: "{{ drupal_composer_install_dir }}"


Let's leave this as the default /var/www/drupalvm/drupal (the composer in the var might confuse people who aren't using composer, which at this point is likely the majority of users).

geerlingguy · 2016-06-05T02:07:27Z

I'm ready to merge once that conflict is resolved. A long road to this point, but it's well worth it. This will be an awesome improvement!

oxyc · 2016-06-05T02:10:17Z

Whops I somehow managed to get in the commit from my other PR. Getting late....

Want me to squash these? A lot of back and forth going on...

geerlingguy · 2016-06-05T02:14:11Z

If it's quick, you can... otherwise I can live with the history. It's not all bad to have some of the pivots as part of the history, though it makes my release note creation a little more fun :D

geerlingguy · 2016-06-05T02:28:42Z

Looks good. Once test is green, merge button will be pressed 👍

geerlingguy added this to the 3.1.0 milestone Jun 3, 2016

oxyc reviewed Jun 3, 2016
View reviewed changes

geerlingguy reviewed Jun 3, 2016
View reviewed changes

oxyc reviewed Jun 3, 2016
View reviewed changes

geerlingguy mentioned this pull request Jun 3, 2016

Updating docs for Drupal VM. acquia/blt#104

Merged

oxyc reviewed Jun 3, 2016
View reviewed changes

oxyc force-pushed the drupal-project branch from eec4e70 to 0a13a2c Compare June 3, 2016 20:56

oxyc force-pushed the drupal-project branch 2 times, most recently from a142bd8 to 838524f Compare June 4, 2016 14:20

oxyc changed the title ~~[WIP] Issue #506: Default to composer based site build~~ Issue #506: Default to composer based site build Jun 4, 2016

oxyc changed the title ~~Issue #506: Default to composer based site build~~ Issue #506: Add support for composer based site builds Jun 4, 2016

oxyc force-pushed the drupal-project branch from 838524f to 27f7053 Compare June 4, 2016 14:57

oxyc reviewed Jun 4, 2016
View reviewed changes

oxyc force-pushed the drupal-project branch from 6174c63 to 95bed67 Compare June 4, 2016 20:50

geerlingguy reviewed Jun 5, 2016
View reviewed changes

oxyc force-pushed the drupal-project branch from eca802e to 03148a5 Compare June 5, 2016 02:09

oxyc added 2 commits June 4, 2016 22:25

Add support for composer-based drupal site builds

dedb520

Add prestissimo as a default global composer package

c5c8c8a

oxyc force-pushed the drupal-project branch from 03148a5 to c5c8c8a Compare June 5, 2016 02:25

geerlingguy merged commit dcc66c9 into geerlingguy:master Jun 5, 2016

geerlingguy mentioned this pull request Jun 5, 2016

Default to Composer-based site building instead of Drush Make #699

Closed

oxyc mentioned this pull request Jun 5, 2016

Use composer project based site build as the default #701

Merged

oxyc deleted the drupal-project branch April 24, 2017 15:59

Issue #506: Add support for composer based site builds #694

Issue #506: Add support for composer based site builds #694

Conversation

oxyc commented Jun 3, 2016

oxyc Jun 3, 2016 • edited Loading

Choose a reason for hiding this comment

oxyc Jun 3, 2016

Choose a reason for hiding this comment

geerlingguy Jun 3, 2016

Choose a reason for hiding this comment

oxyc commented Jun 3, 2016 • edited Loading

geerlingguy Jun 3, 2016

Choose a reason for hiding this comment

oxyc Jun 3, 2016 • edited Loading

Choose a reason for hiding this comment

joestewart commented Jun 3, 2016

oxyc commented Jun 3, 2016

oxyc commented Jun 3, 2016

oxyc commented Jun 3, 2016 • edited Loading

oxyc commented Jun 3, 2016 • edited Loading

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

oxyc Jun 3, 2016 • edited Loading

Choose a reason for hiding this comment

oxyc commented Jun 3, 2016 • edited Loading

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

oxyc Jun 3, 2016

Choose a reason for hiding this comment

oxyc commented Jun 3, 2016

geerlingguy commented Jun 3, 2016 • edited Loading

geerlingguy commented Jun 3, 2016

oxyc commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016 • edited Loading

geerlingguy commented Jun 3, 2016

oxyc commented Jun 3, 2016 • edited Loading

geerlingguy commented Jun 3, 2016

geerlingguy commented Jun 3, 2016

oxyc commented Jun 3, 2016 • edited Loading

thom8 commented Jun 4, 2016

joestewart commented Jun 4, 2016

oxyc commented Jun 4, 2016 • edited Loading

oxyc commented Jun 4, 2016 • edited Loading

oxyc Jun 4, 2016

Choose a reason for hiding this comment

geerlingguy Jun 4, 2016

Choose a reason for hiding this comment

oxyc commented Jun 4, 2016 • edited Loading

oxyc commented Jun 4, 2016 • edited Loading

geerlingguy commented Jun 4, 2016

geerlingguy Jun 5, 2016

Choose a reason for hiding this comment

geerlingguy commented Jun 5, 2016

oxyc commented Jun 5, 2016 • edited Loading

geerlingguy commented Jun 5, 2016

geerlingguy commented Jun 5, 2016

oxyc Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

oxyc Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

oxyc Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

geerlingguy commented Jun 3, 2016 •

edited

Loading

geerlingguy commented Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

oxyc commented Jun 3, 2016 •

edited

Loading

oxyc commented Jun 4, 2016 •

edited

Loading

oxyc commented Jun 4, 2016 •

edited

Loading

oxyc commented Jun 4, 2016 •

edited

Loading

oxyc commented Jun 4, 2016 •

edited

Loading

oxyc commented Jun 5, 2016 •

edited

Loading