We use Varying Vagrant Vagrants (VVV) as the development environment for as many projects as we can. It provides a great foundation not only for developing client projects but also for core development itself.
One great thing about VVV is that it installs all of the tools needed to run the core unit tests right out of the box. Our wp-dev-lib project has a PHPUnit plugin bootstrap that makes use of the core unit tests as installed by VVV, and we have a pre-commit
hook which can run the plugin PHPUnit tests inside the VM even when making a commit from outside (on the host machine). This ability to run PHPUnit tests inside the VM makes it easy for getting developers quickly set up to run PHPUnit tests, and it’s also something that is suggested for core’s PHPUnit Grunt task (#36190).
Nevertheless…
Tests running outside of the VM are a fast and happy place.
— Jeremy Felt (@jeremyfelt) March 10, 2016
Even though VVV lets you run core’s PHPUnit tests right out of the box, the tests can run more slowly in the box. If you have a pre-commit
hook that runs unit tests with each commit, your workflow can be really slowed down (especially since the tests should eventually by run by Travis CI after pushing anyway). To reduce the temptation to git commit --no-verify
to bypass PHPUnit from holding up your commit, you can also run PHPUnit tests out the box, that is to say, outside of the box. (Aside: wp-dev-lib also allows you to selectively skip PHPUnit when committing.)
Running PHPUnit on host connected to DB in guest
Assuming you already have PHP installed, first install PHPUnit on your host machine (e.g. via brew install phpunit
). Secondly you need to set the WP_TESTS_DIR
environment variable to point to where the PHPUnit tests are located on your host machine (inside the Vagrant synced folder). For example, on my machine I’ve modified my .bash_profile
to include:
export WP_TESTS_DIR=~/vvv/www/wordpress-develop/tests/phpunit/
With a PR I made to VVV, the wp-tests-config.php
knows whether or not PHPUnit is running inside Vagrant or not. If it detects it is running outside the Vagrant box, it then sets the DB_HOST
to be the IP address of the guest VM on the private network (192.168.50.4) in the default Vagrantfile
. This allows PHP on your host machine to connect to MySQL in the guest machine.
And with that, you should be able to just run phpunit
on your host machine’s terminal and the unit tests should run immediately without having to SSH into Vagrant first to run the tests. When the wp-dev-lib pre-commit
hook sees that phpunit
is installed and the WP_TESTS_DIR
environment variable is defined, it will run the PHPUnit tests from the host machine without stopping to SSH into the VM. This results in a much faster perceived performance improvement to running the tests. But what about actual overall performance?
$ time vagrant ssh -c 'cd /srv/www/wordpress-develop; phpunit' 2m55.007s $ time phpunit 3m1.222s
So it turns out that running PHPUnit inside the VM is actually faster, although not by a whole lot (~10 seconds). Nevertheless, this went contrary to my expectations 🙁 It seems that latency for PHP connecting to MySQL over the SSH connection is slightly greater than the extra cycles needed to run PHP in the VM.
Let’s try again, but this time run a subset of the test suite, for instance the customize
group:
$ time phpunit --group customize 0m11.172s $ time vagrant ssh -c 'cd /srv/www/wordpress-develop; phpunit --group customize' 0m15.727s
Notice here that running PHPUnit from the host is now significantly faster. 🙂 Since fewer tests are run, the SSH connection to the VM takes a more significant chunk of the time relatively, making it more worthwhile to run them from the host. And in reality, it is unlikely that you’ll be needing to run the full WP core test suite with each release. It is much more likely that you’ll only be wanting to run comparatively very few tests specifically for a plugin you are developing. For instance, running the unit tests for Customize Snapshots takes 0m4.871s from host, but 0m10.724s when connecting over SSH to run completely in the VM: running from the host is over twice as fast.
Running PHPUnit with DB entirely on host
What about abandoning the use of Vagrant altogether for running PHPUnit tests?
@danielbachhuber installed MySQL in OSX… which basically reverses the whole reason I started using Vagrant to begin with 🙂
— Jeremy Felt (@jeremyfelt) March 10, 2016
Let’s try running PHPUnit with the database completely local in the host to compare with running the tests entirely or partly in the VM.
- Assuming you can install MySQL via Homebrew:
brew install mysql ln -sfv /usr/local/opt/mysql/*.plist ~/Library/LaunchAgents launchctl load ~/Library/LaunchAgents/homebrew.mxcl.mysql.plist
This will install MySQL which has a
root
user with a blank password. - Next create the
wordpress_unit_tests
database:CREATE DATABASE IF NOT EXISTS wordpress_unit_tests CHARACTER SET utf8;
- Then amend your
.bash_profile
with the following:export WP_TESTS_DB_HOST=localhost export WP_TESTS_DB_NAME=wordpress_unit_tests export WP_TESTS_DB_USER=root export WP_TESTS_DB_PASSWORD=''
(Don’t forget to re-
source
your.bash_profile
, or just re-open your terminal app.) - Lastly, update your
wp-tests-config.php
as seen in PR #846 for VVV.
With these steps in place, you should now be able to run PHPUnit tests entirely on your host machine without any dependencies on the VM. Running the entire PHPUnit test suite with this configuration now results in:
$ time phpunit 2m28.080s
So basically the tests run 20% faster, as compared with 2m55.007s for running in Vagrant over SSH. And as for running the customize
group:
$ phpunit --group customize 0m10.895s
This is about 1 second (10%) faster, as compared with running PHPUnit from host with DB connection to VM. Running unit tests for the Customize Snapshots plugin is also about 10% faster (half a second).
So for normal sanely-sized PHPUnit test suites for plugins, the difference in speed is negligible between running them entirely on the host machine versus running PHP on the host machine with a connection to the database on the VM. In both cases the unit tests will run faster than if you have to connect over SSH to run them entirely inside the Vagrant box.
I hope this improves you workflow and encourages you to write more PHPUnit tests for your projects!
If you’re using homebrew to install PHPUnit as referenced above and are wondering why
brew install phpunit
returns no results, run this command first:brew tap josegonzalez/php