Drupal 8 Development with Phing build script and Jenkins

Submitted by nigel on Tuesday 3rd October 2017

When you are developing a Drupal 8 website the likelihood is you are going to want to perform the same tasks over and over. These tasks would include building up from scratch, exporting and importing configuration, adding contributed modules and updating the database schema, clearing caches etc. 

It therefore makes sense to automate the process to save time and avoid making mistakes by triggering tasks in the wrong order. To accomplish this I use the PHP build tool Phing which is based on Apache's Ant and similarly uses XML to define the build process.

The starting point is to create a directory structure for the project. I go for this:

nigel@badzilla-d8 /var/www/html/meedjum (master=) $ tree -L 1
.
├── assets
├── build
├── config
├── docroot
└── tools
 
5 directories, 0 files

The assets directory is used for holding all original artwork for the logos of the site etc. For commercial projects this wouldn't be necessary since they would be held elsewhere, but for a hobbyist blog, they are a good safe place to store stuff you may need again in the future. 

The build directory is for the Phing build file and any supporting config files. 

The config directory is my Drupal site's config synchronised directory. Note I have placed it outside the site's docroot.  

The docroot directory is for the website itself. 

The tools directory is for sundry bits and pieces such as the Drush class required to run drush commands within the Phing build script. 

Not seen in the directory listing is the .git directory - this is the base of our git repository. 

Time to create our Phing build script. Initially we need our script to build out our composer template. So that will be our first target. Here's the code, and the supporting properties file which has the variables for my own build. Both this are placed in the build directory (see above)

<?xml version="1.0"  encoding="UTF-8" ?>
 
<project name="badzilla" basedir="../docroot" default="site-rebuild">
    <property file="../build/build.properties" />
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: create-project             		   -->
    <!-- ============================================  -->
    <target name="create-project" description="Composer create-project">
    	<composer command="create-project" composer="${dir.composer}/${exe.composer}">
    		<arg value="${template.composer}" />  	
    		<arg value="${dir.make.drupal}" />  
    		<arg value="${build.version.drupal}" /> 
    		<arg value="--no-dev" />
            <arg value="--no-interaction" />
    		<arg value="--working-dir" />
       		<arg value="${dir.make.drupal}" />   		 		    			
    	</composer>
    </target>
</project>
# Environment
env=sandbox
path.drush=/usr/local/bin/drush
dir.root=../
dir.composer=/usr/local/bin
exe.composer=composer
 
# drupal
dir.make.drupal=../docroot
dir.docroot.drupal=docroot
dir.tools.drupal=../tools
dir.drupal.config=../config/
template.composer=drupal/drupal
build.version.drupal=8.3.*
 
# Site
install.sitename=your_sitename
install.accountmail=admin@yourdomain.com
install.locale=en
install.sitemail=noreply@yourdomain.com
install.profile=standard
install.subdir=default
 
# Database
credentials.dbuser=your_dbuser
credentials.dbpass=your_dbpass
credentials.dbname=your_dbname
credentials.host=localhost
 
# Site
install.accountname=your_username
install.accountpass=your_password
If you now navigate to the build directory you can run the composer build with:
nigel@badzilla-d8 /var/www/html/meedjum (master=) $ cd build
nigel@badzilla-d8 /var/www/html/meedjum/build (master=) $ phing create-project

You should now have your Drupal instance built with the drupal/drupal composer template. A good idea would be to add a composer update target to our build script also. 

Add the following target underneath the previous create-project target. 

    <!-- ============================================  -->
    <!-- Target: composer-update                       -->
    <!-- ============================================  -->
    <target name="composer-update" description="Composer update">
        <composer command="update" composer="${dir.composer}/${exe.composer}"></composer>
    </target>

With the composer-update target, this can be run regularly to determine whether any of the dependencies have been upgrading and need downloading into the project. 

The next activity is to actually install a Drupal 8 site, and this is achieved with the drush site install command. However there are dependencies here that need to be satisfied before the install. Phing/Drush bridge needs to be setup before we run any Drush command, and we need to set file permissions on settings.php. So we create the site-install target and further dependency targets for drush setup and file permissions. Add the following:

    <!-- ============================================  -->
    <!-- Target: site-install                          -->
    <!-- ============================================  -->
    <target name="site-install" 
            depends="setup-phing-drush, chmod-default-dir"
            description="Drupal profile build">
 
        <echo msg="install profile is ${dinstall.profile}" />  
 
        <drush root="${dir.make.drupal}" command="site-install" assume="yes">
            <param>${install.profile}</param>
            <option name="db-url">mysql://${credentials.dbuser}:${credentials.dbpass}@localhost/${credentials.dbname}</option>
            <option name="site-name">${install.sitename}</option>
            <option name="account-name">${install.accountname}</option>
            <option name="account-pass">${install.accountpass}</option>
            <option name="locale">${install.locale}</option>
            <option name="account-mail">${install.accountmail}</option>
            <option name="site-mail">${install.sitemail}</option>   
        </drush>
 
    </target> 
    <!-- ============================================  -->
    <!-- Target: setup-phing-drush                     -->
    <!-- ============================================  -->
    <target name="setup-phing-drush">
 
        <taskdef name="drush" classname="DrushTask"
            classpath="${dir.tools.drupal}" />
 
    </target>   
    <!-- ============================================  -->
    <!-- Target: chmod-default-dir                     -->
    <!-- settings.php doesn't exist first time thro	   --> 
    <!-- ============================================  -->
    <target name="chmod-default-dir">
 
        <chmod file="sites/default" mode="0777" failonerror="true" />
        <if>
        	<available file="settings.php" type="file" />
        	<then>
        		<chmod file="settings.php" mode="0770" failonerror="true" />
        	</then>
        </if>
    </target> 

Most of the leg work in any Drupal 8 development is continually rebuilding the site, so the next target does precisely that. There is additional complexity here because of Drupal's configuration management. It is not possible to rebuild an existing site without the UUIDs changing, and when that happens, the build will choke. This is explained more thoroughly on this thread. Therefore any rebuild script needs the kludgey workaround that you will see in the site-rebuild target below. Also note the first listing above contained the line 

<project name="badzilla" basedir="../docroot" default="site-rebuild">

This is saying the the default target is set to rebuild - so when Phing is run from the command line without any parameters, this target will be selected. With Drupal projects there is always a cutover when it comes to creating content - at that point total rebuilds are no longer made, and incremental changes through the Admin UI with config export is used instead. At that point, remove the rebuild target from being default to stop accidentally rebuilding and destroying your data. 

    <!-- ============================================  -->
    <!-- Target: site-rebuild                          -->
    <!-- Incudes hacks to uninstall shortcuts and      -->
    <!-- and reenable because shortcuts creates uuid   -->
    <!-- based shortcut                                -->
    <!-- https://www.drupal.org/node/2583113           -->
    <!-- ============================================  -->
    <target name="site-rebuild" depends="setup-phing-drush" description="Get the site uuid and update config">
 
        <phingcall target="site-install" />
 
        <phingcall target="composer-update" />        
 
        <exec command="cat ${dir.drupal.config}/system.site.yml | grep uuid | tail -c +7 | head -c 36 | ${path.drush} config-set -y system.site uuid -" dir="${dir.make.drupal}" />
 
        <drush root="${dir.make.drupal}" command="php-eval">        
            <param>"\Drupal::entityManager()->getStorage('shortcut_set')->load('default')->delete();"</param>
        </drush>    
 
        <drush root="${dir.make.drupal}" command="pm-uninstall" assume="yes">           
            <param>shortcut</param>
        </drush>    
 
        <drush root="${dir.make.drupal}" command="pm-enable" assume="yes">          
            <param>shortcut</param>
        </drush>            
 
        <drush root="${dir.make.drupal}" command="config-import" assume="yes">          
            <option name="source">${dir.drupal.config}</option>
        </drush>
 
        <drush root="${dir.make.drupal}" command="php-eval">        
            <param>"node_access_rebuild()"</param>
        </drush>
 
        <drush root="${dir.make.drupal}" command="updatedb" assume="yes"></drush>
 
    </target>    

The final target happens to be the one I run the most frequently - drush export configuration. The target is really just a wrapper for a drush command.

    <!-- ============================================  -->
    <!-- Target: config-export                         -->
    <!-- ============================================  -->
    <target name="config-export" 
        description="Wrapper for drush config-export"
        depends="setup-phing-drush">
 
        <drush root="${dir.make.drupal}" command="config-export" assume="yes">          
            <option name="destination">${dir.drupal.config}</option>
        </drush>    
 
    </target>
The Complete Build File

Lets stitch all the targets together so they are in one place so you can easily copy / paste. Don't forget the properties file which is at the top of the page - you will need this too! 

<?xml version="1.0"  encoding="UTF-8" ?>
 
<project name="badzilla" basedir="../docroot" default="site-rebuild">
    <property file="../build/build.properties" />
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: create-project             		   -->
    <!-- ============================================  -->
    <target name="create-project" description="Composer create-project">
    	<composer command="create-project" composer="${dir.composer}/${exe.composer}">
    		<arg value="${template.composer}" />  	
    		<arg value="${dir.make.drupal}" />  
    		<arg value="${build.version.drupal}" /> 
    		<arg value="--no-dev" />
            <arg value="--no-interaction" />
    		<arg value="--working-dir" />
       		<arg value="${dir.make.drupal}" />   		 		    			
    	</composer>
    </target>
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: composer-update                       -->
    <!-- ============================================  -->
    <target name="composer-update" description="Composer update">
        <composer command="update" composer="${dir.composer}/${exe.composer}"></composer>
    </target>
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: site-install                          -->
    <!-- ============================================  -->
    <target name="site-install" 
            depends="setup-phing-drush, chmod-default-dir"
            description="Drupal profile build">
 
        <echo msg="install profile is ${dinstall.profile}" />  
 
        <drush root="${dir.make.drupal}" command="site-install" assume="yes">
            <param>${install.profile}</param>
            <option name="db-url">mysql://${credentials.dbuser}:${credentials.dbpass}@localhost/${credentials.dbname}</option>
            <option name="site-name">${install.sitename}</option>
            <option name="account-name">${install.accountname}</option>
            <option name="account-pass">${install.accountpass}</option>
            <option name="locale">${install.locale}</option>
            <option name="account-mail">${install.accountmail}</option>
            <option name="site-mail">${install.sitemail}</option>   
        </drush>
 
    </target> 
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: config-export                         -->
    <!-- ============================================  -->
    <target name="config-export" 
        description="Wrapper for drush config-export"
        depends="setup-phing-drush">
 
        <drush root="${dir.make.drupal}" command="config-export" assume="yes">          
            <option name="destination">${dir.drupal.config}</option>
        </drush>    
 
    </target>
 
 
 
    <!-- ============================================  -->
    <!-- Target: site-rebuild                          -->
    <!-- Incudes hacks to uninstall shortcuts and      -->
    <!-- and reenable because shortcuts creates uuid   -->
    <!-- based shortcut                                -->
    <!-- https://www.drupal.org/node/2583113           -->
    <!-- ============================================  -->
    <target name="site-rebuild" depends="setup-phing-drush" description="Get the site uuid and update config">
 
        <phingcall target="site-install" />
 
        <phingcall target="composer-update" />        
 
        <exec command="cat ${dir.drupal.config}/system.site.yml | grep uuid | tail -c +7 | head -c 36 | ${path.drush} config-set -y system.site uuid -" dir="${dir.make.drupal}" />
 
        <drush root="${dir.make.drupal}" command="php-eval">        
            <param>"\Drupal::entityManager()->getStorage('shortcut_set')->load('default')->delete();"</param>
        </drush>    
 
        <drush root="${dir.make.drupal}" command="pm-uninstall" assume="yes">           
            <param>shortcut</param>
        </drush>    
 
        <drush root="${dir.make.drupal}" command="pm-enable" assume="yes">          
            <param>shortcut</param>
        </drush>            
 
        <drush root="${dir.make.drupal}" command="config-import" assume="yes">          
            <option name="source">${dir.drupal.config}</option>
        </drush>
 
        <drush root="${dir.make.drupal}" command="php-eval">        
            <param>"node_access_rebuild()"</param>
        </drush>
 
        <drush root="${dir.make.drupal}" command="updatedb" assume="yes"></drush>
 
    </target>      
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: setup-phing-drush                     -->
    <!-- ============================================  -->
    <target name="setup-phing-drush">
 
        <taskdef name="drush" classname="DrushTask"
            classpath="${dir.tools.drupal}" />
 
    </target>   
 
 
 
 
    <!-- ============================================  -->
    <!-- Target: chmod-default-dir                     -->
    <!-- settings.php doesn't exist first time thro	   --> 
    <!-- ============================================  -->
    <target name="chmod-default-dir">
 
        <chmod file="sites/default" mode="0777" failonerror="true" />
        <if>
        	<available file="settings.php" type="file" />
        	<then>
        		<chmod file="settings.php" mode="0770" failonerror="true" />
        	</then>
        </if>
    </target> 
 
 
</project>
Jenkins
Plugin Enabled

Phing build targets are a great candidate for automation with Jenkins since Jenkins supports Phing providing the plugin has been enabled. You can check this by navigating to pluginManager/installed on your Jenkins box (see above). 

Jenkins Drupal jobs

I have created a view for all my Jenkins jobs for this particular project - see above. You can see I have disabled the rebuild job because my build process has now switched over from continual rebuild to adding content. 

Jenkins Export Configuration
Export Configuraiton

Ignore the multiple Save/Apply on the screenshot - this was caused by the fact it floats at the bottom of the screen and the screenshot app needed to scroll. The crux of the job is at the bottom - Invoke Phing Targets. I have specified config-export. 

Jenkins Job Output
Jenkins Output

Ok here we see the output of a typical configuration export. The beauty of using Jenkins building your targets is you don't need to remember the target names for command line invocation, and there is less possibility of running the wrong job. 

Stay tuned for more tutorials on Phing / Jenkins!