You are currently browsing the archives for the Continuous Integration category


Practices of an Agile Developer

Of the many software engineering books I have read over the years, Practices of an Agile Developer in particular continues to be one book I find myself turning to time and time again for inspiration.

Written by two of my favorite technical authors, Andy Hunt and Venkat Subramaniam, and published as part of the Pragmatic Bookshelf, Practices of an Agile Developer provides invaluable, practical and highly inspirational solutions to the most common challenges we as software engineers face project after project.

What makes Practices of an Agile Developer something truly special is the simplicity and easy to digest format in which it is written; readers can jump in at any chapter, or practically any page for that matter, and easily learn something new and useful in a matter of minutes.

While covering many of the most common subjects on software development, as well as many particularly unique subjects, it is the manner in which the subjects are presented that makes the book itself quite unique. The chapters are formatted such that each provides an “Angel vs. Devil on your shoulders” perspective of each topic. This is quite useful as one can briefly reference any topic to take away something useful by simply reading the chapters title and the “Angel vs. Devil” advice, and from that come to a quick understanding of the solution. Moreover, each chapter also provides tips on “How it Feels” when following one of the prescribed approaches. The “How it feels” approach is very powerful in that it instantly draws readers in for more detailed explanations. Complimentary to this is the “Keeping your balance” suggestions which provide useful insights to many of the challenges one might face when trying to apply the learnings of a particular subject. “Keeping your Balance” tips answer questions which would otherwise be left to the reader to figure out.

I first read Practices of an Agile Developer almost 4 years ago, and to this day I regularly find myself returning to it time and time again for inspiration. A seminal text by all means, I highly recommend it as a must read for Software Developers of all levels and disciplines.

Flex Mojos 3.2.0 Released

Sonatype recently released the latest version of Flex Mojos, which is now at version 3.2.0.

This latest update is a big step forward for Flex / AIR Developers managing their project builds and dependencies with Maven 2 as the updates are focused around unit testing support improvements; including support for headless mode on Linux based CI servers and, more importantly, a fix for running automated unit tests in multi-module builds; which was a big head scratcher for me about a month ago!

Below is a list of what I feel are the most significant updates in 3.2.0:

  • Added support for SWF optimization
  • Multi-module builds now run tests correctly across projects
  • Changes to the way flex-mojos launches flash player when running test harness
  • Long-running flexunit tests no longer cause the build to fail.
  • Fix to NullPointerException during flex-mojos:test-run goal
  • You can view the complete list of release notes here.

    Refactoring Ant Builds with Macrodefs

    Within the past few years the proliferation of Agile Best Practices has pushed the importance of refactoring front and center in the world of Object Oriented Software Design, yet for some odd reason build scripts seem to have been overlooked in this regard by many. Perhaps this is due to the risk and complexity involved in such an effort as well as the lack of a tools by which refactoring build scripts can safely be accomplished.

    For instance, whereas refactoring in typical OO languages relies heavily on Unit Tests for ensuring refactorings do not break existing code along the way, build scripts do not have such safety nets as Unit Tests. Ant is statically typed however it doesn’t provide compile time type checking, additionally build scripts are defined declaratively via XML mark-up however they can not be validated as there are not fixed DTD attributes to validate them against. Perhaps most importantly is that there are not many resources to turn to for guidance when it comes to refactoring Build Scripts. For example, most of what I have learned about the subject comes from Julian Simpson’s work in the ThoughtWorks Anthology, which I highly suggest reading for a much more exhaustive, yet comprehensive and succinct essay on the subject. In any case, based on the above factors I am quite certain that all of these points plays a role in Ant Scripts somehow being overlooked with regard to refactoring.

    So where do you begin?
    That’s a really good question, one which I was forced to ask myself awhile back while being tasked with the daunting challenge of streamlining a very complex Build / CI process. At the time, I was responsible for modifying a Build for a large enterprise class Flex application which required build time transformations of localized content with varying modules being built for n-locales depending on context specific business rules, all of which needed to be built and deployed to multiple environments via a pre-existing CI Process. Further complicating things was that the builds were wrapped by nested DOS batch files. In addition, the existing builds had dependencies on far more complex underlying build Scripts. To make matters worse, up until that point in time no one, including myself, truly knew the build structure and all of it’s dependencies, it was very much a black box. So considering the fact that I needed to modify the build and would be responsible for maintaining the builds moving forward, as well as streamlining the existing build scripts so as to allow them to scale in order to support additional applications to seamlessly become part of the build, to say the least, I was eager to learn the Build Scripts inside out if I was to refactor and maintain them.

    The moral to the story I just bored you with above is that if you have ever had to maintain a build before then this story probably sounds pretty familiar: you have a Build Script which is a black box that no one wants to deal with; it works and that’s all that matters – until it needs to change of course. So again, where does one begin when refactoring a Build Script? Well lets think in terms of typical OO refactoring.

    Remove duplication
    Perhaps one of the most obvious and easiest places to begin consideration for refactoring candidates in an Object Oriented Design is to remove duplication; that is to isolate and thin out common functionality so as to remove redundancy and duplication. Most Ant Scripts are littered with such duplication, and as such should be viewed in the same manner as one would when refactoring Object Oriented applications. In fact, the goal of refactoring is very much the same regardless of the paradigm – beit a declaratively language such as Ant or an Object Oriented language such as ActionScript – provide more efficient, maintainable and easier to work with code.

    I tend to think of Build Script design – yes, it is design – much the same as any other OO design. So just as one would strive to eliminate code duplication in an Object Oriented Design, the same should apply to the design of a Build Script. For example, consider the following build target which packages a series of distributions:

       
    <target name="package" depends="package.src,
                                    package.tests,
                                    package.bin,
                                    package.docs,
                                    package.dist"

                                    />

       
    <target name="package.src" >
        <zip basedir="${src.dir}"
             destfile="${dist.dir}/${dist.src.name}.zip"  
             />

    </target>

    <target name="package.tests" >
        <zip basedir="${src.tests.dir}"
             destfile="${dist.dir}/${dist.src.name}.zip"  
             />

    </target>

    <target name="package.bin" >
        <zip basedir="${bin.dir}"
             destfile="${dist.dir}/${dist.bin.name}.zip"  
             />

    </target>

    <target name="package.docs" >
        <zip basedir="${docs.dir}"
             destfile="${dist.dir}/${docs.name}.zip"
             />

    </target>
       
    <target name="package.dist" >
        <zip basedir="${dist.dir}"
             destfile="${dist.dir}/${package.name}.zip"
             />

    </target>
     

    This kind of Build Script is common, however if you were to think of this in terms of OO Design, whereas each target is analogous to a method, you would quickly realize the code is very redundant. Moreover, the functionality provided by these targets: the packaging of distributions, is a very common task, so just as in an OO design this functionality should be extracted into a reusable library. In Ant 1.6+ we can achieve the same kind of code reuse by extracting these common, redundant targets using Macrodefs.

    Use Macrodefs
    In short, a Macrodef, which is short for “macro definition”, is basically an extracted piece of reusable functionality in an Ant that can be used across Build Scripts for performing common, or specific tasks. Macrodefs can be thought of as a reusable API for Ant. You include macrodefs in your build scripts by importing the macrodef source file. This is analogous to how one would import a class.

    So consider the redundant targets outlined above. Using macrodefs we can extract these common tasks, refactoring them into a single macrodef, import the file which contains the macrodef into our build script and then call the macrodef by wrapping it in a task.

    To extract the target to a Macrodef we would first begin by creating a new XML document named after the functionality of the target, in this case we could call it “dist.xml”. This document would contain a project root node just as any other Ant Script would. We would then define a macrodef node and specify an identifier via the name attribute; this is how we can reference the macrodef once imported to our build script.

    <?xml version="1.0"?>
    <project name="package">
        <macrodef name="package.dist" >
        </macrodef>
    </project>

    Once we have defined the macrodef we can add dynamic properties to its definition. This could be thought of as begin analogous to arguments of a method signiture. By specifying these arguments we can then assign their values whenever we invoke the macrodef. Default values can also be added if needed.

    <?xml version="1.0"?>
    <project name="package">
        <macrodef name="package.dist" >
            <attribute name="base.dir"  />
            <attribute name="dist.dir"  />
            <attribute name="dist.file" />
        </macrodef>
    </project>

    Finally, we specify the behavior of the macrodef via the sequential node, This is where the functional markup is defined. Note that we reference the properties internally using the @{property} notation, just as you would normally however the token is prefixed with an @ sign rather than a $ sign.

    <?xml version="1.0"?>
    <project name="package">
        <macrodef name="package.dist" >
            <attribute name="base.dir"  />
            <attribute name="dist.dir"  />
            <attribute name="dist.file" />
            <sequential>
                <zip basedir="@{base.dir}"
                      destfile="@{dist.dir}/@{dist.file}.zip"
                      />

            </sequential>
        </macrodef>
    </project>

    We now have a parametrized, reusable piece of functionality which we can use across Ant Builds, and as such, simplifying the build while promoting code reuse.

    To use the macrodef in another Ant Build we need only import it and create a target which wraps the macrodef. So we could refactor the distribution targets from the original Build file example to the following:

    <import file = "package.xml"  />

    <target name="package" >
        <package.dist base.dir="${src.dir}"  
                      dist.dir="${dist.dir}"
                      dist.file="${dist.src.name}"
                      />

        <package.dist base.dir="${tests.dir}"
                      dist.dir="${dist.dir}"
                      dist.file="${dist.tests.name}"
                      />

        <package.dist base.dir="${bin.dir}"  
                      dist.dir="${dist.dir}"
                      dist.file="${dist.bin.name}"
                      />

        <package.dist base.dir="${docs.dir}"  
                      dist.dir="${dist.dir}"
                      dist.file="${dist.doc.name}"    
                      />

        <package.dist base.dir="${dist.dir}"  
                      dist.dir="${dist.dir}"
                      dist.file="${dist.package.name}"
                      />

    </target>

    And that’s the basics of using macrodefs to refactor an Ant Build. There is a lot more which can be accomplished with macrodefs in regards to designing and refactoring Ant Builds, specifically antlib, and I encourage you to give it a try as I am sure you will be happy with the results.

    Continuous Integration with Hudson

    Continuous Integration is a fundamental Agile Development process in which members of a team integrate changes on a regular basis, ideally multiple times per day, which in turn results in multiple integrations per day. The integration process itself is facilitated by an automated integration build which is triggered upon a specific interval to check for new commits to the central repository, or mainline. This is necessary for detecting changes which could potentially break the build as quickly as possible, as it is typically easier to fix these errors sooner rather than later, thus resulting in significantly less integration issues, especially when working on large, collaborative projects where there are multiple members of a team developing against the same codebase.

    Continuous Integration does not necessarily require any specific tooling, however it is very common to incorporate a build management tool in order to automate the builds. The most common tool of choice by build managers to facilitate automated builds is Cruise Control. I have been using Cruise Control for years to automate Flex builds as it contains everything one typically would need for automating an integration, staging and production build process. Setting up Cruise Control and modifying the configuration to add new projects is a straight forward process, however I always felt it to be a bit tedious.

    Considering Cruise Control is an industry standard as well as the fact that it provides pretty much everything I have ever needed for automating enterprise build processes, I never had a compelling reason to investigate any of the other tools out there. Then last week I received an email from our engineering groups build manager stating that there is a different build management tool being considering called Hudson. Initially I questioned why Hudson was being considered over Cruise Control as CC is, for the most part, an industry standard, so I decided to do a little investigating to see for myself, first by reading the documentation then by running the simple test drive available at the Hudson site which you can use to get an idea of how the tool works.

    Downloading Hudson is straight forward and best of all installation is a breeze; simply deploy the web archive (hudson.war) to an existing servlet container (I am using Tomcat 6.0.14) and your ready to go.

    Once deployed you can go to the Hudson console, typically located at: http:<host:port>/hudson/ (e.g. http://somedomain:8400/hudson/). The console has all of the features you might expect (as well as some other useful tools). From the console you can easily configure Hudson, create new projects, monitor build progress, view build logs, kick off a build, schedule builds and so forth. Setting up a project is one of the things I really like about Hudson – as opposed to configuring a project in Cruise Control – as Hudson provides a super simple GUI that you can use to create new projects. This is one of its main attractions in my opinion as new builds can be configured in next to no time at all. The Dashboard and UI in general are very intuitive and easy to use, and if you’re like me and still would like to have the ability to look under the hood and modify the configurations, you also have that level of control as well. Kicking off a build manually or scheduling builds is just as easy.

    Overall I have to say that I am a fan of Hudson. Admittedly I just started using it so I am certain there is much more to learn about the tool, however if the rest of the functionality is as easy to configure as the basic features then this is a sure sell. I estimate it took me approximately 10 minutes total to download, install, configure a project and run a build (successfully I might add:).

    Some key features:

  • Master / Slave builds
  • Distributed Builds
  • GUI based project configuration
  • E-mail / AIM / RSS build notifications
  • Plug-ins / extensibility support
  • Simple Installation
  • Console Auto-refresh
  • So if you are interested in streamlining your Continuous Integration process I recommend taking a quick test drive of Hudson.