Setting up travis-ci for automated unit testing of iOS projects on GitHub

This guide is how I set up new projects to run unit tests automatically when I push a commit or merge a branch on GitHub for iOS projects.

The first step is to create a new project in Xcode. I am going to select a single view application and enable storyboards, Automatic Reference Counting and Unit Tests.

Screen Shot 2013-05-06 at 11.42.27 PM

All I’m going to change is the testExample method in the unit testing bundle to something that will pass. For now I have just gone with this:

- (void)testExample
{
    STAssertNil(nil, @"This object should be nil");
}

Check that the tests pass by hitting cmd+u. Now that I have my unit tests passing, I’m going to add this project to GitHub. I’ll leave the details out here, as this isn’t the focus of this tutorial but you can find my example project here: https://github.com/daniel-beard/SettingUpTravisCIForiOS.

Now we can setup the TravisCI build. I use the xctool to build my projects as it has a nicer output and is easier to use than the built in xcodebuild tool. Add xctool as a submodule to the git repository using the following commands:

git submodule add https://github.com/facebook/xctool.git ./xctool
git submodule update --init
git commit ./xctool -m "Added xctool as a submodule"

Then we need to add a config file so that travis-ci knows how to build our project. This file is named .travis.yml and lives in the root of the git repository. Here is the contents of mine:

language: objective-c
before_install: "git submodule init && git submodule update && sudo gem update --system && sudo gem install bundler && bundle install"
script: "bundle exec rake test --trace"

Then we have to add a Rakefile that tells the xctool which project and target to build:

desc 'Run the tests'
task :test do
   exec('xctool/xctool.sh -project SettingUpTravisCIForiOS.xcodeproj -scheme SettingUpTravisCIForiOS test')
end

task :default => :test

And finally the Gemfile

source 'https://rubygems.org'

gem 'rake'

You can then test locally that your project builds using the command rake in your root git repository.
I get build messages, then ** TEST SUCCEEDED: 1 of 1 tests passed ** (25285 ms). Now that we have the unit tests running locally, all that is left is to set up the travis-ci build. Login to https://travis-ci.org/ with your GitHub account and under account settings select the repository that should be unit tested automatically. This automatically sets up a service hook in GitHub so that every time you push to your repository, it will get unit tested.

You can also use the status images from travis-ci to show the test status directly in your README file on GitHub. Check out my example project here: SettingUpTravisCIForiOS

Advertisements

Automated Unit testing an iOS app with Jenkins

  • Setting up and executing automated unit tests is slightly more involved than generating built apps for distribution.
  • The following is the set up that I use for some of my projects, I have just renamed the project to TestProject for convenience.
  • For this article I am assuming that you have Jenkins, Xcode, Xcode Command Line Tools, and the Xcode Jenkins plugin already installed.
  • Because I use cocoaPods for dependency management, I build a workspace using custom build schemes. To build a single project with a unit-test target would make these steps easier.

Dependencies

  • HomeBrew
  • Ruby 1.9.3
    • Needed for other dependencies.
    • Can be installed via brew, or with RVM.
    • I used RVM to set up my ruby install.
  • Sinatra
    • Sinatra is a ruby based server that we use for serving JSON fixtures to the unit tests.
    • Can be installed using the ruby package manager sudo gem install sinatra
  • ios-sim
    • Required because Xcode doesn’t allow unit tests to be run natively in the iOS simulator from the command line.
    • Can be installed using brew: brew install ios-sim

Step 1 – Poll SCM

  • The TestProject Jenkins job polls the SCM looking for changes to the master branch at midnight every night.
  • If no changes have occurred, then the project is not built.
  • If modifications have been made, the next step is executed.
  • If you want, Jenkins can be set up to build on a push to a branch. E.g. Pushing to the remote master branch.

Step 2 – Simulator and Sinatra setup

We run the following script:

#!/bin/bash

#reset the content and settings of the iphone sim
rm -r ~/Library/Application\ Support/iPhone\ Simulator/

#open the iphonesimulator and kill it
#this is required after a system restart
#so the simulator knows to run iPad rather than iPhone apps
echo "Opening iphone simulator"
open "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/Applications/iPhone Simulator.app"
sleep 10
killall 'iPhone Simulator'
echo "iphone simulator killed"

#delete previous build folders
echo "Removing previous build folder"
rm -r ${WORKSPACE}/build
mkdir ${WORKSPACE}/build

#Start sinatra server in the background
ruby TestProject/server.rb &

#get the PID of the process
PID=$!

#save PID to file
echo $PID > ${WORKSPACE}/sinatra.pid
  • We first remove the iPhone Simulator folder
    • This makes sure that no previous TestProject apps are installed on the simulator. Otherwise we may get core data upgrade problems.
  • We then have to open the iphone simulator and then kill it
    • This is a stupid workaround that has to be done so that the iphonesimulator recognizes that we have to run an iPad application rather than an iPhone app.
  • We then remove any previous build folders.
    • Because we have our project set up as a workspace, there are multiple .xcodeproj files and libraries that we have to build, including our pods dependencies. Because of this, our default build location is relative to the project, not located in the iPhone Simulator folder or in Xcode’s DerivedData folder.
  • We then start the sinatra server in the background
    • The & operator detaches the ruby process from the current shell so that once this script has finished, the sinatra server is still running.
    • We store the PID of the process to the PID variable.
    • The $! expands to the process ID of the most recently executed background (asynchronous) command. More details here
    • The PID is then written to file so it persists.

Step 3 – Xcode Build

Below is a screenshot from jenkins showing the fields used for the xcode plugin

Screen Shot 2013-04-23 at 3.33.32 PM

  • Clean before build – we don’t want any cached compiled objects hanging around.
  • Xcode Schema FileTestProjectTests
    • Because of a limitation where workspaces can’t build targets directly, we have to use a Build Scheme to run unit tests. This scheme is set up the run the attached unit test target included in the production scheme TestProject
  • SDKiphonesimulator
    • We are targeting the simulator to run unit tests so we specify it here.
  • ConfigurationDebug
    • Unit tests only execute in Debug mode, so this option has to be this.
  • Custom xcodebuild arguments
    • TEST_AFTER_BUILD – We manually specify that we want to run unit tests after building the project.
    • ARCHS=i386 – We have to force the architecture to i386 because xcode wants to default to armv6, armv7 or armv7s.
    • ONLY_ACTIVE_ARCH=NO – Tell Xcode to not build just the architectures that it wants to by default.
    • VALID_ARCHS=i386 – We have to specify the architecture here again. Xcode does not make this easy for us.
    • SL_RUN_UNIT_TESTS=YES – This is where the magic happens, this will be explained in more detail in the next section.
  • Clean test reports?
    • This outputs clean test reports so we can export them to JUnit reports later.
  • Unlock keychain?
    • Required so we don’t have to enter the password to use debugging.

Step 4 – Unit testing

As explained in the previous step, the SL_RUN_UNIT_TESTS=YES xcodebuild argument is extremely important.

The TestProjectTests target in Xcode has a custom script that it executes after building. The script can be found in Project Settings -> TestProjectTest -> Build Phases -> Run Script

Screen Shot 2013-04-23 at 3.38.26 PM

The script is shown below:

ruby -v
ruby "${SRCROOT}/commandlineunittests.rb"
  • The first line is unnecessary, and just used for outputting the ruby version.
  • The second line calls a ruby script that is present in the repository that kicks off the unit tests.

The second ruby script is shown below:

if ENV['SL_RUN_UNIT_TESTS'] then
    launcher_path = "/usr/local/bin/ios-sim"    
    #File.join(ENV['SRCROOT'], "Scripts", "ios-sim")
    test_bundle_path= File.join(ENV['BUILT_PRODUCTS_DIR'], "#{ENV['PRODUCT_NAME']}.#{ENV['WRAPPER_EXTENSION']}")

    environment = {
        'DYLD_INSERT_LIBRARIES' => "/../../Library/PrivateFrameworks/IDEBundleInjection.framework/IDEBundleInjection",
        'XCInjectBundle' => test_bundle_path,
        'XCInjectBundleInto' => ENV["TEST_HOST"]
    }

    environment_args = environment.collect { |key, value| "--setenv #{key}=\"#{value}\""}.join(" ")

    app_test_host = File.dirname(ENV["TEST_HOST"])
    system("#{launcher_path} launch \"#{app_test_host}\" #{environment_args} --args -SenTest All #{test_bundle_path}")
else
    puts "SL_RUN_UNIT_TESTS not set - Did not run unit tests!"
end
  • The script checks for that magic variable SL_RUN_UNIT_TESTS and if it’s present runs the unit tests.
  • Using the ios-sim dependency, the script dynamically patches the TEST_HOST of the ios simulator and runs the unit tests. This is really complicated to try and do by hand, which is what we were doing before using ios-sim

Step 5 – Cleanup

The following script is executing after the unit tests have finished, regardless of the output status (PASS or FAIL).

#!/bin/bash

PID=$(<${WORKSPACE}/sinatra.pid)
echo "Sinatra server pid $PID"

kill -9 $PID
  • This script reads the process id (PID) from the file we stored earlier containing the sinatra server’s PID.
  • We then kill the sinatra process.
    • We don’t want the sinatra server hanging around after the unit tests have run, because subsequent tests will fail because they will try to start a sinatra server using the same port as the previous process.