How we automatically verified and built a complicated, high-quality app project thousands of times
1,197 hours saved with these tools, tactics and 🤖 magic
We recently wrapped up a complicated mobile app development project that saw the creation of multiple native iOS apps built by a growing and diverse dev team.
A project of this magnitude obviously comes with its challenges…
- Expanding team: Over the course of 9 months, our team grew 7x.
- Language expertise & experience: While at Robots and Pencils, we’ve been using Swift since its 1.0 release, as we added team members, some developers were looking at Swift for the first time.
- Testing: We needed to build and deploy three separate apps against two different API environments, daily.
Choosing the right tools was crucial to our success — it reduced headaches and time costing issues along the way. In fact, getting it right was almost like adding another team member, saving us 1,197 hours in review and build time alone.
Here are the tools, tactics and 🤖 magic we used to automate and streamline our development process.
103 headaches prevented by creating a monorepo
We began with three Git repositories for the projects and another for shared code, but the overhead of keeping all four up to date quickly led us to transition to a single repo instead.
Inside this repo each project and the shared code had its own directory. A single Xcode workspace in the root allowed us easy access to all of the projects at once when we were making changes.
This “monorepo” organization reduced the cognitive load of the four repos and also saved time when compiling CocoaPods because there was only a single Pods project to build instead of four.
260 hours saved by automating our style cop with SwiftLint, Danger and CircleCI
One of the more contentious aspects of a large codebase is code style. Oh, how I wish it weren’t so.
Our robots have developed style guidelines for most of the languages that we use so code is consistent across developers and projects. Having style guidelines helps to smooth over everyone’s individual preferences on how to write code and provides answers to questions that come up when someone is learning a language.
The problem is that it’s not that fun to be a “style cop” and to tell someone that there’s a tedious issue with how they wrote something, as opposed to what that something does and how it does it.
Taking care of style issues during code review distracts from more meaningful discussion and also pushes that feedback later instead of being handled right away during development. It can be an important conversation to consider why one way of writing something is easier to understand than another, but it’s easy to devolve into a pull request review that’s taken over by “Prefer -> Void to -> () here.”
To automate our “style cop”, we turned to SwiftLint for help.
SwiftLint
SwiftLint is a tool originally developed by folks at Realm that checks your code against a set of rules and warns you if they’re broken.
- It’ll warn you in Xcode like the compiler does, so you can get feedback right after a build.
- It comes with a large set of rules, and you can configure which are important to your team and define some of your own.
- You even get to decide if a particular broken rule should be a warning or an error. This could come in handy if your team decides it really doesn’t like force-unwrapping Optionals.
- Some of the rules can automatically transform your code into the preferred style, like gofmt does. We didn’t turn this on yet, but it could get us closer to not caring about the tedium of style.
There are a few ways to install SwiftLint, but we decided to use CocoaPods.
Installing this way, as opposed to downloading a binary or using Homebrew, means that everyone on the team has the same version installed when they clone the repo, and they don’t need to perform any other steps.
We added a build phase for each project that used the binary installed with CocoaPods to check all of the code with our configured rules.
SwiftLint is something that we’ve already adopted on projects for other clients and easily gets a 👍 from the team.
The only issue was that occasionally we found someone would accidentally hide warnings in Xcode or miss checking the pane when reviewing a PR branch locally.
We needed to make these more visible during pull request review, which led us to Danger.
Danger
Danger is a tool that automates steps during code review. It’s perfect for doing extra checks that wouldn’t normally be covered by your CI test suite or checks that were previously done by a human. It can also add a comment with feedback in the pull request.
We initially added Danger in order to see any SwiftLint warnings for the code in files that changed in a pull request, but went on to add some more to safeguard against risky changes that might sneak in.
There are plugins for Danger available as Ruby gems, but because the Danger configuration file is written in Ruby, we were able to write our own rules, like:
- Warn if build settings that affect the default server environment or code signing settings were changed.
- Error if a change was committed that included focused Quick tests, which prevent the entire suite from running.
- Warn of misspellings of some project-specific words that had a particular meaning.
Just like warning in Xcode, if there’s too much feedback from Danger it can become easy to experience fatigue and ignore them. It’s important to clear up warnings that Danger adds to a pull request to prevent this.
Danger ends up being run as part of our CI suite, after tests complete, so that every pull request gets the same treatment. With very little configuration and lots of flexibility, it’s allowed us to catch sneaky issues in pull requests before they’re merged. Definitely a 👍 — we look forward to trying it on other projects soon.
CircleCI
We used CircleCI to run our large test suites and SwiftLint/Danger rules. A problem that we ran into with these apps was that as our codebase grew, our CI runs were starting to take a long time — sometimes more than 25 minutes. Most of this was due to Swift compile times that we couldn’t do much to change.
With our monorepo, including shared code, we wanted to make sure that any changed code as well as anything that depended on it was tested, but there was a lot of code that wasn’t being changed when someone developed a new feature or bug fix for only one of the apps.
We wrote a shell script that, for a given PR branch, got the list of files that changed in each project’s subdirectory and used that to determine which projects to build and test. If only one app’s code was changed then only that app would be built and tested, but if some shared code was changed then all affected apps would be built and tested.
This shell script became the entry point for our CI runs. Once it determined what projects to test it would use our fastlane test lane to run the tests. Because CircleCI automatically picks up multiple JUnit test reports for each project that’s built, we didn’t need to concatenate them and still saw a useful test summary.
When this test script was first written, we’d skip running tests on the master branch to save our CI machines for PR runs. This was later changed for code coverage reasons where we’d want to run everything to get a total number, but another benefit was that it made CircleCI’s Insights feature useful to us again.
The Insights feature allows you to see historical results for a particular branch, and so a long-running branch like master provides better feedback than a PR branch. Using this, we can see whether our test suite is getting slower or faster, or which tests in particular are running slowly or failing (hopefully none!).
We have some development tools written in Ruby that until recently had been installed manually in our circle.yml file and on each developer machine. Like our decision to use CocoaPods to install SwiftLint, there was a better way to make sure developers always had what they needed.
We added a Gemfile (like a Podfile for Ruby gems) and started to use Bundler to manage these dependencies instead. This not only made it easier for new developers to get the tools they needed, but also let CircleCI cache those Ruby gems so they didn’t need to be downloaded every build. Some gems have compiled dependencies and can take a long time to download and build, so this saved up to 2 minutes from each of our CI runs.
CircleCI is already in use on other projects at Robots and Pencils across a variety of platforms. Their great support of iOS projects earns a 👍!
937 hours saved by automating our builds with Fastlane, Jenkins and a bot named sudo
One of the last steps of adding code is to make a new build of the apps with all of the latest changes to share with the client. We have Jenkins servers set up just for this purpose on all of our projects, which means that with the click of a button we can deploy any app.
There’s a centralized configuration file for how each project is to be built, but more and more of this has shifted to use a tool called Fastlane. It allows you to easily and concisely set up common build tasks on iOS projects.
Fastlane + Jenkins
The Fastlane configuration file (called a Fastfile) lives alongside your code, which gives each project more flexibility than a centralized build configuration and lets developers run the same commands locally that the build server would. Because of these benefits, Fastlane has become our preferred way to define how a project should be built, tested and shared with clients.
Fastlane lets you define “lanes”, which are commands that will perform a set of actions. Right now we have lanes called test and testflight_staging. For example, our lane for deploying to TestFlight performs the following actions from one command:
- Updates the build number in the Info.plist
- Builds and signs the app
- Uploads the app to TestFlight
- Creates a new Git tag for the app, configuration and build number
- Pushes the new tag
- Uploads the dSYM to HockeyApp for crash symbolication
Each of these steps comes built-in to Fastlane. All we need to do is provide the configuration for things like code signing or which users to release a build to in TestFlight.
Using a monorepo allowed us to share Fastlane configurations. Each app we built needed to have these same lanes, but with minor tweaks like using the correct code signing settings, so we parameterized the lanes to be able to run them like “fastlane test app:app_one”. Because a Fastfile is just Ruby, we used a hash to store the necessary info for each app name.
Our Jenkins server knows how to run projects that use Fastlane for configuration, and so any developer working on these apps can make changes to how they’re tested and built without knowing anything about Jenkins (lucky them…) Fastlane is something we use on all new projects at this point, so it’s another easy 👍.
sudo
Based on the need for frequent feedback cycles from the client, we scheduled Jenkins to make new builds of each app in the early morning. These builds were automatically shared with internal TestFlight testers for verification before being sent to the client, who were external testers.
For ad-hoc builds or projects that don’t build nightlies, we use Slack bot we created called sudo. Sudo can do many things, including starting a new build for an app in Jenkins. This means we don’t even need to click a button anymore, we simply ask sudo to do it for us, and anyone in the Slack channel can see the progress.
In closing…
Just as it’s important to identify weaknesses in a codebase in order to iterate and refactor, doing the same for the tools that we use makes us more efficient and lets us concentrate on the creative and fun aspects of software development.
From the start of this project to where we ended up, it’s also important to understand that adopting each of these tools was a deliberate decision aimed at solving one particular problem we had at the time. Beyond the recommendations of any particular tool here, that’s the more important lesson.
Got another minute? You may also like:
