I’ve been working on this side project codenamed Caretaker for a while now and since it wasn’t my main activity, I would sometime not do any work for weeks or months, depending on how my life was going (it’s going well, thank you for asking). Every time I wanted to go back to the project, I would start with the same thing: making sure my dependencies were up to date. A couple of months ago, I made a tool for it.

On keeping things up to date

Why should you keep your dependencies up to date? Because that’s the right thing to do. In fact, you should factor this into the decision process when you’re about to add a third party dependency to your project, because it’s time consuming. This is something that burnt me recently, as I tried to build a React application that I had not touched in a while: most of the dependencies were out of date, everything broke, sadness ensued. I made it work in the end, but that cost me a whole afternoon.

Note that you don’t have to always be on the latest and greatest version of the dependencies that you’re using in your project, but you should make sure that you’re keeping up to date with minor versions: they often include bug fixes, performances improvements and - even more important - security updates. Additionally, based on personal experiences I would say that given a pull request that takes a dependency x.y.z and updates it to x.y+n.z; the smaller n is, the easier your life is going to be. This is not always true, as doing proper versioning is sometimes complicated and some projects simply don’t bother, so your mileage may vary.

Finally, I would also mention that updating dependencies does not have to be painful if you’re willing (and able) to put in the work: a decent test coverage and a proper continuous integration setup go a long way to make sure that updating a particular dependency is safe.

Updating Caretaker’s dependencies

Caretaker is a project that involves a couple of small services, all written in Swift. Main dependencies include Vapor for the HTTP server and some NIO based clients to talk to the database, Redis, and so on. Those projects are pretty actives, so a super quick search in my previous pull requests for titles containing the word “dependencies” or “deps” yields around 20 pull requests opened over the course of 2+ years, and usually cover multiple dependencies at once. It’s a smallish project, so I never felt the need to open a separate pull-request for each dependency, unless my project was no longer building and I needed to handle one specific version bump separately. As far as I can remember, this only happened once.

At first, my workflow was the extreme opposite of automated: I would open my Package.swift file in my IDE and command-click on each package’s URL, which would open the repo’s page in my browser. I would then navigate to the releases pages (or just take a look at the list of tags) and compare the version that I’m using with the latest version. Then, I would update the version in the Package.swift file, run swift package resolve, commit the changes to the Package.resolved file and open a new pull request.

My project has a decent code coverage split between unit tests, integration tests and functional verification tests. The first two kinds run every time I open a pull-request (and prevent a merge unless they’re green) and the third one starts automatically once I’ve deployed a new version. This is why I was really happy when GitHub added support for a merge when green feature. I was available to slightly improve that workflow by opening the pull-request and move on to other things. That being said, opening a pull request with a small code change was an easy way to warm up and start working on that project again.

Writing a tool to automate the process

Dependabot was a tool that looked into your repository and automatically opened pull requests that bump dependencies, which made it pretty trivial to keep your dependencies up to date. Each pull request would include (when available) the change log and estimate the risk of merging that change. GitHub acquired this project in 2019 and it’s now baked into the product. Unfortunately, Swift packages are not supported by this tool, but I didn’t want to start working on a tool that would potentially be made obsolete the day after… unless I had a good reason.

Recent versions of Swift came with support for concurrency with async/await. I wanted to play around with those new tools, so all I needed was a small project idea where I could use those new APIs. That’s how “Agamotto” was born. Why Agamotto? Because I like code-naming my projects with Marvel characters, for some nerdy reason.

$ agamotto . --verbose
[jwt] Dependency is up to date
[swift-crypto] Dependency should be updated from 2.0.5 to 2.1.0
[fluent-postgres-driver] Dependency is up to date
[fluent] Dependency is up to date
[async-http-client] Dependency should be updated from 1.9.0 to 1.10.0
[soto] Dependency is up to date
[codablecsv] Dependency is up to date
[leaf] Dependency is up to date
[queues] Dependency is up to date
[queues-redis-driver] Dependency is up to date
[vapor] Dependency is up to date
[swift-backtrace] Dependency is up to date
[swift-log] Dependency is up to date

Before I start digging into how I implemented this tool, I should mention that there are alternatives out there. For example, this GitHub Action by Marco Eidinger that will fail your build if the dependencies are not up to date. I haven’t used it, but since I’m unable to release my tool, you may be interested.

Now, because my main goal was to play around with Swift’s new concurrency stuff, I wanted to keep things simple and tailored to my needs. Agamotto needed to:

• Run on macOS: I wanted a command line tool I would run every time I would start working on Caretaker.
• Only update the dependencies in the Package.swift file, no need to look into the transitive dependencies for now, like Dependabot does.
• Support public dependencies hosted on GitHub since that’s where are all my dependencies are hosted right now. Ideally, the code would be clean enough to be able to add support for new SCMs such as Gitlab or Bitbucket, as well as authentication.

Parsing dependencies

Because I wanted to keep things simple and didn’t want to spend countless hours parsing the Package.swift file, or see how I could include some of the Swift Package Manager’s libraries to do it for me, I went for the simplest route. Did you know that the swift package command line tool has a dump-package subcommand that does exactly what the name says? If you run swift package dump-package from your project’s folder (it doesn’t need to be ran from the root of the project), you get a JSON object that describes your package.

I needed two informations for each of the dependency: the url of the repository and the version currently used in the Package.swift file. As a side note, I should mention that all my dependencies are using exact versions and none of the .upToMajor nonsense. In my opinion, not using pinned version makes it harder to have deterministic builds and I’ve been burnt in the past with other dependency managers like Carthage or Cocoapods, or other dependency management tools for node.js and ruby.

Since the JSON file has many nested levels, I didn’t want to bother with defining a complex Swift struct covering those levels, so I decided to clean the resulting JSON before parsing it, using the jq command line tool. The final command looks like this:

swift package dump-package --package-path <path> | jq -Mc "[ .dependencies[].sourceControl[0] | { name: .identity, cloneURL: .location.remote[0], version: .requirement.exact[0] }]"

Note: the -M option disable colours so we don’t have to worry about ASCII characters and the -c option gives you a compact output. The later is not really required since the result is pretty small, but we’re still saving a few bytes I guess.

This is the result of the command for one of the services:

[
    {
        "name": "vapor",
        "cloneURL": "https://github.com/vapor/vapor.git",
        "version": "4.57.0"
    },
    {
        "name": "swift-crypto",
        "cloneURL": "https://github.com/apple/swift-crypto.git",
        "version": "2.0.5"
    }
]

Once translated to a Swift struct, we get:

public struct CloneUrl: Decodable {
    let value: URL

    public init(from decoder: Decoder) throws {
        self.value = try decoder.singleValueContainer().decode(URL.self)
    }

    // ... some other stuff that extracts the repo name and owner
    // from the URL, for example
}

public struct Dependency: Decodable {
    public let name: String
    public let cloneURL: CloneURL
    public let version: String
}

The final step involves invoking the command I mentioned earlier, grabbing the output and decoding the JSON. Once simplified for brevity, it looks like this:

let magicCommand = "<the command from earlier>"
let output = Pipe()

let process = Process()
process.executableURL = URL(fileURLWithPath: "/bin/bash")
process.arguments = ["-c", magicCommand]
process.standardOutput = output

try process.run()
process.waitUntilExit()

let data = try output.fileHandleForReading.readToEnd()
let deps = try JSONDecoder().decode([Dependency].self, from: data)

Of course, this is not perfect:

• You need to have the jq and swift command line tool installed and available in your $PATH, as well as bash.
• It doesn’t check the version of Swift currently installed, so if the format of the JSON changes between two releases, then the tool will break. As a matter of fact, this happened last week and I had to fix it.
• As I mentioned already, it only supports pinned version. If a dependency is using something else, the version field will contain null and the parsing will fail.

Fortunately, all of those things are definitely fixable shall I decide to address them in the future. This was not the focus for this tool, so those are limitation and quirks that I’m more than willing to live with.

Comparing versions

The first opportunity I had to play with concurrency was writing a small GitHub client that, given a repo owner and name, would retrieve the latest release of that repository using the dedicated endpoint. Specifically, what we care about is the tagName property.

{
    // ...

    "tag_name": "4.57.0"

    // ...
}

There are already many articles out there that cover how to use URLSession’s new concurrency APIs, so I’m not gonna cover it here in details. I would recommend starting with the WWDC session from 2021.

The only issue I ran into is that those APIs are not available on linux, so when I wanted to setup continuous integration for my project, I ran into compilations errors. As far as I can tell, those APIs are still not available on Linux in recent Swift snapshots, but I’m sure it’s only a matter of time. Meanwhile, it was fairly straightforward to write a shim using withCheckedThrowingContinuation so the project would compile on Linux and still look nice:

    func send<T: Decodable>(request: Request<T>) async throws -> (T, HTTPURLResponse) {
        let urlRequest = URLRequest(url: baseEndpoint.appendingPathComponent(request.path))

        return try await withCheckedThrowingContinuation { continuation in
            URLSession.shared.dataTask(with: urlRequest) { data, response, error in
                if let error = error {
                    return continuation.resume(throwing: error)
                }

                guard let data = data, let response = response as? HTTPURLResponse else {
                    return continuation.resume(throwing: GitHubError.invalidResponse)
                }

                guard response.statusCode >= 200 && response.statusCode < 400 else {
                    return continuation.resume(throwing: GitHubError.httpError(response: response))
                }

                do {
                    let payload = try GithubClient.githubJsonDecoder.decode(T.self, from: data)
                    continuation.resume(returning: (payload, response))
                } catch let error {
                    return continuation.resume(throwing: error)
                }
            }.resume()
        }
    }

Checking for multiple repository releases at once

One aspect of concurrency that I was curious about is running multiple tasks in parallel. Even if I’m pretty familiar with concurrency in a bunch of languages, I was curious to see how it was implemented in Swift. As it turns out, it uses Task groups. This is what it currently looks like:

enum DependencyCheckResult {
    case unknown
    case upToDate
    case outdated(currentVersion: String, latestVersion: String)
}

// ...
let statuses = try await withThrowingTaskGroup(of: (Dependency, DependencyCheckResult).self, returning: [(Dependency, DependencyCheckResult)].self) { group in
    var statuses = [(Dependency, DependencyCheckResult)]()
    statuses.reserveCapacity(deps.count)

    for dep in deps {
        group.addTask(priority: .medium) {
            try (dep, await checkDependency(dependency: dep))
        }
    }

    return try await group.reduce(into: statuses) { $0.append($1) }
}

I’m pretty sure it can be improved since I haven’t touched this particular piece of code in months, but it has been working fine so I haven’t bothered yet. It needs to be said but this is massively overkill: I only have about 15 dependencies on this project and it’s unlikely that I will add much more. By querying the releases in parallel here, I don’t think I’m saving any time. Additionally, I don’t think that unbounded concurrency is a good idea anyway. If I was dealing with more objects, I would probably look into streaming with some of the fancy algorithms that Apple released recently.

Comparing versions

After fetching the latest versions of each of the dependencies, we can compare what that version is to what I’m using locally. As I mentioned earlier, I’m only dealing with pinned versions, so it’s only a matter of making sure those values are equal.

for (dep, status) in statuses {
    switch status {
    case .unknown:
        print("[\(dep.name)] Unable to determine the latest release for this dependency")
    case .upToDate:
        print("[\(dep.name)] Dependency is up to date")
    case .outdated(currentVersion: let currentVersion, latestVersion: let latestVersion):
        print("[\(dep.name)] Dependency should be updated from \(currentVersion) to \(latestVersion)")
    }
}

Conclusion

Native support for concurrency, using a well known syntax with async and await is amazing news for Swift. Agamotto, while being a simple project, helped me try it and I really enjoyed the developer experience. That being said, just writing about this tool almost 6 months after I touched this code for the last time made me want to dust it and improve it (by removing the dependency on jq, offering to update the Package.swift file automatically…). I’m still hoping for Swift support on GitHub’s dependency checker, but even if that happens, having a tool that I can run locally is still pretty helpful.