Swiftology

Articles on advanced Swift topics, functional programming, and software design.

written byAlex Ozun
written with 🧡 and ☕️ by Alex Ozun

Pitfalls of Parameterized Tests

Nov 11, 2025

﹒

☕️ 10 min read

I'll start by saying that I really like Swift Testing! It's a library that fully lives up to having 'Swift' in its name, being modern, expressive, and cross-platform. When it was announced at WWDC24, it was already notably stable and full-featured, a distinction in itself in our age of 'promise now, deliver later'. So, adopting it right out of the gate was a complete no-brainer. Indeed, our team had already converted tens of thousands of tests from XCTest to Swift Testing, and is well underway to full adoption.

In the process of this large-scale migration, I had to discover new testing patterns and learn to avoid some newfound pitfalls. And I'm still learning, so if you find yourself disagreeing with anything I'm about to write, don't hesitate to leave a comment below and point out my blind spots!

Ok, enough of the intro, let's talk about Parameterized Tests and the pitfalls we should be aware of.

Parameterized Tests probably don't need much introduction, being one of the most advertised features of Swift Testing. Fundamentally, they let us define a single test that runs for a collection of input arguments, potentially replacing multiple separate tests.

Here's an example:

enum Day: String {
  case monday, tuesday, wednesday,
    thursday, friday, saturday, sunday
}

func greeting(of day: Day) -> String {
  "Happy \(day.rawValue)!"
}

To test greeting(of:Day), we'd normaly write 7 separate tests:

@Test func monday() {
  #expect(greeting(of: .monday) == "Happy Monday!")
}
@Test func tuesday() {
  #expect(greeting(of: .tuesday) == "Happy Tuesday!")
}
@Test func wednesday() {
  #expect(greeting(of: .wednesday) == "Happy Wednesday!")
}
//...4 more..

But Parameterized Tests present a tantalizing opportunity to replace all 7 tests with just one! We can conform Day to CaseIterable and pass all days as an argument for the @Test macro:

enum Day: String, CaseIterable { ... }

@Test(arguments: Day.allCases)
func greeting(day: Day) {
  #expect(greeting(of: day) == "Happy \(day.rawValue)!")
}

Pretty cool, eh? But before we get too excited, let's take a critical look 🕵🏻‍♂️

Pitfall №1: Gaps in test coverage

Astute readers will notice that my implementation of greeting(of:Day) actually contains a mistake!

func greeting(of day: Day) -> String {
  "Happy \(day.rawValue)!"
}

It uses day.rawValue without capitalization, so the resulting greeting is "Happy monday!" instead of "Happy Monday!".

All 7 original tests would catch this mistake because each asserted a day's greeting using a concrete string literal :

#expect(greeting(of: .monday) == "Happy Monday!")
// 🛑 Failed, expected: "Happy Monday!", actual: "Happy monday!"

But the parameterized test happily passes because it asserts using a computed string with an opaque day variable:

#expect(greeting(of: day) == "Happy \(day.rawValue)!")
// ✅ Passed

We have introduced a gap in test coverage that didn't exist before we converted to Parameterized Tests. Admittedly, the mistake in this example may be trivial and inconsequential, but I observed the exact same gap occur around error codes and analytics events. Mistakes with real business impact.

Pitfall №2: Logic in tests

It may not be immediately obvious, but in addition to the gap in test coverage, our new parameterized test has also introduced logic into the tests, mirroring the implementation of the function under test:

@Test(arguments: Day.allCases)
func greeting(day: Day) {
  #expect(greeting(of: day) == "Happy \(day.rawValue)!")
}

Let's make this fact more obvious by updating our greeting(of: Day) function:

func greeting(of day: Day) -> String {
  if day == .friday {
  "Thank God it's Friday!"
  } else {
    "Happy \(day.rawValue)!"
  }
}

As expected, the test now fails for the .friday case, and to make it pass we have to mirror the change in the test itself:

@Test(arguments: Day.allCases)
func greeting(day: Day) {
  if day == .friday {
  #expect(day.greeting == "Thank God it's Friday!")
  } else {
    #expect(day.greeting == "Happy \(day.rawValue)!")
  }
}

Had this been an old-fashioned single-case test, the presence of control flow statements would surely raise eyebrows. But in a parameterized test with varying arguments, it’s easy to assume the logic merely differentiates test cases rather than indicates tight coupling with the implementation of the function under test. This assumption, of course, is flawed. And even if we recognize this problem, we may be tempted to ignore it so long as it still lets us replace 7 near-duplicate tests with just a single one. A small price, perhaps?

We may admit that .friday is special, after all, and deserves its own test, allowing other days to continue enjoying the benefits of parameterization:

@Test func friday() {
  #expect(greeting(of: .friday) == "Thank God it's Friday!")
}

@Test(arguments: [Day]([
  .monday, .tuesday, .wednesday, 
  .thursday, .saturday, .sunday
]))
func greeting(day: Day) {
  #expect(day.greeting == "Happy \(day.rawValue)!")
}

This may look like a reasonable compromise, but it only masks the underlying problem, and kicks the can down the road.

I'll show what I deem a proper solution shortly, but first let's look at other pitfalls.

Pitfall №3: Fragile tests

To illustrate the next pitfall, I'll use a slightly different example. In fact, it's the one that Apple used in WWDC24: Go further with Swift Testing.

We have a bunch of ingredients that can be cooked into dishes:

enum Ingredient {
  case rice, potato, egg, lettuce
}
enum Dish {
  case onigiri, fries, omelette, salad
}
func cook(ingredient: Ingredient) -> Dish {
  switch ingredient {
  case .rice: .onigiri
  case .potato: .fries
  case .egg: .omelette
  case .lettuce: .salad
  }
}

Mappings like this are commonplace in Swift codebases.

Conveniently, Parameterized Tests can take two collections of arguments! We can parameterize our test like this:

enum Ingredient: CaseIterable {}
enum Dish: CaseIterable {}

@Test(arguments: Ingredient.allCases, Dish.allCases)
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

The first argument provides input for the function under test, and the second represents the expected output. Looks promising!

But as the WWDC presenter points out, this would actually produce 16 test cases, combining each Ingredient case with each Dish case: 4X4=16. Including invalid combinations like .lettuce cooked into an .omelette.

To remedy this, Apple suggests to use a built-in zip(_:_:) function, which instead of 16 permutations of all elements, creates 4 pairs in the original order:

zip(Ingredient.allCases, Dish.allCases)
[             [               [
  .rice,    +   .onigiri,  ->   (.rice, .onigiri),
  .potato,  +   .fries,    ->   (.potato, .fries),
  .egg,     +   .omelette, ->   (.egg, .omelette),
  .lettuce  +   .salad     ->   (.lettuce, .salad)
]             ]               ]

This is exactly what we need to pair inputs with outputs. Let's take it for a spin:

//Zipped to 4 test cases
@Test(arguments: zip(Ingredient.allCases, Dish.allCases))
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

This works perfectly, correctly mapping each Ingredient to its expected Dish. The test has no gaps in coverage and no logic tied to the function under test. It’s a solid example of how to parameterize a test properly.

But it turns out this test is fragile in a subtle way. It’s sensitive to the order of cases in the Ingredient and Dish enums:

enum Ingredient: CaseIterable {
  case rice, potato, egg, lettuce
}
enum Dish: CaseIterable {
  case onigiri, fries, omelette, salad
}

It's normal for Swift programmers to reorder enum cases at will. In particular, it's common for teams to alphabetize enums, indeed, some even set up their code formatters to do this automatically:

// Alphabetized
enum Ingredient: CaseIterable {
  case egg, lettuce, potato, rice 
}
// Alphabetized
enum Dish: CaseIterable {
  case fries, omelette, onigiri, salad
}

But if we reorder cases in these enums, our tests will suddenly start failing, now expecting .egg to be cooked into .fries, .lettuce into .omelette, and so on.

Most frustrating, the tests fail for no good reason other than a purely syntactic code change. Reordering has no effect on production code, since the cook() mapping doesn’t depend on case order.

Test-induced damage

Let’s take a step back and look at what caused this case-order-sensitive fragility. It came from our decision to make Ingredient and Dish conform to CaseIterable so we could use the auto-generated .allCases arrays in our parameterized tests.

Crucially, the only reason we made these enums conform to CaseIterable was for the sake of testing, with no obvious concern or benefit to the production code. That alone should raise eyebrows, indeed some folks even call such phenomena test-induced damage.

When a collaborator looks at the Ingredient and Food enums, it won’t be clear to them why they conform to CaseIterable, nor will anything indicate that their cases must not be reordered, lest something breaks.

Further, if we later decided to introduce associated values to some cases, for example .rice(.brown) and .rice(.white), it would be impossible without breaking CaseIterable conformance. A collaborator might wrongly assume that these enums must not introduce associated values because the CaseIterable conformance is somehow important for production code.

Pitfall №4: (Larger) gaps in test coverage

Let's add a new Ingredient.tomato that cooks into a Dish.salad, just like .lettuce.

enum Ingredient: CaseIterable {
  case rice, potato, egg, lettuce, tomato
}
func cook(ingredient: Ingredient) -> Dish {
  switch ingredient {
  case .rice: .onigiri
  case .potato: .fries
  case .egg: .omelette
  case .lettuce, .tomato: .salad
  }
}

We expect our Parameterized Test to remain unchanged and still compile, relying on Ingredient.allCases to generate an array of all cases, with the new .tomato automatically included:

@Test(arguments: zip(Ingredient.allCases, Dish.allCases))
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

And if we re-run the test....it still passes ✅

Wait, how does the test know that .tomato should produce .salad?

The problem is, .tomato case is never even tested! This happens because when we zip(_:_:) two arrays of different lengths, the resulting array of pairs is cut off to the length of the shorter array.

Ingredient.allCases now has 5 elements, but Dish.allCases still has 4 elements, which means that the last .tomato case is simply dropped:

zip(Ingredient.allCases, Dish.allCases)
[              [              [
  .rice,    +   .onigiri,  ->   (.rice, .onigiri),
  .potato,  +   .fries,    ->   (.potato, .fries),
  .egg,     +   .omelette, ->   (.egg, .omelette),
  .lettuce, +   .salad    ->    (.lettuce, .salad)
  .tomato //⚠️ dropped...
]              ]              ]

Unless we pay close attention to the test report, we won't notice that .tomato case isn't being tested. We now have an even larger gap in test coverage, where not just part of the expected result is ignored, but an entire test case!

Pitfall №5: Reduced readability and transparency

So far our main source of grief has been CaseIterable and the use of auto-generated .allCases in our parameterizes tests. Instead, let’s create dedicated, hand-crafted test cases:

extension Ingredient {
  static let testCases: [Ingredient] = [
    .rice, .potato, .egg, .lettuce, .tomato
  ]
}
extension Dish {
  static let testCases: [Dish] = [
    .onigiri, .fries, .omelette, .salad, .salad
  ]
}

We can define these testCases in a test file without affecting production code and retain full control over the elements and their order. Unlike CaseIterable, this also works for enum cases with associated values. Apple demonstrates this approach in Implementing parameterized tests.

This also allows multiple test functions to reuse the same test cases:

@Test(arguments: zip(Ingredient.testCases, Dish.testCases))
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

@Test(arguments: Ingredient.testCases)
func isFresh(ingredient: Ingredient) {
  #expect(ingredient.isFresh)
}

While this approach solves some of the problems we had with CaseIterable , it still suffers from compromises inherent in code outlining - reduced locality and transparency.

At a glance, it's impossible to tell what ingredients are tested or what dishes are expected by this test, we have to jump to the definitions of the testCases variables to find this out:

                                 🤔              🤨
@Test(arguments: zip(Ingredient.testCases, Dish.testCases))
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

Readability-related opinions are very subjective, so it's totally cool if you find yourself disagreeing with me here.

Whenever we extract a piece of code into a separate function or variable, an operation known as code outlining or indirection, we reduce local reasoning and transparency in exchange for abstraction.

Code outlining and other forms of abstraction can greatly benefit design and even improve readability when applied to implementation details that are secondary to the key idea being expressed.

But in tests, What is being tested and What is expected are the key ideas. They should ideally be specified explicitly and directly in the test itself. There is a notable exception with property-based testing that I'll talk about in a separate section.

We can reclaim transparency and local reasoning by passing arguments and expected results directly as array literals:

@Test(arguments: zip(
  [Ingredient]([.rice, .potato, .egg, .lettuce, .tomato]),
[Dish]([.onigiri, .fries, .omelette, .salad, .salad])
)
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

I think this greatly improves readability, even at a cost of some code repetition and verbosity. But we can do even better...

Better ways to Parameterize Tests

In the example above, even though we explicitly pass arguments as array literals, we're still relying on the built-in zip(_:_) function that may cut-off elements when two arrays have different lengths:

@Test(arguments: zip(
  [Ingredient]([.rice, .potato, .egg, .lettuce, .tomato]),
  [Dish]([.onigiri, .fries, .omelette, .salad]) //⚠️ dropped...
)
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

To avoid this problem altogether, I recommend not using the built-in zip(_:_:), which wasn’t designed with Swift Testing in mind and is a poor tool when input and output arrays must be the same length.

Option 1: Single array of pairs

Instead of using the built-in zip(_:_:) to produce an array of pairs for us, we can just create it directly ourselves:

@Test(arguments: [
  (Ingredient.rice,    Dish.onigiri),
  (Ingredient.potato,  Dish.fries),
  (Ingredient.egg,     Dish.omelette),
  (Ingredient.lettuce, Dish.salad),
  (Ingredient.tomato,  Dish.salad),
])
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

This way, it’s impossible to add a new argument without pairing it with an expected result, or vice versa. @Test macro also conveniently destructures tuples into separate function arguments.

Option 2: Dictionary

A similar, and perhaps even more intuitive, option is to use a humble dictionary1

@Test(arguments: [
  Ingredient.rice   : Dish.onigiri,
  Ingredient.potato : Dish.fries,
  Ingredient.egg    : Dish.omelette,
  Ingredient.lettuce: Dish.salad,
  Ingredient.tomato : Dish.salad,
])
func cook(_ ingredient: Ingredient, into dish: Dish) {
  #expect(cook(ingredient) == dish)
}

The @Test macro again has our backs by automatically mapping the Key and Value types into two separate function arguments, very convenient!

Thanks a lot @simonomi for suggesting this option in the comment below!

Option 3: Fixed-size Zip, backed by InlineArray (Swift 6.2 +)

Swift 6.2 introduced InlineArray - a fixed-size array where a generic type parameter specifies the length of the array at compile time.

This allows us to create a new version of zip(inputs:outputs:) that enforces two arrays to have the same length at compile time.

The interface looks like this:

func zip<let length: Int, Input, Output>(
  inputs: InlineArray<length, Input>,
  outputs: InlineArray<length, Output>
) -> Zip2Sequence<[Input], [Output]>

The key part is the type parameter let length: Int which is applied to both arrays, binding them together by the same-length requirement.

This way, if we pass arrays of different lengths the code won't even compile!

@Test(arguments: zip(
  inputs: [Ingredient.rice, Ingredient.potato],
  outputs: [Dish.onigiri]
  // 🛑 Error: Expected 2 elements, but got 1.
)
func cook(_ ingredient: Ingredient, into dish: Dish) {...}

Why use the fixed-size zip(inputs:outputs:) instead of a single array of pairs or a dictionary, as in the previous examples? It's a purely formatting choice, when inputs and outputs take a lot of horizontal space and splitting them into separate vertical sections may be preferential.

Here's a visual comparison.

Fixed-size Zip:

@Test(arguments: zip(
  inputs: [
    Recipe(day: .monday, ingredient: .lettuce),
    Recipe(day: .friday, ingredient: .potato),
  ],
  outputs: [
    "Happy Monday! Let's start the week with a healthy Salad!",
    "Thank God it's Friday! How about some crispy Fries with your 🍻!",
  ]))
func message(for recipe: Recipe, output: String) {...}

Array of pairs:

@Test(arguments: [
  (
    Recipe(day: .monday, ingredient: .lettuce),
    "Happy Monday! Let's start the week with a healthy Salad!"
  ),
  (
    Recipe(day: .friday, ingredient: .potato),
    "Thank God it's Friday! How about some crispy Fries with your 🍻!"
  )
])
func message(for recipe: Recipe, output: String) {...}

Here's a full implementation of the fixed-size zip(inputs:outputs:):

func zip<let length: Int, Input, Output>(
  inputs: InlineArray<length, Input>,
  outputs: InlineArray<length, Output>
) -> Zip2Sequence<[Input], [Output]> {
  var ins: [Input] = []
  for i in inputs.indices {
    ins.append(inputs[i])
  }
  var outs: [Output] = []
  for i in outputs.indices {
    outs.append(outputs[i])
  }
  return zip(ins, outs) // Passing two same-length arrays to the built-in zip
}

In Swift 6.2, we can only iterate over an inline array using indices, not having access to map or even for-in loops, but this limitation is not a big deal here, and may be lifted at some point.

Property-based testing

So far, I’ve been building a case (pun intended) against using CaseIterable.allCases in parameterized tests. I've also recommended specifying expected results using concrete values rather than opaque variables or computed properties.

Now I'll demonstrate valid scenarios where going against these recommendations is the whole point of the test.

Property-based testing is a fundamentally different approach to testing, and its specifics are well beyond the scope of this already long article (and I don't claim expertise), but here's a gist.

Traditional example-based testing uses pre-defined arguments and verifies correctness by asserting that they produce pre-defined results:

@Test("Verify Integer doubling using some examples")
func double() {
  #expect(1 * 2 == 2)
  #expect(2 * 2 == 4)
  #expect(3 * 2 == 6)
  ...
}

Property-based testing, by contrast, must only use auto-generated arguments and is forced to verify correctness by asserting that some universal property holds true:

@Test(
  "For any number, doubling it is the same as adding it to itself."
  arguments: Int.random(range: Int.min...Int.max, count: 100)
)
func double(n: Int) {
  #expect(n * 2 == n + n)
}

Notice that both the argument and the result are opaque variables and not concrete values. That’s because the whole point of this test is to verify a universal property that holds for all numbers: doubling a number is the same as adding it to itself. Crucially, each time the test runs, it generates a new set of 100 random numbers.

Here's an example of using CaseIterable, which is good way of supplying auto-generated arguments for a property-based test.

Let's create an Orientation enum to describe where an arrow is pointing:

enum Orientation: CaseIterable {
  case up, down, right, left
}

...and create a function that turns the arrow clockwise ⬆️➡️⬇️⬅️

   func turnClockwise() -> Orientation {
    switch self {
    case .up: .right
    case .right: .down
    case .down: .left
    case .left: .up
    }
  }

We can write a test parameterized with Orientation.allCases to verify a property that for all orientations turning clockwise 4 times returns to the original orientation, completing a full circle:

@Test(
  "For all orientations, turning clockwise 4 times returns to the original orientation",
  arguments: Orientation.allCases
)
func fullCircle(orientation: Orientation) {
  #expect(
    orientation
      .turnClockwise()
      .turnClockwise()
      .turnClockwise()
      .turnClockwise()
    == orientation
  )
}

Another test could assert a different property - that clockwise and counterclockwise turns cancel each other out:

@Test(
  "For all orientations, clockwise and counter-clockwise turns cancel each other out",
  arguments: Orientation.allCases
)
func turnsCancelOut(orientation: Orientation) {
  #expect(
    orientation
      .turnClockwise()
      .turnCounterClockwise()
    == orientation
  )
  #expect(
    orientation
      .turnCounterClockwise()
      .turnClockwise()
    == orientation
  )
}

Conclusions

Parameterized Tests are a powerful tool, but one that must be wielded responsibly. Not every test should be parameterized, and learning when to apply them effectively takes practice, so don't worry if you stumble at first.

Pay attention to:

  • Test arguments that appear on both sides of the #expect expression. This may indicate gaps in test coverage and tight coupling to the implementation details of the function under test.
  • Conforming enums to CaseIterable only for the sake of testing may create unnecessary and non-obvious friction in production code. Try to limit the use of .allCases in parameterized tests when it's already used in production.
  • Test arguments extracted into a separate function or variable may reduce readability and transparency. Specify arguments directly in @Test(arguments:) using array literals whenever possible.
  • The built-in zip(_:_:) function may drop test cases when two arrays have different lengths, leaving large gaps in test coverage. Prefer a single array of tuples or a custom fixed-size zip(inputs:outputs:) function.
  • When the system under test has universal properties that hold true for various sets of arguments, parameterized tests can be a great tool for property-based testing.

Thanks for reading! Until next time.

Footnotes

  1. Dictionary requires the Key type to be Hashable which might not be practical. You can work around this requirement by casting the dictionary literal to KeyValuePairs which allows non-Hashable keys:

    ↩️ 
struct Person { // not Hashable
  let name: String 
} 
@Test(arguments: [
  Person("Bob")   : "Hello, Bob"
  Person("Alice") : "Hello, Alice"
] as KeyValuePairs)
func test(person: Person, greeting: String) {...}
Buy me a coffee

Discuss on Twitter

Discuss on LinkedIn

Subscribe via RSS