Press → to see first slide Press ESC to see all slides

Jakub Sowiński

jakubsowinski.com | mail[at]jakubsowinski.com

How to improve code quality with property-based testing

📚 Topics

What's missing in the way we test our software?
How can Property Based Testing fix it?
How to write Property Based Tests?
What else can they be used for?

What's missing in the way we test our software?

Why do we write unit tests?

To check if code works as intended
To help future us modify it and be sure it still works

If the tests all pass, it’s unlikely that the change broke something unexpected. The tests make small changes virtually risk free
Robert C. Martin

source: http://it4kt.cnl.sk/c/zsi/resources/Martin2007.pdf

How do we write unit tests?

/numbers/index.js

function sum(a, b) {
  return a + b;
}

describe('sum function', () => {
  test('checks if 1 + 2 equals 3', () => {
    expect(sum(1, 2)).toBe(3);
  });
});

/numbers/__tests__/exampleBased.test.js

describe('sum function', () => {

  test('checks if 1 + 2 equals 3', () => {
    expect(sum(1, 2)).toBe(3);
  });

  test('checks if -1 + 1 equals 0', () => {
    expect(sum(-1, 1)).toBe(0);
  });

  test('checks if 0 + 0 equals 0', () => {
    expect(sum(0, 0)).toBe(0);
  });

});

Why this sucks? 👎

It's a waste of time
You won't find bugs you don't expect to find

True wisdom is knowing what you don't know.
Confucius

source: https://www.umbel.com/blog/services/know-what-you-dont-know/

Fuzzy Testing

Fuzzy Testing

Passing randomly-generated data (often a purely random bytestream) to a program in the hopes of finding an input that causes a crash.

source: https://blog.nelhage.com/post/property-testing-is-fuzzing/

Property Based Testing

Write statements that should be true of your code
Automatically generate random inputs of an appropriate type
Observe whether the properties hold for that input

source: https://blog.nelhage.com/post/property-testing-is-fuzzing/

/numbers/__tests__/propertyBased.test.js

test('checks if adding two random positive numbers results in their sum', () => {
  const numberLimit = 1000;

  const number1 = Math.random() * numberLimit;
  const number2 = Math.random() * numberLimit;
  const expectedResult = number1 + number2;

  console.log(`asserting if ${number1} + ${number2} equals ${expectedResult}`);
  expect(sum(number1, number2)).toBe(expectedResult);
});

/numbers/__tests__/propertyBased.test.js

test('checks if adding two random numbers results in their sum x 100 times', () => {
  const numberLimit = 1000;

  for (let i = 0; i < 100; i++) {
    let number1 = Math.random() * numberLimit;
    number1 = Math.random() >= 0.5 ? number1 : -number1;

    let number2 = Math.random() * numberLimit;
    number2 = Math.random() >= 0.5 ? number2 : -number2;

    const expectedResult = number1 + number2;

    expect(sum(number1, number2)).toBe(expectedResult);
  }
});

Why this is cool? 👍

Your test runs on random set of data every time
You can find issues you would have not thought of

/numbers/__tests__/propertyBased.test.js

test('checks if adding two random numbers results in their sum x 100 times', () => {
  const numberLimit = 1000;

  for (let i = 0; i < 100; i++) {
    let number1 = Math.random() * numberLimit;
    number1 = Math.random() >= 0.5 ? number1 : -number1;

    let number2 = Math.random() * numberLimit;
    number2 = Math.random() >= 0.5 ? number2 : -number2;

    const expectedResult = number1 + number2;

    expect(sum(number1, number2)).toBe(expectedResult);
  }
});

Why does it still suck, though? 👎

Defining properties and preconditions for each test manually takes time and effort
Debugging such test in case of failure is inconvenient

Tools

Libraries for Property Based Testing

Java - JUnit-QuickCheck or QuickTheories
PHP - PHP-QuickCheck or Eris
JavaScript - JSVerify or Fast-Check

source: https://github.com/dubzzz/fast-check

/numbers/__tests__/propertyBased.test.js

import { sum } from '../';
import fc from 'fast-check';

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    fc.assert(
      fc.property(fc.float(), fc.float(), (a, b) => {
        expect(sum(a, b)).toBe(a + b);
      }),
      { verbose: true },
    );
  });
});

Configuration of assertions runner

source: https://github.com/dubzzz/fast-check

/numbers/__tests__/propertyBased.test.js

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    fc.assert(
      fc.property(fc.float(), fc.float(), (a, b) => {
        expect(sum(a, b)).toBe(a + b);
      }),
      {
        verbose: true,
        examples: [[1, 2], [-1, 1], [0, 0]],
      },
    );
  });
});

/numbers/__tests__/propertyBased.test.js

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    fc.assert(
      fc.property(fc.float(), fc.float(), (a, b) => {
        expect(sum(a, b)).toBe(a + b);
      }),
      { verbose: true },
    );
  });
});

Property

fc.property(fc.float(), fc.float(), (a, b) => { ... })

for all (x, y, ...) such as precondition(x, y, ...) holds property(x, y, ...) is true
for all values that meet given condition(s) given statement is true
consists of arbitraries and predicate

source: https://github.com/dubzzz/fast-check

Arbitrary

fc.float(), fc.float()

Responsible for the random - but deterministic - generation of data

>source: https://github.com/dubzzz/fast-check/blob/master/documentation/Arbitraries.md

Arbitraries

numbers (up to MAX_SAFE_INTEGER)
strings (ascii, json, lorem)
collections (arrays, sets, objects)
combinors (option, oneof)
custom

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Arbitraries.md

Predicate

(a, b) => { ... }

Accepts values provided by arbitraries
Performs an assertion

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Runners.md

/numbers/__tests__/propertyBased.test.js

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    fc.assert(
      fc.property(fc.float(), fc.float(), (a, b) => {
        expect(sum(a, b)).toBe(a + b);
      }),
      { verbose: true },
    );
  });
});

/strings/index.js

function contains(text, pattern) {
  return text.indexOf(pattern) >= 0;
}

/strings/__tests__/propertyBased.test.js

describe('contains function', () => {
  test('checks if randomly generated string contains itself x 100 times', () => {
    fc.assert(
      fc.property(fc.string(), (text) => {
        expect(contains(text, text)).toBe(true);
      })
    );
  });
});

/strings/__tests__/propertyBased.test.js

describe('contains function', () => {
  test('checks if randomly generated string contains it\'s parts x 100 times', () => {
    fc.assert(
      fc.property(fc.string(), fc.string(), fc.string(), (a, b, c) => {
        expect(contains(a + b + c, a)).toBe(true);
        expect(contains(a + b + c, b)).toBe(true);
        expect(contains(a + b + c, c)).toBe(true);
      })
    );
  });
});

How to come up with scenarios?

Round-tripping

parsing
serialization

source: https://blog.ssanj.net/posts/2016-06-26-property-based-testing-patterns.html

Commutativity

filter and sort <-> sort and filter

source: https://blog.ssanj.net/posts/2016-06-26-property-based-testing-patterns.html

Invariants

content of list after sorting
size of list after mapping

source: https://blog.ssanj.net/posts/2016-06-26-property-based-testing-patterns.html

🐛 Debugging

/numbers/__tests__/propertyBased.test.js

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    fc.assert(
      fc.property(fc.float(), fc.float(), (a, b) => {
        expect(sum(a, b)).toBe(1);
      }),
      { verbose: true },
    );
  });
});

🐛 Debugging utilities

/numbers/__tests__/propertyBased.test.js

describe('sum function', () => {
  test('checks if adding two random numbers results in their sum x 100 times', () => {
    console.log(
      fc.check(
        fc.property(fc.float(), fc.float(), (a, b) => {
          expect(sum(a, b)).toBe(1);
        }),
        { verbose: true },
      )
    );
  });
});

fc.check result interface

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Runners.md

🐛 Debugging utilities

Statistics of assertion runner
Samples from arbitraries
Shrinking
Targeted search

QuickTheories

public class SomeTests {

  @Test
  public void addingTwoPositiveIntegersAlwaysGivesAPositiveInteger(){
    qt()
      .forAll(
        integers().allPositive(),
        integers().allPositive()
      )
      .check((i,j) -> i + j > 0);
  }

}

source: https://github.com/quicktheories/QuickTheories

PHP-QuickCheck

$stringsAreNeverNumeric = Gen::forAll(
  [Gen::asciiStrings()],
  function($str) {
    return !is_numeric($str);
  }
);

Quick::check(1000, $stringsAreNeverNumeric);

source: https://github.com/steos/php-quickcheck

Beyond Property Based Testing

Model Based Testing

Define simplified version of the system
Define commands that can be applied to the system
Framework picks commands randomly and executes them sequentially
Tests fail when framework cannot execute command despite the fact it should be able to
Framework returns full scenario description in case of failure

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

Command - methods

check - true if the command can be executed given the current state
run - actually impacts the system

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

System and model

class List {
  data: number[] = [];
  push = (v: number) => this.data.push(v);
  pop = () => this.data.pop()!;
  size = () => this.data.length;
}

type Model = { num: number };

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

Push command

class PushCommand implements fc.Command<Model, List> {
  constructor(readonly value: number) {}
  check = (m: Readonly<Model>) => true;
  run(m: Model, r: List): void {
    r.push(this.value);
    ++m.num;
  }
  toString = () => `push(${this.value})`;
}

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

Pop command

class PopCommand implements fc.Command<Model, List> {
  check(m: Readonly<Model>): boolean {
    return m.num > 0;
  }
  run(m: Model, r: List): void {
    assert.equal(typeof r.pop(), 'number');
    --m.num;
  }
  toString = () => 'pop';
}

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

Size command

class SizeCommand implements fc.Command<Model, List> {
  check = (m: Readonly<Model>) => true;
  run(m: Model, r: List): void {
    assert.equal(r.size(), m.num);
  }
  toString = () => 'size';
}

source: https://github.com/dubzzz/fast-check/blob/master/documentation/Tips.md

Model Based Test setup

const allCommands = [
  fc.integer().map(v => new PushCommand(v)),
  fc.constant(new PopCommand()),
  fc.constant(new SizeCommand())
];

fc.assert(
  fc.property(fc.commands(allCommands, 100), cmds => {
    const s = () => ({ model: { num: 0 }, real: new List() });
    fc.modelRun(s, cmds);
  })
);