Federation Tests

With the testsuite you can write simple unit tests ensuring that your server is actually capable of sending/receiving federated payloads.

Maybe you want to checkout the introduction post before continuing!

Vocabulary

  • Testsuite: TS
  • Testsuite Server: TSS
  • Merge-Request: MR
  • Pull-Request: PR
  • Bash Automated Testing System: BATS

Workflow

testsuite-and-server.png

  1. MR was created
    • Webhook is triggered (to testsuite.feneas.org/hook)
  2. Testsuite Server
    • Parsing and validating Webhook
    • Triggering new TS build
  3. Testsuite
    • Running tests for specified project
    • Reporting results to TSS
  4. Testsuite Server
    • Evaluate and store test results
    • Reporting results to MR

Testsuite

Say we have a project called testproject and it can communicate with other testproject instances. If we have two servers running on 172.17.0.4:3000 and 172.17.0.5:3000 we can send 172.17.0.4 a simple message by executing following command:

curl http://172.17.0.5:3000/add@172.17.0.4:3000

If we ask the server directly whether it received a message or not:

curl http://172.17.0.4:3000

it should return one, otherwise it will return zero!

Getting started

You can answer following questions with the testsuite now:

  1. can we send a message via 172.17.0.5
  2. was the message received on 172.17.0.4

Dependencies

In case you want to test specific parts locally you need docker. Make sure it is up and running:

sudo systemctl start docker.service

Then you can install the testsuite dependencies:

  • github.com/stedolan/jq
  • github.com/sstephenson/bats

Simply run the install script or do it manually:

bash scripts/install.sh

Docker Image

Before we can start testing we need a docker image and a custom entrypoint. You can use your own Dockerfile but we need a special entrypoint in case a user creates a new PR via GitHub or MR via GitLab! Then the image will start running with special variables.

In our case it will set PRREPO to the repository URL of the user and PRSHA to the commit SHA to be tested with the suite.

Start by cloning/forking the testsuite first:

git clone https://git.feneas.org/feneas/federation/testsuite.git

Switch into the repository and create a new folder called testproject:

cd testsuite
mkdir testproject
cd testproject

Environment

Following environment variables will be available while tests are running:

$DATABASE      (on every run)
$PORT          (on every run)
$PROJECT       (only on pull_requests from a user)
$PRREPO        (only on pull_requests from a user)
$PRSHA         (only on pull_requests from a user)
$(hostname -i) (if you need the container IP address)

Dockerfile

For our testproject that could look something like this:

FROM debian:jessie

RUN apt-get update
RUN apt-get install -y \
  git-core postgresql-client netcat curl

RUN git clone https://git.feneas.org/feneas/federation/testsuite-testproject.git
WORKDIR testsuite-testproject

ADD entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh

Now for our special entrypoint you can see it will change the source branch if a custom build is used:

#!/bin/bash

if [ "$PROJECT" == "testproject" ]; then
  git checkout . \
    && git remote add custom $PRREPO \
    && git fetch custom \
    && git merge -m merge $PRSHA \
    && git log -1 || {
      echo "Cannot pull from repo=$PRREPO sha=$PRSHA"
      exit 1
    }
fi

./testproject.sh $DATABASE postgreshost $PORT

Building

Since you do not have permission to upload images to the feneas namespace, create a MR with your changes here! If you want to test locally proceed..

Last final step before we have a fully functional docker image is to build the actual image:

docker build --no-cache -t feneas/testsuite_testproject:v1.0.3-testproject .

The testsuite requires a special naming for every image: feneas/testsuite_<project>:<version>-<project>

If we succesfully build the docker image we have to commit our changes and create a project tag and we are done with the image part:

cd <workspace>/testsuite/testproject
git add .
git commit -m "Add testproject docker image"
git tag v1.0.3-testproject

Testing

After we created a docker image we can start testing to ensure testproject is actually capable of sending/receiving mentioned messages.

For this we switch in our testsuite again and create a test file called testproject.bats:

cd <workspace>/testsuite
vi testproject.bats

I suggest we start with loading the test helper first since it holds many helper functions which can come in handy:

# vim:ft=sh

load test_helper

Since we need two instances of our testproject we have to create two postgresql databases first:

@test "$btf create databases" {
  for tp in tp1 tp2; do
    create_database $tp
    [ "$?" -eq 0 ]
  done
}

and start-up the mentioned servers:

@test "$btf start testproject#1 server" {
  start_app tp1 3000 testproject$(latest_tag testproject)
  [ "$?" -eq 0 ]
}

@test "$btf start testproject#2 server" {
  start_app tp2 3000 testproject$(latest_tag testproject)
  [ "$?" -eq 0 ]
}

Now the interesting part starts.. We want to test a request to tp2! If we execute the curl command (mentioned at the beginning of this document) tp2 should federated/relay the message to tp1.

To prove this we send two requests:

  1. to tp2 requesting the message to be forwarded
  2. to tp1 ensuring the message was received
@test "$btf increase count on tp1 via tp2" {
  get "http://$(container_ip tp2):3000/add@$(container_ip tp1):3000"
  [ "$HTTP_STATUS_CODE" == "200" ]
}

@test "$btf test count on tp1, it should be one" {
  get "http://$(container_ip tp1):3000"
  echo "expected 200, got $HTTP_STATUS_CODE"
  [ "$HTTP_STATUS_CODE" == "200" ]
  echo "expected 1, got $HTTP_BODY"
  [ "$HTTP_BODY" == "1" ]
}

Last but not least we do some cleaning and we are done with our first unit test for the federated web \m/

@test "$btf stop and delete the containers" {
  for tp in tp1 tp2; do
    stop_app $tp
    [ "$?" -eq 0 ]
    remove_app $tp
    [ "$?" -eq 0 ]
  done
}

@test "$btf drop databases" {
  for tp in tp1 tp2; do
    drop_database $tp
    [ "$?" -eq 0 ]
  done
}

You will find the full version of testproject.bats here!

That should be it! You can add and commit your changes and request a MR upstream:

git add testproject.bats
git commit -m "Add testproject.bats"

Run BATS

If you installed the dependencies from scripts/install.sh locally you can run the testsuite by executing:

cd <workspace>/testsuite
bats testproject.bats
 ✓ testproject.bats create databases
 ✓ testproject.bats start testproject#1 server
 ✓ testproject.bats start testproject#2 server
 ✓ testproject.bats test initial count, it should be zero
 ✓ testproject.bats increase count on tp1 via tp2
 ✓ testproject.bats test count on tp1, it should be one
 ✓ testproject.bats check the database on tp1 too
 ✓ testproject.bats stop and delete the containers
 ✓ testproject.bats drop databases

9 tests, 0 failures

Thats it ! :)


Sometimes doing tests manually helps developing a new feature. You can run single docker images by using the helper script. Setup the environment by executing following command once:

BATS_TEST_FILENAME=local . ./test_helper.bash

Then starting can be done by executing:

# start_app <database-name> <port> "<project>"$(latest_tag <project>)
# for starting a ganggo image that could look like following:
start_app tp1 3000 testproject$(latest_tag testproject)

Testsuite Server

Why and how do I integrate my project with the server?

Assuming we have some unit tests for your project in the testsuite repository you can authenticate your personal project against our integration service. Then every time someone creates a merge request an automatic build is triggered and the testsuite will verify that the unit tests still succeed.

In GitLab that would look something like this:

gitlab-pipeline

An external pipeline will be added and it will update the status whether tests are failing or not.

The same applies for GitHub:

github-checks

So how can you add your project?

Simply visit testsuite.feneas.org and click on
gitlab-or-github
The GitLab integration also supports self-hosted instances \m/

If you clicked on the buttons follow the instructions and as soon as you hit Submit your project should be listed on the index page of testsuite.feneas.org.

Now the testing can begin :)