447 lines
18 KiB
Plaintext
447 lines
18 KiB
Plaintext
i3 testsuite
|
||
============
|
||
Michael Stapelberg <michael+i3@stapelberg.de>
|
||
September 2011
|
||
|
||
This document explains how the i3 testsuite works, how to use it and how to
|
||
extend it. It is targeted at developers who not necessarily have been doing
|
||
testing before or have not been testing in Perl before. In general, the
|
||
testsuite is not of interest for end users.
|
||
|
||
|
||
== Introduction
|
||
|
||
The i3 testsuite is a collection of files which contain testcases for various
|
||
i3 features. Some of them test if a certain workflow works correctly (moving
|
||
windows, focus behaviour, …). Others are regression tests and contain code
|
||
which previously made i3 crash or lead to unexpected behaviour. They then check
|
||
if i3 still runs (meaning it did not crash) and if it handled everything
|
||
correctly.
|
||
|
||
The goal of having these tests is to automatically find problems and to
|
||
automatically get a feel for whether a change in the source code breaks any
|
||
existing feature. After every modification of the i3 sourcecode, the developer
|
||
should run the full testsuite. If one of the tests does not pass (but fails),
|
||
the corresponding problem should be fixed (or, in some cases, the testcase has
|
||
to be modified). For every bugreport, a testcase should be written to test the
|
||
correct behaviour. Initially, it will fail, but after fixing the bug, it will
|
||
pass. This ensures (or increases the chance) that bugs which have been fixed
|
||
once will never be found again.
|
||
|
||
Also, when implementing a new feature, a testcase might be a good way to be
|
||
able to easily test if the feature is working correctly. Many developers will
|
||
test manually if everything works. Having a testcase not only helps you with
|
||
that, but it will also be useful for every future change.
|
||
|
||
== Implementation
|
||
|
||
For several reasons, the i3 testsuite has been implemented in Perl:
|
||
|
||
1. Perl has a long tradition of testing. Every popular/bigger Perl module which
|
||
you can find on CPAN will not only come with documentation, but also with
|
||
tests. Therefore, the available infrastructure for tests is comprehensive.
|
||
See for example the excellent http://search.cpan.org/perldoc?Test::More
|
||
and the referenced http://search.cpan.org/perldoc?Test::Tutorial.
|
||
|
||
2. Perl is widely available and has a well-working package infrastructure.
|
||
3. The author is familiar with Perl :).
|
||
|
||
Please do not start programming language flamewars at this point.
|
||
|
||
=== Mechanisms
|
||
|
||
==== Script: complete-run
|
||
|
||
The testcases are run by a script called +complete-run.pl+. It runs all
|
||
testcases by default, but you can be more specific and let it only run one or
|
||
more testcases. Also, it takes care of starting up a separate instance of i3
|
||
with an appropriate configuration file and creates a folder for each run
|
||
containing the appropriate i3 logfile for each testcase. The latest folder can
|
||
always be found under the symlink +latest/+. It is recommended that you run the
|
||
tests on one or more separate X server instances (you can only start one window
|
||
manager per X session), for example using the provided Xdummy script.
|
||
+complete-run.pl+ takes one or more X11 display specifications and parallelizes
|
||
the testcases appropriately:
|
||
|
||
.Example invocation of complete-run.pl+
|
||
---------------------------------------
|
||
$ cd ~/i3/testcases
|
||
|
||
# start two dummy X11 instances in the background
|
||
$ ./Xdummy :1 &
|
||
$ ./Xdummy :2 &
|
||
|
||
$ ./complete-run.pl -d :1,:2
|
||
# output omitted because it is very long
|
||
All tests successful.
|
||
Files=78, Tests=734, 27 wallclock secs ( 0.38 usr 0.48 sys + 17.65 cusr 3.21 csys = 21.72 CPU)
|
||
Result: PASS
|
||
|
||
$ ./complete-run.pl -d :1 t/04-floating.t
|
||
[:3] i3 startup: took 0.07s, status = 1
|
||
[:3] Running t/04-floating.t with logfile testsuite-2011-09-24-16-06-04-4.0.2-226-g1eb011a/i3-log-for-04-floating.t
|
||
[:3] t/04-floating.t finished
|
||
[:3] killing i3
|
||
output for t/04-floating.t:
|
||
ok 1 - use X11::XCB::Window;
|
||
ok 2 - The object isa X11::XCB::Window
|
||
ok 3 - Window is mapped
|
||
ok 4 - i3 raised the width to 75
|
||
ok 5 - i3 raised the height to 50
|
||
ok 6 - i3 did not map it to (0x0)
|
||
ok 7 - The object isa X11::XCB::Window
|
||
ok 8 - i3 let the width at 80
|
||
ok 9 - i3 let the height at 90
|
||
ok 10 - i3 mapped it to x=1
|
||
ok 11 - i3 mapped it to y=18
|
||
ok 12 - The object isa X11::XCB::Window
|
||
ok 13 - i3 let the width at 80
|
||
ok 14 - i3 let the height at 90
|
||
1..14
|
||
|
||
All tests successful.
|
||
Files=1, Tests=14, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.19 cusr 0.03 csys = 0.23 CPU)
|
||
Result: PASS
|
||
|
||
$ less latest/i3-log-for-04-floating.t
|
||
----------------------------------------
|
||
|
||
==== IPC interface
|
||
|
||
The testsuite makes extensive use of the IPC (Inter-Process Communication)
|
||
interface which i3 provides. It is used for the startup process of i3, for
|
||
terminating it cleanly and (most importantly) for modifying and getting the
|
||
current state (layout tree).
|
||
|
||
See [http://i3wm.org/docs/ipc.html] for documentation on the IPC interface.
|
||
|
||
==== X11::XCB
|
||
|
||
In order to open new windows, change attributes, get events, etc., the
|
||
testsuite uses X11::XCB, a new (and quite specific to i3 at the moment) Perl
|
||
module which uses the XCB protocol description to generate Perl bindings to
|
||
X11. They work in a very similar way to libxcb (which i3 uses) and provide
|
||
relatively high-level interfaces (objects such as +X11::XCB::Window+) aswell as
|
||
access to the low-level interface, which is very useful when testing a window
|
||
manager.
|
||
|
||
=== Filesystem structure
|
||
|
||
In the git root of i3, the testcases live in the folder +testcases+. This
|
||
folder contains the +complete-run.pl+ and +Xdummy+ scripts and a base
|
||
configuration file which will be used for the tests. The different testcases
|
||
themselve can be found in the conventionally named subfolder +t+:
|
||
|
||
.Filesystem structure
|
||
--------------------------------------------
|
||
├── testcases
|
||
│ ├── complete-run.pl
|
||
│ ├── i3-test.config
|
||
│ ├── t
|
||
│ │ ├── 00-load.t
|
||
│ │ ├── 01-tile.t
|
||
│ │ ├── 02-fullscreen.t
|
||
│ │ ├── ...
|
||
│ │ ├── omitted for brevity
|
||
│ │ ├── ...
|
||
│ │ ├── 74-regress-focus-toggle.t
|
||
│ │ └── lib
|
||
│ │ └── i3test.pm
|
||
│ └── Xdummy
|
||
--------------------------------------------
|
||
|
||
== Anatomy of a testcase
|
||
|
||
Learning by example is definitely a good strategy when you are wondering how to
|
||
write a testcase. Let's take +t/11-goto.t+ as an easy example and go through it
|
||
step by step:
|
||
|
||
.t/11-goto.t: Boilerplate
|
||
----------------------
|
||
#!perl
|
||
# vim:ts=4:sw=4:expandtab
|
||
|
||
use i3test;
|
||
use File::Temp;
|
||
|
||
my $x = X11::XCB::Connection->new;
|
||
-----------------------
|
||
|
||
This is what we call boilerplate. It exists at the top of every test file (to
|
||
some extent). The first line is the shebang, which specifies that this file is
|
||
a Perl script. The second line contains VIM specific settings on how to
|
||
edit/format this file (use spaces instead of tabs, indent using 4 spaces).
|
||
Afterwards, the +i3test+ module is used. This module contains i3 testsuite
|
||
specific functions which you are strongly encouraged to use. They make writing
|
||
testcases a lot easier and will make it easier for other people to read your
|
||
tests.
|
||
|
||
The next line uses the +File::Temp+ module. This is specific to this testcase,
|
||
because it needs to generate a temporary name during the test. Many testcases
|
||
use only the +i3test+ module.
|
||
|
||
The last line opens a connection to X11. You might or might not need this in
|
||
your testcase, depending on whether you are going to open windows (etc.) or
|
||
only use i3 commands.
|
||
|
||
.t/11-goto.t: Setup
|
||
----------------------
|
||
my $tmp = fresh_workspace;
|
||
|
||
cmd 'split h';
|
||
----------------------
|
||
|
||
The first line calls i3test's +fresh_workspace+ function which looks for a
|
||
currently unused workspace, switches to it, and returns its name. The variable
|
||
+$tmp+ will end up having a value such as +"/tmp/87kBVcHbA9"+. Note that this
|
||
is not (necessarily) a valid path, it's just a random workspace name.
|
||
|
||
So, now that we are on a new workspace, we ensure that the workspace uses
|
||
horizontal orientation by issuing the +split h+ command (see the i3 User's
|
||
Guide for a list of commands). This is not strictly necessary, but good style.
|
||
In general, the +cmd+ function executes the specified i3 command by using the
|
||
IPC interface and returns once i3 acknowledged the command.
|
||
|
||
.t/11-goto.t: Setup
|
||
----------------------
|
||
#####################################################################
|
||
# Create two windows and make sure focus switching works
|
||
#####################################################################
|
||
|
||
my $top = open_window($x);
|
||
my $mid = open_window($x);
|
||
my $bottom = open_window($x);
|
||
----------------------
|
||
|
||
In every major section of a testcase, you should put a comment like the one
|
||
above. This makes it immediately clear how the file is structured.
|
||
|
||
The +open_window+ function opens a standard window, which will then be put into
|
||
tiling mode by i3. If you want a floating window, use the
|
||
+open_floating_window+ function. These functions accept the same parameters as
|
||
+X11::XCB::Window->new+, see the i3test documentation at TODO.
|
||
|
||
.t/11-goto.t: Helper function
|
||
----------------------
|
||
#
|
||
# Returns the input focus after sending the given command to i3 via IPC
|
||
# end sleeping for half a second to make sure i3 reacted
|
||
#
|
||
sub focus_after {
|
||
my $msg = shift;
|
||
|
||
cmd $msg;
|
||
sync_with_i3 $x;
|
||
return $x->input_focus;
|
||
}
|
||
----------------------
|
||
|
||
This section defines a helper function which will be used over and over in this
|
||
testcase. If you have code which gets executed more than once or twice
|
||
(depending on the length of your test, use your best judgement), please put it
|
||
in a function. Tests should be short, concise and clear.
|
||
|
||
The +focus_after+ function executes a command and returns the X11 focus after
|
||
the command was executed. The +sync_with_i3+ command makes sure that i3 could
|
||
push its state to X11. See <<i3_sync>> to learn how this works exactly.
|
||
|
||
.t/11-goto.t: Test assumptions
|
||
----------------------
|
||
$focus = $x->input_focus;
|
||
is($focus, $bottom->id, "Latest window focused");
|
||
|
||
$focus = focus_after('focus left');
|
||
is($focus, $mid->id, "Middle window focused");
|
||
----------------------
|
||
|
||
Now, we run the first two real tests. They use +Test::More+'s +is+ function,
|
||
which compares two values and prints the differences if they are not the same.
|
||
After the arguments, we supply a short comment to indicate what we are testing
|
||
here. This makes it vastly more easy for the developer to spot which testcase
|
||
is the problem in case one fails.
|
||
|
||
The first test checks that the most recently opened window is focused.
|
||
Afterwards, the command +focus left+ is issued and it is verified that the
|
||
middle window now has focus.
|
||
|
||
Note that this is not a comprehensive test of the +focus+ command -- we would
|
||
have to test wrapping, focus when using a more complex layout, focusing the
|
||
parent/child containers, etc. But that is not the point of this testcase.
|
||
Instead, we just want to know if +$x->input_focus+ corresponds with what we are
|
||
expecting. If not, something is completely wrong with the test environment and
|
||
this trivial test will fail.
|
||
|
||
.t/11-goto.t: Test that the feature does not work (yet)
|
||
----------------------
|
||
#####################################################################
|
||
# Now goto a mark which does not exist
|
||
#####################################################################
|
||
|
||
my $random_mark = mktemp('mark.XXXXXX');
|
||
|
||
$focus = focus_after(qq|[con_mark="$random_mark"] focus|);
|
||
is($focus, $mid->id, "focus unchanged");
|
||
----------------------
|
||
|
||
In this new major section, a random mark (mark is an identifier for a window,
|
||
see "VIM-like marks" in the i3 User’s Guide) will be generated. Afterwards, we
|
||
test that trying to focus that mark will not do anything. This is important: Do
|
||
not only test that using a feature has the expected outcome, but also test that
|
||
using it without properly initializing it does no harm. This command could for
|
||
example have changed focus anyways (a bug) or crash i3 (obviously a bug).
|
||
|
||
.t/11-goto.t: Test that the feature does work
|
||
----------------------
|
||
cmd "mark $random_mark";
|
||
|
||
$focus = focus_after('focus left');
|
||
is($focus, $top->id, "Top window focused");
|
||
|
||
$focus = focus_after(qq|[con_mark="$random_mark"] focus|);
|
||
is($focus, $mid->id, "goto worked");
|
||
----------------------
|
||
|
||
Remember: Focus was on the middle window (we verified that earlier in "Test
|
||
assumptions"). We now mark the middle window with our randomly generated mark.
|
||
Afterwards, we switch focus away from the middle window to be able to tell if
|
||
focusing it via its mark will work. If it does work (next test), the goto
|
||
command works.
|
||
|
||
.t/11-goto.t: Test corner case
|
||
----------------------
|
||
# check that we can specify multiple criteria
|
||
|
||
$focus = focus_after('focus left');
|
||
is($focus, $top->id, "Top window focused");
|
||
|
||
$focus = focus_after(qq|[con_mark="$random_mark" con_mark="$random_mark"] focus|);
|
||
is($focus, $mid->id, "goto worked");
|
||
----------------------
|
||
|
||
Now we test the same feature, but specifying the mark twice in the command.
|
||
This should have no effect, but let’s be sure: test it and see if things go
|
||
wrong.
|
||
|
||
.t/11-goto.t: Test second code path
|
||
----------------------
|
||
#####################################################################
|
||
# Check whether the focus command will switch to a different
|
||
# workspace if necessary
|
||
#####################################################################
|
||
|
||
my $tmp2 = fresh_workspace;
|
||
|
||
is(focused_ws(), $tmp2, 'tmp2 now focused');
|
||
|
||
cmd qq|[con_mark="$random_mark"] focus|;
|
||
|
||
is(focused_ws(), $tmp, 'tmp now focused');
|
||
----------------------
|
||
|
||
This part of the test checks that focusing windows by mark works across
|
||
workspaces. It uses i3test's +focused_ws+ function to get the current
|
||
workspace.
|
||
|
||
.t/11-goto.t: Test second code path
|
||
----------------------
|
||
done_testing;
|
||
----------------------
|
||
|
||
The end of every testcase has to contain the +done_testing+ line. This tells
|
||
+complete-run.pl+ that the test was finished successfully. If it does not
|
||
occur, the test might have crashed during execution -- some of the reasons why
|
||
that could happen are bugs in the used modules, bugs in the testcase itself or
|
||
an i3 crash resulting in the testcase being unable to communicate with i3 via
|
||
IPC anymore.
|
||
|
||
[[i3_sync]]
|
||
== Appendix A: The i3 sync protocol
|
||
|
||
Consider the following situation: You open two windows in your testcase, then
|
||
you use +focus left+ and want to verify that the X11 focus has been updated
|
||
properly. Sounds simple, right? Let’s assume you use this straight-forward
|
||
implementation:
|
||
|
||
.Racey focus testcase
|
||
-----------
|
||
my $left = open_window($x);
|
||
my $right = open_window($x);
|
||
cmd 'focus left';
|
||
is($x->input_focus, $left->id, 'left window focused');
|
||
----------
|
||
|
||
However, the test fails. Sometimes. Apparantly, there is a race condition in
|
||
your test. If you think about it, this is because you are using two different
|
||
pieces of software: You tell i3 to update focus, i3 confirms that, and then you
|
||
ask X11 to give you the current focus. There is a certain time i3 needs to
|
||
update the X11 state. If the testcase gets CPU time before X11 processed i3's
|
||
requests, the test will fail.
|
||
|
||
image::i3-sync.png["Diagram of the race condition", title="Diagram of the race condition"]
|
||
|
||
One way to "solve" this would be to add +sleep 0.5;+ after the +cmd+ call.
|
||
After 0.5 seconds it should be safe to assume that focus has been updated,
|
||
right?
|
||
|
||
In practice, this usually works. However, it has several problems:
|
||
|
||
1. This is obviously not a clean solution, but a workaround. Ugly.
|
||
2. On very slow machines, this might not work. Unlikely, but in different
|
||
situations (a delay to wait for i3 to startup) the necessary time is much
|
||
harder to guess, even for fast machines.
|
||
3. This *wastes a lot of time*. Usually, your computer is much faster than 0.5s
|
||
to update the status. However, sometimes, it might take 0.4s, so we can’t
|
||
make it +sleep 0.1+.
|
||
|
||
To illustrate how grave the problem with wasting time actually is: Before
|
||
removing all sleeps from the testsuite, a typical run using 4 separate X
|
||
servers took around 50 seconds on my machine. After removing all the sleeps,
|
||
we achieved times of about 25 seconds. This is very significant and influences
|
||
the way you think about tests -- the faster they are, the more likely you are
|
||
to check whether everything still works quite often (which you should).
|
||
|
||
What I am trying to say is: Delays adds up quickly and make the test suite
|
||
less robust.
|
||
|
||
The real solution for this problem is a mechanism which I call "the i3 sync
|
||
protocol". The idea is to send a request (which does not modify state) via X11
|
||
to i3 which will then be answered. Due to the request's position in the event
|
||
queue (*after* all previous events), you can be sure that by the time you
|
||
receive the reply, all other events have been dealt with by i3 (and, more
|
||
importantly, X11).
|
||
|
||
image::i3-sync-working.png["Diagram of the i3 sync solution", title="Diagram of the i3 sync solution"]
|
||
|
||
=== Implementation details
|
||
|
||
The client which wants to sync with i3 initiates the protocol by sending a
|
||
ClientMessage to the X11 root window:
|
||
|
||
.Send ClientMessage
|
||
-------------------
|
||
# Generate a ClientMessage, see xcb_client_message_t
|
||
my $msg = pack "CCSLLLLLLL",
|
||
CLIENT_MESSAGE, # response_type
|
||
32, # format
|
||
0, # sequence
|
||
$root, # destination window
|
||
$x->atom(name => 'I3_SYNC')->id,
|
||
|
||
$_sync_window->id, # data[0]: our own window id
|
||
$myrnd, # data[1]: a random value to identify the request
|
||
0,
|
||
0,
|
||
0;
|
||
|
||
# Send it to the root window -- since i3 uses the SubstructureRedirect
|
||
# event mask, it will get the ClientMessage.
|
||
$x->send_event(0, $root, EVENT_MASK_SUBSTRUCTURE_REDIRECT, $msg);
|
||
-------------------
|
||
|
||
i3 will then reply with the same ClientMessage, sent to the window specified in
|
||
+data[0]+. In the reply, +data[0]+ and +data[1]+ are exactly the same as in the
|
||
request. You should use a random value in +data[1]+ and check that you received
|
||
the same one when getting the reply.
|
||
|
||
== Appendix B: Socket activation
|