Running your first ATS

Assuming you have installed pathways and the examples (see: Installing Pathways), you can run a quick demo:

[albert@pathways:]% cd Pathways/examples/calculators/webapp/tst
[albert@pathways:.../webapp/tst]% pw ATS/demo.py --vector ../../shared/tst-data/telop.csv
Final Test result: OK; after 1 ATS and 1 test
[albert@pathways:.../webapp/tst]%

It just works …

During this test you will see (all very quickly) that the (Firefox) browser is opened, a very simple web-calculator is loaded, and the default user is logged-in. Then several additions are done where each results are verified; that is the core of this test. Finally, the user is logged-out to end the session and the browser is closed. Note: On some systems, you may see a (security) pop-up, in which you should confirm it is allowed this python-application acts as server.

You can simply change this test, by using another vector. Copy the file telop.csv (dutch voor addition) and add some rows, or change the current numbers. You will find the format easy to understand: two numbers (integers only), an empty column to separate the feed (inputs) from the expected results: the sum of those two numbers. You can edit the file with any editor, or a spreadsheet-programma, as long as you save it in plain csv! Save the file and start the test again with your vector (say myfirst.csv):

[albert@pathways:.../webapp/tst]% # Specify the path+file to your own file
[albert@pathways:.../webapp/tst]% pw ATS/demo.py --vector myfirst.csv

Fake a bug

Now edit the file again, and put a wrong anser in a row; an example is given and the end of this section. This simulates a bug in the calculator; but is a lot easier. And start the test again.:

[albert@pathways:.../webapp/tst]% pw ATS/demo.py --vector buggy.csv
      adding (2, 5) => got 7, expected: 9 -- see log for trace and details
Final Test result: FAILED (See log for details); after 1 ATS and 1 test

As expected, the test will fail. The exact message(s) may differ, but it will report the final-test-result as FAILED. Usually, it is also show the operation that found the flow. Here it is expecting 2+5 results in 9, but got 7. Surely, this is due to the buggy test-vector, but you got the idea.

While the test failed, and to make it possible (for programmers) to debug it, the PUT (the PUT is here the web-calculator)` is usually not closed. It typically stops right where the failure occurred! This however depends on the PUT, and how it is configured. For now, just close the browser (of play around with this simple app, to get a felling for it)

As the output announces there is more information available: each test-run adds log information to the file -Pathways.log; (yes, it starts with a dash). This file contains information of every step, of every test, for all runs (succesful or not). The amount of information can be configured with the log option. You may (and should) remove those log-files on regulair base, or your disk may fill up quickly. Typically, I remove it after a series of succesful-test runs, or just before I want to dive into the details.

Read the logs

For now, remove the logfile and rerun the failing test with lots of logging:

[albert@pathways:.../webapp/tst]% rm ./-Pathways # Use ‘./’ prefix to select the file!
[albert@pathways:.../webapp/tst]% pw ATS/demo.py --vector buggy.csv --log=DEBUG

Again the test will fail. Now open the logfile. Even for this simple setup it will contain only around 200 lines, depending on the length of the buggy-test-vector and the line you put-in the fake expected value. As can be expected with DEBUG, a lot of details is shown.Every tiny step, in the pathway-framework itself and in the libraries it uses, leave a trail. By example, you will find many lines labeled with remote_connection. This lines show in details how selenium talks with the browser; for now you can ignore them. As the lines labeled with pathways.puts.webput abstract that a bit; on those lines you can follow the communication between pathway’s and the PUT.

You will find lines the lines we need near the end of the file. For every failing ATStest a traceback is shown – just in case you may need it. But again, ignore it for now, the lines around it satisfy to signal the failure-cause:

## 2016-05-16 00:41:40,199     DEBUG::pathways.puts.webput           webput[199]read          # Read: 7, from display
## 2016-05-16 00:41:40,199      INFO::plugin.cobblestones.webMath   webMath[037]__calc        # put=RekenApp[1013db160]; op=+ feed=(2, 5) =>7
## 2016-05-16 00:41:40,199     ERROR::Pathways::.../bin/pw              run[043]run_ATS       # ATS FAILED (AssertionError): adding (2, 5) => got 7, expected: 9
Traceback (most recent call last):
  File ".../pathways/runners/run.py", line 37, in run_ATS
    status = test_fixer.run_with_fixtures()
  File ".../pathways/core/annex.py", line 239, in run_with_fixtures
    rv = self._ats(**parms)
  File ".../examples/calculators/webapp/tst/ATS/demo.py", line 46, in ATS_add
    assert actual == expect[0], "adding %s => got %s, expected: %s" % (feed, actual, expect[0])
AssertionError: adding (2, 5) => got 7, expected: 9
## 2016-05-16 00:41:40,201  CRITICAL::Pathways::.../bin/pw              run[044]run_ATS       #       adding (2, 5) => got 7, expected: 9 -- see log for trace and details
## 2016-05-16 00:41:40,201  CRITICAL::Pathways::.../bin/pw               pw[052]main          # Final Test result: FAILED (See log for details); after 1 ATS and 1 test

It shows that pathways read 7 from the ‘display’, after it did the ‘+ operation’ on 2 and 5. Normally that is correct, but the (buggy) test specified it should be 9. And so, the test FAILED.

Fix the bug

In daily use the DEBUG-level is usually not needed. As you can see in the example above, the same flaw can be found with the INFO-level. And to find failing tests even as ‘CRITICAL’ level will do. There are several ways to reduce the number of loglines. For now, you can simply grep '::[Pp]athways' to find all message from the framework (lowercase pathways) and the ATSes (uppercase Pathways).

Now, rerun the test with a lower log-level and find the same flow:

[albert@pathways:.../webapp/tst]% rm ./-Pathways
[albert@pathways:.../webapp/tst]% pw ATS/demo.py --vector buggy.csv --log=INFO # or: CRITICAL

Finally, fix the flaw! And prove it is corrected by retesting it; repeat when needed until the test is OK.

The ATS explained

This demo-test verifies the (web) calculator can add-up two numbers. The ATS is only a handfull of lines and a few, short, reusable fixture functions. This passage shows the full file, part by part; introducing the basic concepts. Details are explained later.

The ATS itself

Each ATStest is function 1 which starts with ‘ATS_’ 2. They are (typically) automatically run; in “file order”. The ATSfile demo.py contains one ATStest:

1
2
3
4
5
6
7
8
9
def ATS_add(calcApp, tstvectors):
    """A basic test to add 2 numbers and verify the result (as demo)"""

    kb = calcApp.widget('Keyboard')

    for feed, expect in tstvectors:
        kb.clear()
        actual = calcApp.add(feed[0],feed[1])
        assert actual == expect[0], "adding %s => got %s, expected: %s" % (feed, actual, expect[0])

ATS_add() is basically a for-loop over all test-vectors. For each row the two input-values are feed to the calcApp and the returned (actuals) value is compared to the (single) expected one. When they are :not equal, the test is aborted (with the assert statement). Before each calculation, the screen is :cleared; as we need that button several times, the keyboard “widget” is searched only once; before the loop is :started.

As you can see, the ATS focus on the “adding” and “calculators”. The technical details of how the web-calculator works, how “buttons” are implemented, and how to enter (long) numbers is abstracted in generic modules. This make the ATS highly reusable and stable for all kind of changes.

Fixtures & Bricks

There are several kind of generics. The brick is the most used once; but not needed for this trivial example.

The other (main) generalisation, fixtures, are used: this are functions to setup and/or teardown a test. They are frequently used for many tests. And ‘selected’ by passing them as arguments to the ATStest.

Setup fixtures

A setup-fixtures is (automatically) called before the ATS is run. It are functions decorated with @pathways.fixtures.Setup. The returned setup-value is passed as parameter to the ATS.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
@pathways.fixtures.Setup
def calcApp(cfg):                      ## NOTE: `cfg` is a 'build-in annex'
    """Connect to the selected reken app. And login"""

    Gate, put = cfg['gate'], cfg['put']
    calculator = Gate(put) # connect to the PUT, using the selected GATE (a plugin)

    calculator.loginOK()

    return calculator

First, the gate to interface with the PUT and the adress of the “put” itself are read from the configuration. Then both are used to connect to the calculator; in the demo above this will fire-up your browser and load the js_calc.html page, as specified in the config.cfg file. Then a standard account is used to login 4. Last, the now ready calculator is returned.

This fixture is generic: it can be used in many ATStests. Typically these generic fixture are defined in a separate file and imported. Here it is part of the demo.py, to show a complete test in one file.

Also the second setup is generic and quite readable.

TstData fixtures

A testData-fixture is a bit like a setup-fixture, but prepares the test data (and/or test-vector). It’s a function decorated with @pathways.fixture.TstData. It typically returns a TstVectors

1
2
3
4
5
6
7
8
@pathways.fixtures.TstData
def tstvectors(cfg):
    """Read the feed and expected values from a csv file
       FORMAT:    a,b,,c -- where c === a+b """

    filename = cfg['vector']
    data = pathways.TstVectors.from_csv(filename, castTo=int)
    return data

Again, the configuration is read. Now, to find file containing the test-data; it is the file specified as argument above. That csv-formated file is read, converted to integers, and passed to the ATS.

Note that the setup-fixtures are selected by the ATS. By it arguments, or by the files that are imported. So when a test needs other data (eg floating-point numbers, or read from another source) that is trivial to do: write one similar setup-fixture and use that for all those tests.

Teardown fixtures

A teardown-fixture is automatically called after the ATS (when mentioned as argument). A stand-alone teardown-fixture should be decorated with @pathways.fixtures.TearDown. However, as a setup typically also has a teardown those two can be combined; then they are mentioned only once as ATS-argument. For combined-fixtures, the setup is defined normally. After that, the teardown is defined, with same function-name and decorated with @<FuncName>.Teardown; using the Teardown() method which is “added” to the fixture defined earlier 3. An example of this is shown below:

1
2
3
4
5
6
@calcApp.Teardown
def calcApp(calcApp):
    """Logoff and disconnect"""

    calcApp.logoutOK()
    calcApp.close()

After the test in run, the user is logged-out and the connection to the webapp is closed. In the demo above; this last method closes the browser.

Test Vectors

The essential data for an ATS is often stored in a csv-file. Notice the empty column between the feed and the expected results.

The demo above used the following file. You can change/extend the file to get a better test. Without coding any line!

vector example (used in the demo above)

# (C) Albert Mietus

Part of

Pathways project

Use at will

#feed

;

expected

2

5

7

1

6

7

0

7

7

Simulating a bug

This is an example of a testvector with a flaw, to simulate a failing test.

Buggy vector example

#a

b

;

som

1

6

7

2

5

9

#bug

0

7

7

footnotes

1

Strictly speaking: any callable is allowed.

2

This ATS_ prefix is needed for the auto-run feature. All callables starting with this prefix (in an ATSfile) are automatically discovered as ATStest

3

Reversing the order of the setup- and teardown-fixture is possible too. When the teardown is defined “above” the setup, that one is defined normally. Always the first one should use the @pathways.fixture prefixed decorator and the second the <FixtureName> one, both with the correct Teardown or Setup method.

4

The OK suffix is a hint this function will abort when the login does fail. As verifying the login-feature is not part of the objective of this ATS; it is assumed that will work. Other ATSes will test that, using more basic functions.