Build Script Interaction with TeamCity - Reporting Tests

Reporting Tests to TeamCity

To use the TeamCity on-the-fly test reporting, a testing framework needs dedicated support for this feature to work (alternatively, XML Report Processing can be used).
If TeamCity doesn't support your testing framework natively, it is possible to modify your build script to report test runs to the TeamCity server using service messages. This makes it possible to display test results in real-time, make test information available on the Tests tab of the Build Results page.

Supported test service messages

Test suite messages:
Test suites are used to group tests. TeamCity displays tests grouped by suites on Tests tab of the Build Results page and in other places.

##teamcity[testSuiteStarted name='suiteName'] <individual test messages go here> ##teamcity[testSuiteFinished name='suiteName']

All the individual test messages are to appear between testSuiteStarted and testSuiteFinished (in that order) with the same name attributes.
Suites may also be nested.

Test start/stop messages:

##teamcity[testStarted name='testName' captureStandardOutput='<true/false>'] <here go all the test service messages with the same name> ##teamcity[testFinished name='testName' duration='<test_duration_in_milliseconds>']

Indicates that the test "testName" was run. If the testFailed message is not present, the test is regarded successful.

duration (optional numeric attribute) - sets the test duration in milliseconds (should be an integer) to be reported in TeamCity UI. If omitted, the test duration will be calculated from the messages timestamps. If the timestamps are missing, from the actual time the messages were received on the server.
captureStandardOutput (optional boolean attribute) - if true, all the standard output (and standard error) messages received between testStarted and testFinished messages will be considered test output. The default value is false and assumes usage of testStdOut and testStdErr service messages to report the test output.

Icon

• All the other test messages (except for testIgnored) with the same name attribute should appear between the testStarted and testFinished messages (in that order).
• Currently, the test-related service messages cannot be output with Ant's echo task until flowId attribute is specified.

It is highly recommended to ensure that the pair of test suite + test name is unique within the build.
For advanced TeamCity test-related features to work, test names should not deviate from one build to another (a single test must be reported under the same name in every build). Include absolute paths in the reported test names is strongly discouraged.

Ignored tests:

##teamcity[testIgnored name='testName' message='ignore comment']

Indicates that the test "testName" is present but was not run (was ignored) by the testing framework.
As an exception, the testIgnored message can be reported without the matching testStarted and testFinished messages.

Test output:

##teamcity[testStarted name='className.testName'] ##teamcity[testStdOut name='className.testName' out='text'] ##teamcity[testStdErr name='className.testName' out='error text'] ##teamcity[testFinished name='className.testName' duration='50']

The testStdOut and testStdOrr service messages report the test's standard and error output to be displayed in the TeamCity UI. There must be only one testStdOut and one testStdErr message per test.
An alternative but a less reliable approach is to use the captureStandardOutput attribute of the testStarted message.

Test result:

##teamcity[testStarted name='MyTest.test1'] ##teamcity[testFailed name='MyTest.test1' message='failure message' details='message and stack trace'] ##teamcity[testFinished name='MyTest.test1']   ##teamcity[testStarted name='MyTest.test2'] ##teamcity[testFailed type='comparisonFailure' name='MyTest.test2' message='failure message' details='message and stack trace' expected='expected value' actual='actual value'] ##teamcity[testFinished name='MyTest.test2']

Indicates that the "testname" test failed. Only one testFailed message can appear for a given test name.
message contains the textual representation of the error.
details contains detailed information on the test failure, typically a message and an exception stacktrace.
actual and expected attributes can only be used together with type='comparisonFailure to report comparison failure. The values will be used when opening the test in the IDE.

Here is a longer example of test reporting with service messages:

##teamcity[testSuiteStarted name='suiteName'] ##teamcity[testSuiteStarted name='nestedSuiteName'] ##teamcity[testStarted name='package_or_namespace.ClassName.TestName'] ##teamcity[testFailed name='package_or_namespace.ClassName.TestName' message='The number should be 20000' details='junit.framework.AssertionFailedError: expected:<20000> but was:<10000>|n|r    at junit.framework.Assert.fail(Assert.java:47)|n|r    at junit.framework.Assert.failNotEquals(Assert.java:280)|n|r...'] ##teamcity[testFinished name='package_or_namespace.ClassName.TestName'] ##teamcity[testSuiteFinished name='nestedSuiteName'] ##teamcity[testSuiteFinished name='suiteName']

Interpreting test names

The Tests tab of the Build Results page allows grouping by suites, packages/namespaces, classes, and tests. Usually the attribute values are provides as they are reported by your test framework and TeamCity is able to interpret which part of the reported names is the test name, class, package as follows:

• TeamCity takes the suite name from the corresponding suite message
• if the reported test name starts with the suite name, it is truncated
• the part of the test name after the last dot is treated as a test name
• the part of the test name before the last dot is treated as a class name
• the rest of the test name is treated as a package/namespace name

Source: https://confluence.jetbrains.com/display/TCD8/Build+Script+Interaction+with+TeamCity