TestNG default order when there are multiple test classes-CodePudding

When executing a Maven testng setup, with mvn verify, I see the tests ordered alphabetically as documented at https://www.tutorialspoint.com/what-is-the-order-of-execution-of-tests-in-testng.

However this only applies to the tests within a class. What is the default test class execution order? Is it controlled by testng or the maven surefire plugin? e.g.

Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@646d64ab
Setup bTest
bTest1
bTest2
bTest3
bTest4
bTest5
bTest6
Setup AppTest
aATest
aFastTest
aSlowTest
aTest
alphabeticTest
zSlowTest
Setup cTest
cTest1
cTest2
cTest3
cTest4
cTest5
cTest6
Setup aaTest
aaTest1
aaTest2
aaTest3
aaTest4
aaTest5
aaTest6
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.221 sec

Update

Including the pom.xml as requested.

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>my.project</groupId>
  <artifactId>test-ng</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>test-ng</name>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.testng</groupId>
      <artifactId>testng</artifactId>
      <version>7.7.0</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

In short I'm using the default surefire configuration.

Update #2

After searching for a testng.xml file I confirmed none was present. But I did come across target/surefire-reports/old/Command line suite/testng.xml.html

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Command line suite" verbose="0">
  <test thread-count="5" name="Command line test" verbose="0">
    <classes>
      <class name="my.project.bTest"/>
      <class name="my.project.AppTest"/>
      <class name="my.project.dTest"/>
      <class name="my.project.cTest"/>
      <class name="my.project.eTest"/>
      <class name="my.project.aTest"/>
    </classes>
  </test> <!-- Command line test -->
</suite> <!-- Command line suite -->

It does not explain the class name order. But thread-count="5" looks suspicious. I tried override this value in the pom.xml with

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.0.0-M7</version>
      <configuration>
        <parallel>methods</parallel>
        <threadCount>2</threadCount>
      </configuration>
    </plugin>
  </plugins>
</build>

but the thread count remained at 5. That makes me suspect a mismatch between maven-surefire and testng. e.g. Maven: how to set thread count for testng

The play project I'm using is available at https://github.com/sregger/test-ng

CodePudding user response：

The default order of execution within TestNG is Order by instances

So if there are two or more test classes (which is what is true in your case) then there are two or more instances.

In that case, TestNG orders them by applying the below rules for ordering:

Collect all the @Test methods from all the test classes and then

Order them based on priority, and then
Order them based on the class to which they belong to, and then
Order them based on their names, and then
Order them based on the toString of them (which is internally equivalent to their signature), and then
Order them based on their parameters (if present)

You can look at the implementation of this ordering by referring to the code here.

There are totally 3 types of ordering that TestNG supports/honours:

methods ( Just order them based on their names )
instances (This is the default choice that TestNG picks up)
none (wherein you are saying do not resort to any ordering)

You can toggle between these 3 modes by using the JVM argument -Dtestng.order and passing in one of the above mentioned 3 values.

EDIT:

If you would like to control the order of the test methods to the order of the classes, then you should ideally speaking use a TestNG suite xml wherein you set the preserve-order attribute to true and set parallel as none. For more information on this aspect of TestNG please refer to the official documentation here

UPDATE:

Please note the following:

When you run TestNG tests via surefire plugin wherein you don't specify any configuration (even if you don't have the surefire plugin, it comes to you from a parent pom), then the test class discovery is ALWAYS done by Surefire and NOT TestNG. The way that happens is perhaps by scanning the classpath and then creating a list of them. You can see this yourself by cat the contents of target/surefire/surefire-<TimestampGoesHere>_1tmp wherein you will see entries like tc.5=my.project.aTest tc.3=my.project.cTest and so on.
By default thread-count attribute has a value of 5 and it always gets printed as part of dumping the Suite XML because that's how the TestNG suite xml file gets created by default. But unless and until you have the parallel attribute set to a valid value of tests|methods|classes|instances there is NO CONCURRENCY getting enabled. The best way to be convinced on this is to add a print statement that prints Thread.currentThread().getId() and see the thread ids yourself. You will NOT see unique thread ids if tests are running SEQUENTIALLY
If TestNG is left to discover the test classes by itself then it would resort to the ordering that I have explained earlier.
If you would like to control the order of how the test classes should be picked up, then you would use a TestNG suite xml file.
If you would still like to figure out how surefire plugin is basically scanning the classpath for tests, then you would need to do the following:

Run maven using debug mode (See the official documentation here for details)
Add up dependencies (not as plugins but as <dependency>) on the following so that you can setup debug points ( surefire-booter, surefire-testng)