Clarification

I am making the test fail on purpose. I have enabled 1 retry on failed tests.

The Problem

If the test fails on Attempt 1, I expect it should also fail on Attempt 2.

Actual behaviour

The test fails on Attempt 1 but then is marked as passed on Attempt 2 without actually retrying.

Some Details

I have a number of Cypress's tests. In this particular test, I'm checking part of the URL to contain a specific string and making it fail on purpose. That fails on timeout on the first attempt but always passes on the second attempt without re-trying.

Why doesn't it retry or fail on a second attempt? I believe it's something to do with my assertion.

I have also tried with expect and then as they are run asynchronously

         cy.url().then(url => {
            expect(url).to.include("homssse.faces")
        });

Setup Details

On my command.js file I have added a login command

    Cypress.Commands.add('login', (username, password) => {
      cy.visit('signOn.faces')
        .get(".global-header__logo")
        .should("be.visible")

      cy
        .get('input[name="username"]')
        .type(Cypress.env('username'))
        .should('have.value', Cypress.env('username'));

      cy
        .get('input[name="password"]')
        .type(Cypress.env('password'))
        .should('have.value', Cypress.env('password'));

      cy
        .get('[name="submit"]')
        .click();
    })

In my critical_user_journeys.spec.js i have multiple tests, but I am showing here the first one. In this test, after I call login I want to assert I have landed on my Home page, but here I am failing the test on purpose to verify that it indeed fails when it actually breaks. I have enabled retries on all the tests coming from cypress.json, so naturally, when a test fails, it will attempt to run again. I have also set timeouts, details are further below.

    describe("Critical User Journeys", function () {

        it("signs in user and lands on 'HomePage'", () => {
            cy
                .login()

            cy.location().should((loc) => {
                expect(loc.href).to.include(
                    'this_doesnt_exist_on_the_url' // break tests on purpose
                )
            })
        })
    })

Some of the setup on the cypress.json file

    ...
      "pageLoadTimeout": 30000,
      "defaultCommandTimeout": 20000,
      "chromeWebSecurity": false,
      "retries": {
        "runMode": 1,
        "openMode": 1
      }
    ...

Important Note

When the assertion is changed to expect(true).to.be.false; both attempts fail, and that is the behaviour I expect to see with the other assertion, too.

Thanks

CodePudding user response：

There are a few different ways of adding the assertion. Do they all fail on the first try ?

cy.url().should('eq', 'http://localhost:8000/users/1/edit') // => true

or

cy.url().should('include', '/index.html')

CodePudding user response：

You can add a timeout to make sure the url is resolved. The default timeout is 4000 ms.

cy.url({timeout: 10000}).should('include', 'home.faces')

CodePudding user response：

I managed to have the expected behaviour by adding an empty beforeEach hook inside describe. I am not sure why adding this changes the behaviour, but when I saw it mentioned on the documentation, I added one to check the outcome and got the results I was looking for. Still not sure if this is by design or a bug, though.

This is how critical_user_journeys.spec.js file looks now

        describe("Critical User Journeys", function () {

            beforeEach(function () {})

            it("signs in user and lands on 'HomePage'", () => {
                cy
                    .login()

                cy.location().should((loc) => {
                    expect(loc.href).to.include(
                        'this_doesnt_exist_on_the_url' // break tests on purpose
                    )
                })
            })
        })

With this implementation Attempt 1 fails, then Attempt 2 fails, too. As a result, the whole test is marked as 'Failed'.