HTMLUNIT Java only redirects to final page when debugging

I have a problem when trying to reach a final page (after log in) when using HTMLUNIT (version 2.67.0)

When I debug my code (line by line), I can put my e-mail and password and go the final page after the log in. But, when I run the code normally, I get stuck on the Wait... Redirecting to the Final Page. I have tried MANY MANY stuff (creating new threads to execute the critical parts, synchronize, etc), but nothing seems to work so...

HELP PLEASE! =)

Here is the code...

WebClient webClient = createWebClient();
    try {
            System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
            
            HtmlPage page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation");
            
            threadWait(2);

            HtmlSubmitInput button = (HtmlSubmitInput) page.getElementByName("submit.IdentificarUsuario");
            
            threadWait(1);
            
            HtmlTextInput textField1 = (HtmlTextInput) page.getElementById("username");
            
            threadWait(1);
            
            textField1.setValueAttribute("[email protected]");
            
            threadWait(2);
            
            button.click();

            threadWait(3);
            //Retrieve the page again, with the e-mail/user value, so I can enter the pwd
            page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation");

            threadWait(3);
            
            HtmlPasswordInput textField2 = (HtmlPasswordInput) page.getElementById("password");
            
            threadWait(1);
            
            HtmlSubmitInput button2 = (HtmlSubmitInput) page.getElementByName("submit.Signin");
            
            threadWait(1);
            
            textField2.setValueAttribute("myPwd123");

            threadWait(2);
            //The critical parts: when I click the "log in" btn
            synchronized(button2) {
                button2.click();
            }

            threadWait(5);
            //The other critical part, when I retrieve the page again. 
            //When debbuging, this part gets me to the final page... when running normally, 
            // it gets stucked on the "Redirecting..." page
            synchronized (page) {
                page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation");
            }
            
            threadWait(10);
            
            System.out.println(page.asXml());

The createWebClient() method:

public WebClient createWebClient() {
    
    WebClient webClient = new WebClient(BrowserVersion.CHROME);
    try {
        //parâmetros do webclient
        webClient.setJavaScriptTimeout(10000);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        webClient.getOptions().setUseInsecureSSL(true);
        webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
        webClient.getOptions().setTimeout(0);
        webClient.getOptions().setRedirectEnabled(true);
        
        java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(Level.OFF);
        java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(Level.OFF);
        return webClient;
    } catch (Exception e) {
        logger.error("Error when creating WebClient: {}", e.getMessage());
        return null;
    }
}

The threadWait() method: (it implements a minimum plus a random interval of time between every action to simulate a real user trying to log in)

public void threadWait(int segundos) {
    try {
        Random r = new Random();
        int min = 1000;
        int max = 5001;
        int random = r.nextInt(max - min)   min;
        
        segundos = (segundos * 1000)   random;
        
        Thread.sleep(segundos);
    } catch (Exception e) {
        System.out.println("Error in Thread.sleep() ... "   e.getMessage());
    }
}

Reading all over StackOverflow, the idea is to Thread.sleep(mills) to simulate the debug, which I did as you can see. But, it just does not work.

Please, how the hell I can get this thing to work?

CodePudding user response：

After many many tests and failures... This is the code that works...

public WebClient loginONS() {
    
    WebClient webClient = createWebClient();
    try {
            System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
            configureWebClient(webClient);
            page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation");
            
            threadWait(2);

            HtmlSubmitInput button = (HtmlSubmitInput) page.getElementByName("submit.IdentificarUsuario");
            
            threadWait(1);
            
            HtmlTextInput textField1 = (HtmlTextInput) page.getElementById("username");
            
            threadWait(1);
            
            textField1.setValueAttribute("[email protected]");
            
            threadWait(2);
            
            button.click();

            threadWait(3);
            
            page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation");

            threadWait(3);
            
            HtmlPasswordInput textField2 = (HtmlPasswordInput) page.getElementById("password");
            
            threadWait(1);
            
            HtmlSubmitInput button2 = (HtmlSubmitInput) page.getElementByName("submit.Signin");
            
            threadWait(1);
            
            textField2.setValueAttribute("MyPwd123!");

            threadWait(2); 
            
            button2.click();

            threadWait(5);
            
            synchronized (page = webClient.getPage("https://pops.ons.org.br/ons.pop.federation")) {
                page.wait(5000);
            }
            
            if (page.asXml().toUpperCase().contains("AGUARDE")) {
                page = webClient.getPage("http://pop.ons.org.br/pop");
            }

            if (page.asXml().toUpperCase().contains("TODOS OS AVISOS")) {
                logger.info("Success on Log in");
                return webClient;   
            }
            else return null;
    } catch (Exception e) {
        logger.error("Exception {}", e.getMessage());
        return null;
    }
}

The configureWebClient() method:

public void configureWebClient(WebClient webClient) {
    webClient.getOptions().setCssEnabled(true);
    webClient.setJavaScriptTimeout(0);
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.getOptions().setUseInsecureSSL(true);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webClient.getOptions().setTimeout(0);
    webClient.getOptions().setJavaScriptEnabled(true);
    webClient.getOptions().setRedirectEnabled(true);
    
    CookieManager cookies = new CookieManager();            
    cookies.setCookiesEnabled(true);
    webClient.setCookieManager(cookies);
    
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());
    
    webClient.waitForBackgroundJavaScript(10000);
    webClient.waitForBackgroundJavaScriptStartingBefore(10000);
    
    webClient.getCache().setMaxSize(0);
    
    java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);
    java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
    java.util.logging.Logger.getLogger("org.apache.commons.httpclient").setLevel(java.util.logging.Level.OFF);
}