Cockpit project: output seem more less well, but exceptions inside logs

[jscotka]

I've scheduled ./rapidast.py --config config.yaml on Cockpit web UI https://github.com/cockpit-project/cockpit

I've used spiderAjax, beucause it widely uses JS,

everything seems to work well

...
Job report generated report /zap/results/reports/zap-report.html
Job report finished, time taken: 00:00:00
Automation plan succeeded!
INFO:The ZAP process finished with no errors, and exited with code 0
INFO:Running postprocess for the ZAP Podman environment
INFO:Extracting report, storing in ./results/cockpit/DAST-20240105-163854-RapiDAST-cockpit/zap
INFO:Saving the session as evidence
INFO:Running cleanup for the ZAP Podman environment
rapidast_cockpit_zap_MgXxkO
INFO:scanner: 'zap' completed successfully

But there are several issues what I think what are there:

• It displays to found just 18 URLs, so my question is, it uses just <a> links or also possible to click and continue with another pages?
• Is it possible to incorporate into cockpit python unittests, our tests uses chromium CDP protocol to testweb UI to be able to test more, than just static page with links
• when looked into zap.log, there are exceptions, although it succeed like failed to copy the output of process 3838 ... java.io.IOException: Stream closed

024-01-05 15:37:34,725 [Thread-70] WARN  ExternalProcess - failed to copy the output of process 3351
java.io.IOException: Stream closed
	at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:176) ~[?:?]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:342) ~[?:?]
	at java.io.InputStream.transferTo(InputStream.java:704) ~[?:?]
	at org.openqa.selenium.os.ExternalProcess$Builder.lambda$start$0(ExternalProcess.java:209) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2024-01-05 15:37:40,060 [ZAP-AjaxSpiderAuto] INFO  CrawlController - Received shutdown notice. Reason is Exausted

So I do not know if it succeed or there are false negative results of that?

zap.log

when I've used alone spider: not AJAX it leads to errors that connection refused, so I do not know what's bad there.
it went to errors:

Job spider started
Job spider requesting URL http://localhost:9090/
Job spider failed to access URL http://localhost:9090/ : Connect to http://localhost:9090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Job spider finished, time taken: 00:00:00
Automation plan failures:
	Job spider failed to access URL http://localhost:9090/ : Connect to http://localhost:9090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARNING:The ZAP process did not finish correctly, and exited with code 1

[jeremychoi]

Re: failed to copy the output of process => A further investigation will be necessary, but for now, some information is found at SeleniumHQ/selenium#13096 and https://groups.google.com/g/zaproxy-develop/c/Rn44qfR6Fvg

Re: 'spider', maybe because of 'http://'? versus 'https://' which I see in the zap.log

[jscotka]

I've tested https://localhost:9090 with spider as well, thats why I've tried to use also http, and results are same.

...
Job spider requesting URL https://localhost:9090/
Job spider failed to access URL https://localhost:9090/ : Connect to https://localhost:9090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Job spider finished, time taken: 00:00:00
Automation plan failures:
	Job spider failed to access URL https://localhost:9090/ : Connect to https://localhost:9090 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
WARNING:The ZAP process did not finish correctly, and exited with code 1
INFO:Running postprocess for the ZAP Podman environment
Traceback (most recent call last):
  File "/home/jscotka/git/rapidast/./rapidast.py", line 231, in <module>
    run()
  File "/home/jscotka/git/rapidast/./rapidast.py", line 216, in run
    ret = run_scanner(name, config, args, defect_d)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jscotka/git/rapidast/./rapidast.py", line 106, in run_scanner
    scanner.postprocess()
  File "/home/jscotka/git/rapidast/scanners/zap/zap_podman.py", line 133, in postprocess
    raise RuntimeError(
RuntimeError: No post-processing as ZAP has not successfully run yet.

But this could be caused by cockpit, that it uses widely JS and have some restrictions for simple browsing, so maybe this could be issue with spider.

• And back to AJAXspider browsing what is more important for me, Is it possible to incorporate fetch rendered screens and browser logs and page sources, selenium allows to get page souces and screenshots via methods of driver, so should be possible without any problem. Should be nice to put these files also into archive, to be able to debug, what browser really see. Because my feeling is that AJAX spider reports false negative , but I cannot see what it inspects.
• Also it reminds me if there is no issue how it works and when using localhost isn't there some difference when not using FQDN and using podman container. against local service, this could also leads to some different behaviours of both spider browsers.
• As we uses CDP protocol for testing cockpit pages and also internally also selenium, we have also options to use full browser locally instead of headless to see real actions on screen to be able to debug it. is is also possible there. I know that selenium containers also I thing provide to run them in debug mode and I thins it provides VNC connection to the machine with browser to see actions inside. If not possible to run it locally with local browser would be handy to provide option to open VNC client against browser inside container.