Added Web Techonology Identifier Module

[kunalsz]

In reference to issue #1541

Idea

• The idea to implement this was done was inspired by the famous browser extension Wappalyzer
• Using regex patterns for headers,cookies,meta tags,urls one can "guess" the service being used and possibly the version too.

Changes made

• A new module WebtechIdentifier has been added,working with the help of regex patterns stored in artemis/modules/data/web_tech_identifier_data.json
• docker-compose.yaml has been updated

Work Remaining

• Due to system limitations I am not able to run the tests as the process is very resource intensive when running all the containers and tests.If you could tell me how to just run a specific test. Here test/modules/test_web_tech_identifier.py
• Based upon your views I'll add a lot more services to artemis/modules/data/web_tech_identifier_data.json
• There's a linting issue on my side, I am attaching the pre-commit log hereby
  pre-commit.log

@kazet Looking forward to your insights !

[kazet]

I think the biggest issue is that there are thousands of possible technologies and writing detection signatures for all of them would be a waste of time. What do you think of using an existing detector or an existing signature database? Be mindful of the license though ;)

To run one test, you might replace the python -m unittest discover test.modules command in docker-compose.test.yaml with an invocation of a specific test. This will start all the helper containers though so that wouldn't save you a lot of resources. Do you have an access e.g. to a cloud VPS where you can start it?

[kunalsz]

@kazet All the patterns are available for 1000s of technologies here . There would be no issue in implementing all of them.
No I don't have access to a VPS.

[kazet]

The only issue I have with that repository is that it's not updated for very long. Is there a similar repository that keeps getting updates?

[kunalsz]

@kazet The repo isn't getting update but I don't think this affects the signatures used to detect technologies. I can add more technologies to test if they work or not.

[kazet]

The fact that it's not getting updates means that it will be us who need to update it if a new technology appears ;) I strongly prefer finding another one, when there is somebody else adding signatures for each new PHP backend framework

[kunalsz]

@kazet I get your point. I'll see if I can find something which is regularly updated.

[kunalsz]

Hi @kazet I found some repos which are updated more frequently and actively

• https://github.com/enthec/webappanalyzer/tree/main/src/technologies
• https://github.com/s0md3v/wappalyzer-next/blob/main/wappalyzer/data/technologies.json
• https://github.com/HTTPArchive/wappalyzer/tree/main/src/technologies

[kazet]

which one would you choose?

[kunalsz]

@kazet https://github.com/enthec/webappanalyzer/tree/main/src/technologies this one would be the best,the project aims to maintain the technologies.

[kazet]

It's nice indeed - the only problem is that it's GPL-licensed and the GPL license is incompatible with Artemis core license (BSD). Do you have such signature repo that is MIT- or BSD-licensed?

If you have only GPL-licensed ones, we'd need to move the web_tech_identifier module to Artemis-modules-extra, which is a separate repository so the license requirements can be relaxed, but I think it'd be better to have such a module in core.

[kunalsz]

@kazet I will ask the maintainers of that project if they could update their license or I'll try to find another resource

[kazet]

This is a good idea, thank you!