Quarterly update for 2024-Q1

[mtesauro]

DefectDojo Community members,

Starting this year, I’m going to endeavor to create a community update every quarter (aka every 3 months) throughout the year. I’m going to call this my first update for 2024 so there will be at least 3 more after this. Exact timing is TBD and we’ll figure that out as we go forward.

2024-Q1 DefectDojo Updates

There are two topics in this update.

Upcoming Deprecations

As part of our continuing effort to drive up the quality of DefectDojo, we’re going to be deprecating some formerly supported technology options. The following options will no longer be supported by DefectDojo:

• MySQL as the database for DefectDojo
• RabbitMQ as a message broker for DefectDojo
• Testing PostgreSQL HA as part of our Github Actions - see here

Timeline:

• MySQL and RabbitMQ will be deprecated and no longer supported as of July 1st, 2024
• PostgreSQL HA testing will be removed in the 2.33.0 release (April 2024)

What this means to people using MySQL currently:

• DefectDojo stores the bulk of its data in the database so this will need to be migrated
• You will need to move from using MySQL with DefectDojo to PostgreSQL prior to July 1st to remain on a tested and supported option
• There’s already been discussions about migrating from MySQL to DefectDojo which provide options, Pros and Cons and other considerations
• This will be a one-time event
• For those running older versions of DefectDojo, it’s advisable to move to as close to the most recent version as possible prior to migrating. If for no other reason but to keep the scope of change as small as possible.

What this means to people using RabbitMQ currently:

• DefectDojo through Celery uses RabbitMQ and Redis to hold queued work items
• Since these are async items, durable, long term storage of them is not required
• Migration will consist of changing the configured from RabbitMQ to Redis - mostly the DD_CELERY_* settings especially the DD_CELERY_BROKER_SCHEME

What will happen after July 1st, 2024?

Once the transition period is over, the following will happen:

• MySQL and RabbitMQ specific tests in Github Actions will be retired. This should cut the time to run GH Actions roughly in half. 🎉
• The existing reference docker-compose configurations in the repo will have the MySQL and RabbitMQ options removed also greatly simplifying the yaml configs. 🎉
• After several releases, we’ll remove the Python libraries required by MySQL and RabbitMQ. Dockerfiles will also have those dependencies removed. 👋

All of these are obviously ‘to be created’ PRs which won’t be submitted until after the deprecation

New Guidelines on Parsers, especially multi-parsers

Recently, more and different types of security tools are being offered by the same vendor. While it seems like the natural thing to do is to create a ‘multi-parser’ to handle all the formats at once, this can lead to some problems:

• Parsers become harder to debug
• Unit tests for parsers become more complicated
• Creating a dedup algorithm for multiple tool types becomes difficult to impossible

Going forward, we will not accept PRs which are muti-parsers. 🚫

So, what’s a multi-parser?

• A parser written to handle more than one type of security scanner from a single vendor
• e.g. Snyk provides SCA, SAST and Container scans - a multi-parser would attempt to parse the outputs from 2+ of these tools in one parser (aka Test_Type in DefectDojo)

Are there any exceptions to this new guideline?

• Yes, if a tool happens to produce more than one file format, a single parser to handle that may be accepted if:
  • The type of security scanner isn’t different for the different file formats
  • Each of the file formats is handled separately in separate helper methods to increase visibility of what code is handling which format
• An example of this is supporting Fortify .xml and .fpr files in a single parser since both are file exports from the same type of security scanner (SAST)

Look for a PR to update “How to write a parser” shortly as well.

That’s all I have for 2024-Q1. 🥂

[navzen2000]

Can the decision to deprecate rabbitmq and mysql be re-considered?

The Redis dual license is not acceptable to many companies, and it would help to keep the current platform afloat.

Also, do you see any significant advantages by deprecating above technologies?

[mtesauro]

Thanks for the question about the deprecation. At this time, we're not open to reconsidering them.

About the dual-license for Redis:

We're aware of the licensing change. We're considering moving from redis to an alternate like KeyDB or ValKey, however:

• The dual-license change won't happen until redis version 7.4 and DefectDojo currently ships with 7.2 so still under the old license scheme
• We have time to evaluate the best path forward and want to avoid an unplanned change when there isn't urgency
• We'll let the community know if and when a change is happening as early as possible to allow a smooth transition should we move away from Redis.

About significant advantages of deprecating above technologies:

Beyond those already listed above, I'd also say:

• Ability to focus on making DefectDojo's functionality as rugged, bug free and performant with limited resources
  • Tech that supports an app has little to do with how it functions as long as that underlying tech supports the needed use-case
  • Postgres and Redis provide what's needed to run and use DefectDojo (support the use-case)
  • Time currently spent maintaining multiple, mix-and-match deployment options cannot be spent on making DefectDojo the most awesome open source ASPM/Vuln Mgmt platform.
  • Anytime you can remove complexity from a system, I'd call that a win
• MySQL 8.0.0 was released in 2016-09-12. Since that time, no contributor has created a merge-able PR to move from MySQL 5.x to 8.x.
  • Same for MariaDB - no merge-able PR has been created
  • As of Django 4.2, MySQL 5.x is no longer supported. We've delayed that upgrade to allow the community to migrate to Postgres as long as we're comfortable doing
  • As some point, MySQL 5.x will be EOS/EOL - TBH, I'm surprised Oracle hasn't EOL'ed MySQL 5.x already
  • Personally, I've been using Postgres as my preferred RDMS for over 20 years and based on my personal experience running DefectDojo in various places with Postgres, I've found it to perform exceedingly well.
• Maintaining dependency updates on a 2x2 matrix of technologies spends contributor time on updates that don't actually make DefectDojo better for users. It's just 2x the routine maintenance for no feature gain.

DefectDojo is an open source project - you have the full source code. If you feel strongly about MySQL or RabbitMQ, nothing is stopping you from running DefectDojo using those. Just note:

• We will no longer be running CICD/tests/GH Actions against either of those technologies. You'll own that work.
• We won't knowingly make changes in DefectDojo which will break MySQL but also make no promises that won't happen inadvertently

Repeating those from above to save some scrolling for the curious:

• MySQL and RabbitMQ specific tests in Github Actions will be retired. This should cut the time to run GH Actions roughly in half. 🎉
• The existing reference docker-compose configurations in the repo will have the MySQL and RabbitMQ options removed also greatly simplifying the yaml configs. 🎉
• After several releases, we’ll remove the Python libraries required by MySQL and RabbitMQ. Dockerfiles will also have those dependencies removed.

[StephanPillhofer]

Any updates when MySQL and RabbitMQ will be removed from the docker-compose file? It's already past July the 1st and they are still included.

[mtesauro]

That process is ongoing currently (e.g. #10512) - MySQL and RabbitMQ are not only in docker-compose but many of the GH Actions, Helm, documentation, etc reference MySQL and RabbitMQ. We're backing them out in the least disruptive method so that other PRs can be merged without GH Action failures, etc.

So, as of 2.36.0, we've deprecated MySQL and RabbitMQ. We should have MySQL and RabbitMQ removed fully from the repo no later then the 2.37.0 release (August 2024). This also gives the community a wee bit longer to transition away from those technologies.

HTH