TheHive release model leads to unexpected regressions and other issues

[rolinh]

Problem Description

First, let me be clear: TheHive is a very nice product to which I did not find any open or closed source equivalent. I am very grateful that you publish this as open source with an open development model.

In order to make TheHive even better, I'd like to bring a topic to the table: I often experience regressions or new bugs when upgrading to what are supposed to be bugfix releases.

The main concern I have is that these "bugfix" releases often are not since they often also include new features (which you sometimes label as "enhancements").
For instance, TheHive v3.0.7 introduced the "delete case" feature (#100). This has lead to issues such as #553 or #534. TheHive v3.0.4 also brought its share of new features (I count 4 new features) and issues.

Experience now tells me to wait at least 1-2 weeks before upgrading from a release to the next "bugfix" release to make this experience as painless as possible. I believe this could be avoided.
I see that you already included 2 enhancements in the next bugfix release (3.0.10)... 😢
TBH, I'm satisfied with TheHive feature wise and I'd really favor stability over new features.

Possible Solutions

Change the release model to strictly include bug fixes changes in a bugfix release (ie when only the patch level version number is bumped). At the very least, bump the minor version number if you include new features or "enhancements" in a release.
If you need to publish new features more often, maybe release new feature versions more often or switch to a more frequent release model than once or twice a year?

[nadouani]

Hello, thanks for the feedback.

Yes, we almost all agree with what you said, yes we include "small" "features" or "enhancements" when we release a "hotfix/patch/bugfix releases".

Theoretically, hotfix should not include features, but just bug fixes, but sometimes there are features that could help some people and we don't want to wait for months to ship them.

We usually alter our roadmap based on what is requested by the community, and that's probably the issue.

We will discuss it since it's a good point, and we are also open to any feedback about it, other opinions etc..

[SHSauler]

New features bring new bugs - there's no way around that, even after vigorous testing.

We are rather in favor of new features, even in bugfix releases. However, following policy we don't immediately upgrade, but first deploy on testing and leave it for a week or two.

[saadkadhi]

Our QA needs to be improved. And regressions are a terrible thing and we had quite a share of those lately. We fully agree on that and we are looking for ways to address this properly while adding features we believe are important for our community to level the fight against attackers.

Confidence in our ability to execute and deliver is an absolute must.

Thank you for your feedback and for taking the time to share your thoughts. We truly appreciate it.

Regards,

[nadouani]

One more point, releases other than hotfix usually introduce new features that require database migrations, which we prefer to minimise and make less frequent as possible.

[rolinh]

Theoretically, hotfix should not include features, but just bug fixes, but sometimes there are features that could help some people and we don't want to wait for months to ship them.

I definitely understand this point hence my suggestion of switching to a more frequent release models, ie more frequent releases with fewer new features.

One more point, releases other than hotfix usually introduce new features that require database migrations, which we prefer to minimise and make less frequent as possible.

Why not keep these for new major releases and use the minor releases to bring in easier/less intrusive new features? And keep hotfix releases to fix bugs :)

[crackytsi]

I wonder why there are many issues that are obviously bugs, but they are not planned to be included in the next bugfix Release.
I really look forward to new features, but open bugs have a higher prioirity for me. E.g. the missing short-reports or the MISP sync errors

[nadouani]

This is a valid question. Well it depends on:

• can we reproduce?
• what’s the effort to put in to reproduce?
• how many users are facing the problem?
• what’s the effort to fix the issue?

And some other questions.

For example, concerning the mini reports issue, I’ve personally spent a lot of time to reproduce the issue, whitout success. After the long discussion we had, we ended saying: we need to create 2 flat analyzers to try to reproduce the issue, which is time consuming without any certainty that we will reproduce the issue, and nobody else asked for it. I’m not saying that the issue does for sure not exist.

To summarize, we really do our very best to fix everything.
We all agree that we don’t want to keep things broken :)

[crackytsi]

Thank you very much, for this comments.
Of course you do this, and I have to repeate once more, that your work is so great and I love it so much.
I will thinking about a way to reduce load from your shoulders but on the other side prevent bugs from getting old ;)

[vacmf]

First of all thanks once again for the effort that you all put in this open source project development!!
I support the idea of having separate bugfix and features releases. I want to make a logical consideration here to support my thought.
When you introduce The Hive in your environment you define a workflow around it and decide what features and options to use, and what not. Then it gets likely integrated in your IR environment to interact with other tools and data sources (Cortex, MISP, API, incident automation, etc).
When there is a regression bug, or a newly introduced bug that affects something already in use, there are high chances to break that workflow, which is quite annoying and potentially disruptive.
New features (not enhancements) on the other hand, need to be adopted before being used, so the likely hood of breaking the current workflow it lower.
Have targeted and more frequent bugfix releases separated by features and major released might help to quickly address key issues and allocate needed time for longer development efforts, database migrations, etc.
Also writing regression tests for critical bugs might help.

[petiepooo]

(Edited to use "feature" instead of "nightly" for the releases with update features)
I'll suggest the LTS model: pick a current stable release with most of the features you want to have out there, declare it a "long-term support" release, and stick to strictly bug fixes on that until it's no longer supported. After some time, declare a newer LTS version and end updates for the older one.
It adds a little testing load, as you'll always want someone to be able to upgrade from the most current LTS version to the latest feature version if they need a newly-released feature. DB migrations accumulate between LTS releases, so regression testing has to include not only feature-to-feature but LTS-to-feature. As long as LTS-to-feature always works, when it's time to declare one of the feature builds as the next LTS release, the DB migration is already working.
Users get their choice between stability (on the LTS track) or features (on the feature track).