Add configurable rules for advanced log retention policies

[jongpie]

Hey, @jongpie we discussed cases where we use the logger and came up with a similar idea. Potentially we will create a lot of logs per day and in most cases, they aren't really relevant to be stored for always. We looked into your Batch Deletion feature. But as far as we understand the feature does delete all logs after x days regardless of whether there is only info, an error, or in which context the log was created. What are your thoughts on setting up multiple retention rules depending on the scenario and included error?

For example:

• We have feature A which is very important. Logs of this feature must not be deleted at all.
• For Feature B we want to see the logs for 5 days. Then they can be deleted. Except there was an error. Then they should be deleted after 30 days.
• Feature C isn't really important. Logs should be deleted after 3 days.

All of the features have an identifier using the scenario field. And then there could be a custom metadata type where you can define a scenario, a retention day, and whether error logs should be skipped. For all logs without any scenario, you can have a default setting.

We probably will develop such a feature, but wanted to discuss it with you before. Maybe you say, that it might be relevant in general, then we can find some synergies.

Originally posted by @fentes in #207 (comment)

[fentes]

Hey @jongpie,

as promised, I wanted to share my feedback to the new plugin. Currently we have the issue, that the rules are not working (see #354 ), but a few weeks ago I was able to test it and there it was working.

In general, this feature is amazing and really helps to keep overview of everything and not blowing up the database with tons of logs. Since we are logging user inputs as part of the customer community, we need to keep these logs forever, if there are any discussions with user. But other logs are just interesting for debugging purposes and can be deleted aber some days.

Together with the rest of my team, we brainstormed about setting up the rules:

• Technically a rule consists of at least 2 metadata records (rule & 1+ rules). Since the management of custom metadata is not as easy as inserting everything via data loader, the setup is hard work, when doing by hand
• We defined our rules in a csv file, based on Scenario and Errors, converted the csv to JSON, installed your CustomMetadataSaver and wrote a small script to insert everything. It worked fine for us, but I guess users without coding skills have to do it manually then.
• Since we have many different scenarios used, it will be a many rules. But we discovered, that most of them are quite similar. As mentioned above, we want to keep logs about user inputs of the customer portal forever, Error logs from the portal maybe for a year, some features some days, and a specific feature some other days. So our idea was, to combine rules

Currently we don't completly understand all capabilities of this tool. Maybe you can give some insights, of what is possible 😉

• What happens, if a log is matching with multiple rules? Which rule is used then? Can we influence the order?
• To combine some rules, I was trying to use tags instead of scenarios. Is that somehow possible? Workaround we discovered is to add a prefix to the scenario and use the contains operator: "Portal: Feature A" + "Portal: Feature B" to group all Portal stuff
• At which time, the date is calculated? So Rollup Summaries are working, but are also formula fields possible? Would it be possible to use Apex Rollup, to get the tag to the log and use the rules then? Maybe it is possible to rerun the calculation for some logs (Or am I thinking to complex?)
• Is it easily possible to define the days in the call. For example: Logger.setRetentionDays(30);
• Which objects can be used: Log__c (of course), but also LogEntry__c (the documentation sounded a bit like that)
• The conditions have a mandatory relationship to to the rule. Are there some ways to reuse conditions? For example: "Is there an error" will be used quite often, and creating 30 conditions that all look the same, just with different rule assigned

[jongpie]

Hi @fentes - this is some incredible feedback, thanks so much for brainstorming with your team & sharing everyone thoughts! I have a couple of follow up thoughts & questions for you & your team

• In general, this type of configuration data feels like it should be stored in a custom metadata type, especially since custom metadata type records can be deployed easily between environments. But since this is a plugin, I'm open to doing things a bit differently for this - do you think it would be better to use some custom objects to store the rules & conditions? Salesforce has a much better UI for custom objects, and the data could be exported/imported between environments (not quite as easily as deploying custom metadata types, but it's still feasbile)
  • Another possible option is to only have a LogRetentionRule__mdt record, and instead of using the child object LogRetentionRuleCondition__mdt, it could do something similar to the cleverness that @jamessimone has implemented for Apex Rollup, and just use a single WHERE clause field that the code parses. This would let you create a single, deployable custom metadata record with all of the relevant conditions (using SOQL syntax), although there would be some additional effort needed to implement the parsing logic
• I think it's great to hear that you were able to use my CustomMetadataSaver repo to help with this - I think the better option would of course be for Salesforce to improve the UI for creating custom metadata type records, but glad that you were able to find a workaround to make it a little easier.

Regarding your other questions, I've added my responses below each question - hopefully this format is readable, but let me know if you have any follow up questions

• What happens, if a log is matching with multiple rules? Which rule is used then? Can we influence the order?
  • This is controlled by the optional field LogRetentionRule__mdt.ExecutionOrder__c - the first rule that matches a log entry/log will be used
• To combine some rules, I was trying to use tags instead of scenarios. Is that somehow possible? Workaround we discovered is to add a prefix to the scenario and use the contains operator: "Portal: Feature A" + "Portal: Feature B" to group all Portal stuff
  • At the moment, tags cannot be used for the log retention rules plugin. It might be possible to incorporate that in the future, but it would involve some bigger changes to how the plugin is currently built. Let me know if you think this would be useful, I'm happy to explore this idea (I think it's a great idea).
• At which time, the date is calculated? So Rollup Summaries are working, but are also formula fields possible? Would it be possible to use Apex Rollup, to get the tag to the log and use the rules then? Maybe it is possible to rerun the calculation for some logs (Or am I thinking to complex?)
  • This would require some small changes to the log retention rules plugin - it currently only runs AFTER_INSERT for the LogEntry__c object - in order to work like you're suggesting, the plugin would need to also run for the LogEntryTag__c object and/or run on AFTER_UPDATE on the LogEntry__c object
• Is it easily possible to define the days in the call. For example: Logger.setRetentionDays(30);
  • This isn't currently possible, but I like the idea. I'll do some exploration on if this could be implemented.
• Which objects can be used: Log__c (of course), but also LogEntry__c (the documentation sounded a bit like that)
  • Both LogEntry__c and Log__c should work. In your screenshot on Log Retention Rules are not working #354, you are already specifying Log__r.Scenario__c to use the Log__c object, so if you want to use a field on LogEntry__c, you would just specify the condition's field with the API name of the LogEntry__c field (such as OriginType__c, etc.)
• The conditions have a mandatory relationship to to the rule. Are there some ways to reuse conditions? For example: "Is there an error" will be used quite often, and creating 30 conditions that all look the same, just with different rule assigned
  • At the moment, there's not a way to re-use them. I could possibly add a third custom metadata type object to act as a junction object between LogRetentionRule__mdt and LogRetentionRuleCondition__mdt, but that would make the initial setup even more cumbersome for all of the rules. Let me know what you think of this idea

Thanks again for all of your feedback!

[fentes]

Hey @jongpie,

• Single rule record + Where clause:
  • I really like that approach! Then I guess you need a picklist field to define the object (Log, Entry,...). I'm quite interested in how the logic behind looks like, because I can use it for something else as well 😁
  • I totally agree with you, that these rules should be custom metadata. Storing logic in custom objects is always a pain in the ass, both when deploying to production and refreshing a sandbox.
• Order of Execution
  • Ok cool, that is helpful
• Using tags in rules
  • If you need to change a lot in the background, I would leave it out then
• Logger.setRetentionDays(30);
  • If you think that this is easier to achieve than using tags, it might be solving a similar problem and is then fine enough
  • Maybe it is then also possible to not set the days as an integer but the name of a rule, to easily adjust the days, without changing the code.
• Time of calculation
  • Running the calculation also in AFTER_UPDATE could solve some issues. Would that have some bad impact on performance? Not sure whether BEFORE_UPDATE (in case of Log__c) would do behave the same with less impact (saving update one DML)
• Reusing rule conditions
  • when going with the approach of WHERE clauses, this becomes obsolet (at least of the where allows OR)
  • I wouldn't go with a separate junction object. If the rules would be stored in a custom object, that might be okay, but for custom metadata, it would be too complex for too less benefits.

[jongpie]

@fentes thanks for the feedback! I'm going to try to do 3 things:

1. Over the next few weeks, I'm going to try to fix the existing implementation so that it works as intended (to resolve Log Retention Rules are not working #354)
2. I'll also try to make some small enhancements to support more trigger events (at least BEFORE_UPDATE and/or AFTER_UPDATE), and try to add support for additional objects like LogEntryTag__c
3. Over the next few months, I'm going to see about eliminating LogRetentionRuleCondition__mdt altogether, and instead changing the implementation to use a single field on LogRetentionRule__mdt to specify SOQL-style filtering syntax (like @jamessimone does for Apex Rollup). This will be a fairly complex change, but I'd like to try to at least create a proof of concept of this idea, as I think this would be a nicer setup.

I have a few other releases that I need to finish before I can work on this, but I'll keep you updated on my progress.

And since we're having some great discussions here, I'm re-opening this issue for now - I'll close it again once I've implemented some of the above items.