How to save rule files in Azure storage for Ruler service

[geekyspartan]

Hey All,
First of all thanks for maintaining Cortex. I am having a little trouble understanding the multi-tenant aspect of Ruler service.

Per the documentation mentioned here https://cortexmetrics.io/docs/guides/ruler-sharding/#ruler-storage,
for local storage, rules can be stored like this:

 /tmp/cortex/rules/<tenantid>/rules1.yaml
 /tmp/cortex/rules/<tenantid>/rules2.yaml

But how do I have to store the rule files in azure storage if I specify 'azure' as the backend using 'ruler_storage_config'?
Do I have to create separate folders for each tenants in azure container?

Also, if I specify tenantid as the subfolder, then that means only rules defined inside each tenantid folder will be run for that particular tenant?

And can I have multiple rule files for each tenant(inside each 'tenant id' folder) and code will automatically take care of running all the rules?

Thanks,

[pstibrany]

Hello, when using azure rule-store (or any other object store based rule store), you need to use Ruler API to upload rules. It's not enough to upload rule configuration to object store manually, as ruler does additional processing of the uploaded rule groups, before storing them to object store.

Ruler API is described here: https://cortexmetrics.io/docs/api/#ruler, and must explicitly be enabled using -experimental.ruler.enable-api command line option for now.

To simplify uploading rules, you can use cortextool command from https://github.com/grafana/cortex-tools repository.

[geekyspartan]

@pstibrany thanks for replying. I looked at the ruler api(Set rule group), it doesn't mention anything related to multi-tenant environment. Is 'namespace' used to differentiate between tenants?

Assuming, my ruler service is running for 2 tenants and I have few rule files in my rule-store, how will ruler service decide which rules are application for which tenant?

[pstibrany]

Set rule group requires authentication, i.e. special header recognized by Cortex, that indicates tenant. Note that Cortex fully trusts this header.

Eg. one way to upload rules for a tenant would be: curl -X POST -H "X-Scope-OrgID: tenant" --data-binary=rules.yaml http://cortex/api/v1/rules/some_namespace

You can thinks of "namespace" as a "file name" (ie. "rules.yaml" in the above example). When you have multiple files with rules prepared, you would upload each file (each group from the file) to a different namespace.

(I would recommend taking a look at cortextool from https://github.com/grafana/cortex-tools, which can do all of this for you)

[geekyspartan]

@pstibrany thanks, this clear lot of things. So, once files are uploaded and proper tenant are marked, ruler instance will know which rule files are targeted towards which tenant and will only execute tenant only specific rules?

[pstibrany]

ruler instance will know which rule files are targeted towards which tenant and will only execute tenant only specific rules?

Yes, exactly.

[geekyspartan]

Thank you for your help.