Hey Matthias,

Thank you for sharing! I thought of something similar to your script as a "quick fix" to resolve the problem temporarily.

Clarification:

The "cgroup.subtree_control" under "/sys/fs/cgroup/" and "/sys/fs/cgroup/system.slice" are created correctly.

Our BASE_CGROUP = system.slice/condor.service

Basically:

/sys/fs/cgroup/
    ├── cgroup.controllers 
    ├── cgroup.subtree_control
    ├── system.slice/
        ├── cgroup.controllers
        ├── cgroup.subtree_control
        ├── condor.service/
            ├── cgroup.controllers
            ├── cgroup.subtree_control (empty)
            └── <HTCondor jobs/subgroups>/
                ├── cgroup.controllers (empty)
                └── cgroup.subtree_control (empty)

I hope this adds more clarity to my question. Not sure why HTCondor is not inheriting the parent "cgroup.subtree_control" correctly from the "system.slice" and probably this is the reason why the job/subgroup specific dirs are not getting configured properly. I will set up a test instance and see if the "quick fix" works for me. I hope someone has a fix to our problem.

On 8/15/2024 5:18 PM, Matthias Schnepf wrote:
> Hi,
>
> I'm not sure why at point 6 of your "cgroup.subtree_control" file is empty and what manages it (condor or systemd, I think).
> We have a similar problem that the cgroup.the controller does not get set correctly.
> I hope someone else has an idea to fix your/our problem with the empty "cgroup.subtree_control" file.
>
> But here an idea of our "quick fix" we currently use.
> We use the development version of condor (23.7.2) and RHEL8.
> Our condor settings for cgroup v2 are:
>
> BASE_CGROUP = htcondor
> CGROUP_MEMORY_LIMIT_POLICY = custom
> CGROUP_HARD_MEMORY_LIMIT_EXPR = 2 * Target.RequestMemory
> CGROUP_LOW_MEMORY_LIMIT = 0.75 * Target.RequestMemory
>
> The job cgroups are created in /sys/fs/cgroup/htcondor. We set the cgroup.subtree_control file via a cronjob at boot time.
>
>
> #!/bin/bash
>
> echo +cpu +cpuset +memory +pids >> /sys/fs/cgroup/cgroup.subtree_control
> export cgroup_name="/sys/fs/cgroup/htcondor"
> if [ ! -d ${cgroup_name} ]; then
>     mkdir ${cgroup_name}
> fi
> echo +cpu +cpuset +memory +pids >> /sys/fs/cgroup/htcondor/cgroup.subtree_control
>
> With that, CPU, memory, and pids controller are set for the htcondor cgroup and its jobs/subgroups. With that, condor sets the correct memory limits, CPU weights, and monitors the memory.
>
> Best regards,
>
> Matthias
>
>
> On 8/15/24 4:47 PM, Sanjay Kumar Srikakulam wrote:
>> Hi,
>>
>> We run an HTCondor cluster and recently noticed we are missing the Cgroups accounting. Our setup,
>>
>> HTCondor:
>>
>> $CondorVersion: 23.0.6 2024-03-14 BuildID: 720565 PackageID: 23.0.6-1 $
>> $CondorPlatform: x86_64_AlmaLinux9 $
>>
>> 1. We are using Rocky 9 on workers
>> 2. CgroupV2 is mounted on the workers
>> 3. CgroupV2 controllers file as the list: "cpuset cpu io memory hugetlb pids rdma misc"
>> 4. HTCondor is configured to use CGroups:
>>
>> BASE_CGROUP = system.slice/condor.service
>> CGROUP_MEMORY_LIMIT_POLICY = hard
>> RESERVED_MEMORY = 2048
>>
>> 5. I can see the "condor.service" directory under "/sys/fs/cgroup/system.slice"
>> 6. HTCondor is inheriting the parent controllers properly: I see the "cgroup.controllers" file and has the same list of controllers as the parent (above). However, the "cgroup.subtree_control" file is empty (the parent has the list of controller names; so this is not getting created or inherited properly)
>> 7. As per the HTCondor doc (https://htcondor.readthedocs.io/en/latest/admin-manual/ep-policy-configuration.html#cgroup-based-process-tracking), that once the BASE_CGROUP is defined, for every condor job there will be a dedicated dir in the BASE_CGROUP path for cgroup accounting. When jobs are submitted, I see the subdirectories "condor_var_lib_condor_execute_slot1_7@hostname". However, the "cgroup.controllers" file is empty in these sub-directories and is somehow not inheriting the parent. Similarly, the "cgroup.subtree_control" file is also empty.
>>
>> 8. We also added the "CREATE_CGROUP_WITHOUT_ROOT = True" to our HTCondor config and restarted the condor services without luck.
>> 9. Also, from the starter log: "StarterLog.slot1_1:08/15/24 14:21:09 (pid:3758318) ProcFamilyDirectCgroupV2::track_family_via_cgroup error writing to /sys/fs/cgroup/system.slice/condor.service/cgroup.subtree_control: Device or resource busy", HTCondor seem to be hitting the "no internal processes" rule (https://unix.stackexchange.com/questions/680167/ebusy-when-trying-to-add-process-to-cgroup-v2; https://manpath.be/f35/7/cgroups#L557).
>>
>> Any help on resolving this is much appreciated! 

#!/bin/bash
echo "Hello World"
echo $(hostname)

sleep 60

for ((i=1; i<=10000000; i++)); do
    :
done

# Cgroup stuff added by Galaxy to each job script
if [ -e "/proc/$$/cgroup" -a -d "/sys/fs/cgroup" -a ! -f "/sys/fs/cgroup/cgroup.controllers" ]; then
    cgroup_path=$(cat "/proc/$$/cgroup" | awk -F':' '($2=="cpuacct,cpu") || ($2=="cpu,cpuacct") {print $3}');

    if [ ! -e "/sys/fs/cgroup/cpu$cgroup_path/cpuacct.usage" ]; then
        cgroup_path="";
    fi;

    for f in /sys/fs/cgroup/{cpu\,cpuacct,cpuacct\,cpu}$cgroup_path/{cpu,cpuacct}.*; do
        if [ -f "$f" ]; then
            echo "__$(basename $f)__" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics;
            cat "$f" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics 2>/dev/null;
        fi;
    done;

    cgroup_path=$(cat "/proc/$$/cgroup" | awk -F':' '$2=="memory"{print $3}');

    if [ ! -e "/sys/fs/cgroup/memory$cgroup_path/memory.max_usage_in_bytes" ]; then
        cgroup_path="";
    fi;

    for f in /sys/fs/cgroup/memory$cgroup_path/memory.*; do
        echo "__$(basename $f)__" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics;
        cat "$f" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics 2>/dev/null;
    done;
fi

if [ -e "/proc/$$/cgroup" -a -f "/sys/fs/cgroup/cgroup.controllers" ]; then
    cgroup_path=$(cat "/proc/$$/cgroup" | awk -F':' '($1=="0") {print $3}');

    echo "$cgroup_path"
    ls -la /sys/fs/cgroup/${cgroup_path}/
    for f in /sys/fs/cgroup/${cgroup_path}/{cpu,memory}.*; do
        echo "__$(basename $f)__" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics;
        cat "$f" >> /data/jwd05e/main/test_condor_submit_cgroup/__instrument_cgroup__metrics 2>/dev/null;
    done;
fi

sleep 10