New Glideins When Frontend Job Queue is Empty

[namrathaurs]

Describe the bug
After setting up a container environment with GlideinWMS v3.11.0.rc2 (development series) to test the credentials refactoring module, it was observed that when the Frontend has completed running the submitted jobs and there are no more jobs in the Frontend queue, there are new Glideins being requested in the Factory. This continues to be the case until the glideclient classad expires. In contrast, this behavior is not being observed in my development environment (running GlideinWMS v3.10.7).

To Reproduce
Upgrade existing installation of GlideinWMS to 3.11.0.rc2.

Expected behavior
No new glideins should be requested or created in the factory when there are no jobs submitted in the frontend queue.

Screenshots
Did some basic investigation following Marco Mambelli's suggestions to understand whether the problem was on the Factory or Frontend side. Here's the report:

Before submitting jobs:

# nothing found since glideclient classad does not yet exist (both in test and dev environments)

After submitting jobs:

# on test environment
ReqIdleGlideins = 3
ReqIdleLifetime = "0"
ReqMaxGlideins = 12
    
# on dev environment (production version from master)
ReqIdleGlideins = 3
ReqIdleLifetime = "0"
ReqMaxGlideins = 12

After jobs were completed:

# on test environment
ReqIdleGlideins = 3
ReqIdleLifetime = "0"
ReqMaxGlideins = 12

# on dev environment (production version from master)
ReqIdleGlideins = 0
ReqIdleLifetime = "0"
ReqMaxGlideins = 0
    
# even after 10 minutes of completing jobs, on test environment it still was:
ReqIdleGlideins = 3
ReqIdleLifetime = "0"
ReqMaxGlideins = 12

When I saw a new glidein created in the factory:

# on test environment
ReqIdleGlideins = 3
ReqIdleLifetime = "0"
ReqMaxGlideins = 12

# on dev environment (production version from master)
# no such classad attributes found like in the test environment

Info (please complete the following information):
Stakeholders and components can be a comma-separated list or on multiple lines.
If you add a new stakeholder or component, not on the sample list, add it on a line on its own.

• GlideinWMS version: 3.11.0.rc2
• Python version: 3.9
• OS version: EL9
• HTCondor version: 10.9.0
• Priority: medium
• Stakeholders: FactoryOps, HEPCloud
• Components: frontend

Additional context
Add any other context or supporting files about the problem here.