add dynamic hyperparameter scaling

[GeorgWa]

Dynamic scaling of batch size and learning rate in NN classifier.

• Prevents FDR collapse for small libraries
• Improves speed for large libraries
  Overrides static config of batchsize and lr

[anna-charlotte]

lgtm!

[anna-charlotte]

Is it intentional that the max batch_size is not closer to 5000 anymore, as we used it previously?

[GeorgWa]

This kinda sneaked in here :D
I saw improved performance going down from 5000 to 1000

[github-actions]

Simplify the batch size calculation by combining min/max logic into a single line using max() and min() functions instead of np.clip(). This makes the logic more explicit and easier to follow.

def get_scaled_training_params(df, base_lr=0.001, max_batch=1024, min_batch=64):

    n_samples = len(df)

    # For >= 1M samples, use max batch size
    if n_samples >= 1_000_000:
        return max_batch, base_lr

    # Calculate scaled batch size (linear scaling between min and max)
    batch_size = int(max(min(max_batch * n_samples / 1_000_000, max_batch), min_batch))

    # Scale learning rate using square root relationship
    learning_rate = base_lr * np.sqrt(batch_size / max_batch)

    return batch_size, learning_rate

[github-actions]

Use debug level logging instead of info for hyperparameter tuning details since this is diagnostic information. Also simplified the log message formatting.

if self.experimental_hyperparameter_tuning:
            self.batch_size, self.learning_rate = get_scaled_training_params(x)
            logger.debug(
                f"Using scaled hyperparameters - samples: {len(x):,}, batch_size: {self.batch_size:,}, lr: {self.learning_rate:.2e}"
            )

[github-actions]

Remove explicit batch_size and learning_rate since they will be set automatically by experimental_hyperparameter_tuning. Add clarifying comment.

classifier_base = fdrx.BinaryClassifierLegacyNewBatching(
    test_size=0.001,
    epochs=10,
    experimental_hyperparameter_tuning=True, # Batch size and learning rate will be scaled automatically
)

[github-actions]

Improve test case clarity by: 1) Using power operator (**) instead of np.sqrt() for more readable learning rate calculations, 2) Adding descriptive comments for each test case, 3) Aligning values for better readability

@pytest.mark.parametrize(
    "n_samples,expected_batch,expected_lr",
    [
        (1_000_000, 1024, 0.001),          # Base case
        (2_000_000, 1024, 0.001),          # Above max samples
        (500_000, 512, 0.001 * (512/1024)**0.5),  # 50% scaling
        (250_000, 256, 0.001 * (256/1024)**0.5),  # 25% scaling
        (50_000, 64, 0.001 * (64/1024)**0.5),     # Min batch size
        (1_000, 64, 0.001 * (64/1024)**0.5),     # Below min batch size
    ],
)

[github-actions]

Number of tokens: input_tokens=28412 output_tokens=1087 MAX_NUM_OUTPUT_TOKENS=4096

[vbrennsteiner]

LGTM!

[vbrennsteiner]

Might be a small point but could we consider renaming 'max_batch' to 'max_batch_size' and 'min_batch' to 'min_batch_size'?

[vbrennsteiner]

You mention in a previous comment that a batch size closer to 1000 is actually optimal, would it make sense to change this default?

[vbrennsteiner]

Same again, I would recommend 'expected_batch_size' for clarity