guess_is_lognorm exploit, patch

[zboldyga]

In the current implementation of cell-eval, it is possible to bypass the intended normalization by submitting prediction files that contain data in both a raw integer count and log1p scale (or some other large and small values). guess_is_lognorm treats the data as log1p normalized if it sees any fractional component, and so this opens a loophole to 'game' PDS and DES.

This proposed code change does not fundamentally change anything about the cell-eval metrics as proposed by ARC, but it does help ensure the checks more accurately reflect their promise. This sets a tighter boundary around what is considered a log1p value and does not allow mixing of significantly larger values.

This does not entirely eliminate transformation 'hacks' to the score. However, it should significantly limit one category of these non-biological transformations, and I've done this in a way that does not change the rules of the competition.

Note that this does mean that submissions cannot contain an individual log1p gene expression count >= 11, which is more than an order of magnitude higher than anything in the ARC H1 training data. Individual gene expression values in data submitted in log1p format would have to be clipped by contestants if they were to exceed this threshold, otherwise the code will throw an error.

I believe there is also pdex code with similar behavior that would benefit from a change, but I did not inspect it yet.

[gemini-code-assist]

Summary of Changes

Hello @zboldyga, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a critical security patch to the cell-eval system by enhancing the guess_is_lognorm function. The update prevents an exploit that allowed submission files to contain a mix of raw counts and log1p scaled data, thereby bypassing intended normalization checks. By introducing stricter validation for log1p data, the system now rejects prediction files with values outside the expected range (e.g., negative values or excessively large log1p values), ensuring the integrity and fairness of the evaluation metrics without changing the competition rules.

Highlights

• Exploit Mitigation: Patches the guess_is_lognorm function to prevent an exploit where mixed raw integer counts and log1p scaled data could bypass normalization checks in cell-eval.
• Stricter Validation: Introduces a validate parameter to guess_is_lognorm, which, when True, enforces a tighter boundary for log1p values, rejecting data with negative values or values exceeding a max_threshold (default 11.0).
• Prediction Data Integrity: The _convert_to_normlog function now explicitly enables this stricter validation for prediction data, ensuring that submissions adhere to the expected log1p scale.
• Comprehensive Testing: New unit tests have been added to cover various scenarios, including valid log1p data, integer data, values near and exceeding the threshold, negative values, and mixed-scale data, ensuring the robustness of the new validation logic.
• Development Environment Cleanup: Adds .idea/* to .gitignore to prevent IDE-specific files from being committed.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively addresses a potential exploit in guess_is_lognorm by introducing a validation step for predicted data. The new logic ensures that data appearing to be log-normalized also falls within a reasonable value range, preventing the submission of mixed-scale data. The changes are robust, well-structured, and are accompanied by a comprehensive suite of new tests that validate the fix across various scenarios. I have one suggestion to refactor a small piece of duplicated code to improve maintainability.

[zboldyga]

To be clear, this does not stop all transformations that game the metrics by manipulating perturbed cell values. But it does limit the amount you can push things apart in space quite a bit -- because everything now has to get normalized by log1p (or some functions that keep the values in the [0,11) range)