Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

write quantiles to value repetitions #64

Closed
wants to merge 3 commits into from
Closed

write quantiles to value repetitions #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6929c6b
write quantiles to value repetitions
Jan 31, 2024
485aa4b
typo
Jan 31, 2024
a16443a
keep old counts as well
Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions outrank/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ def main():
'--feature_set_focus',
type=str,
default=None,
help='Collection of which feature transformations to consider',
help='Collection of which feature transformations to consider. To consider only --reference_model_JSON features, set _all_from_reference_JSON',
)

parser.add_argument(
Expand Down Expand Up @@ -238,7 +238,13 @@ def main():
default=1.0,
help='If < 1.0, MI algorithm will further subsample data in stratified manner (equal distributions per value if possible).',
)


parser.add_argument(
'--histogram_max_bins',
type=int,
default=100,
help='Number of histogram bins in value_repetitions.json',
)

args = parser.parse_args()

Expand Down
10 changes: 8 additions & 2 deletions outrank/task_ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import numpy as np
import pandas as pd
from statistics import quantiles
Copy link
Collaborator

@SkBlaz SkBlaz Feb 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that numpy has quantile functions too, even though this requires no extra dependencies, so it's fine


from outrank.algorithms.importance_estimator import rank_features_3MR
from outrank.core_ranking import estimate_importances_minibatches
Expand Down Expand Up @@ -280,9 +281,14 @@ def outrank_task_conduct_ranking(args: Any) -> None:
with open(f'{args.output_folder}/value_repetitions.json', 'w') as out_counts:
out_dict = {}
for k, v in GLOBAL_ITEM_COUNTS.items():
actual_hist = np.array(list(v.default_counter.values()))
frequencies = np.array(list(v.default_counter.values()))
more_than = lambda n, ary: len(np.where(ary > n)[0])
out_dict[k] = {x: more_than(x, actual_hist) for x in [0] + [1 * 10 ** x for x in range(6)]}
out_dict[k] = {str(x): str(more_than(x, frequencies)) for x in [0] + [1 * 10 ** x for x in range(6)]}
if len(frequencies) < args.histogram_max_bins:
out_dict[k]["quantiles"] = str(sorted(list(frequencies)))
else:
out_dict[k]["quantiles"] = str(quantiles(list(frequencies), n=args.histogram_max_bins))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list(frequencies) is redundant, when defining frequencies you can just .tolist() and reuse that

print(out_dict)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why print this? This will clog the output/logs

out_counts.write(json.dumps(out_dict))

with open(f'{args.output_folder}/combination_estimation_counts.json', 'w') as out_counts:
Expand Down
Loading