refactor(verbosity): better handling for verbosity #109

cohow · 2024-08-31T15:48:56Z

Resolves #94

Reward is now calculated based on (count^expo) * multiplier * score * multiplierFactor.multiplier

for example, im going to use issue #95
reward calculation was the following

the new verbosity algorithm would take ((65^0.85) * 0.1 * 1 * 3) + ((16^0.85) * 0.1 * 0 * 3) which would equal to 10.42 instead of the old 19.5

Please check and let me know if there's any issues, also I not sure if it's an issue but I had to create a different _calculateFormattingTotal for calculations due to cognitive complexity related issues from lint

chore: merge development into main

Merge develop into main

feat: split reward between multiple assignees

Merge develop into main

…please--branches--main chore(main): release 1.3.0

Merge development into main

feat: using latest ChatGpt version and fixed truncated results

…please--branches--main chore(main): release 1.3.1

Merge develop into main

…ment Merge development into main

…-please--branches--main chore(main): release 1.4.0

gentlementlegen · 2024-09-01T03:26:32Z

@cohow Thank you for the PR. I am fine with the changes, please fix the Jest tests that all broke due to the results changing with the formula.

cohow · 2024-09-01T20:32:51Z

@cohow Thank you for the PR. I am fine with the changes, please fix the Jest tests that all broke due to the results changing with the formula.

I was looking at the tests and it seems like the problem is coming from the results not equaling the mock results and was wondering if there was a way to generate a new mock results file without having to edit all the rewards separately, if not then I'll just manage to figure something out.

gentlementlegen · 2024-09-02T04:20:45Z

@cohow When I want to update them I copy paste the diff of the proper result so I don't have to manually fix all the numbers.

However according to your screenshot there are lots of decimals to the results. I don't know if that's what we want.

cohow · 2024-09-02T17:48:06Z

I believe these changes should fix all the issues with the mock results being incorrect.

I've also changed the calculation to cut off at 2 decimal places because exponents tend to produce a lot of decimals. If you want I can also change it to round up or down, but that would require all the tests to be changed again.

let me know if anything goes wrong or any expected results appear, or you require any changes, I'll try to get a fix out ASAP.

whilefoo · 2024-09-02T19:42:39Z

src/parser/formatting-evaluator-module.ts

+        for (const symbol of Object.keys(curr.symbols)) {
+          const count = new Decimal(curr.symbols[symbol].count);
+          const symbolMultiplier = new Decimal(curr.symbols[symbol].multiplier);
+          const formattingElementScore = new Decimal(curr.score);
+          const exponent = this._wordCountExponent;
+
+          sum = sum.add(
+            count
+              .pow(exponent) // (count^exponent)
+              .mul(symbolMultiplier) // symbol multiplier
+              .mul(formattingElementScore) // comment type multiplier
+              .mul(multiplierFactor.multiplier) // formatting element score
+          );


word count exponent is being applied to all symbols, either we leave it like that but we should rename it to symbol exponent, or we can limit it to only words @0x4007

I'm not quite sure but I feel like word count specifically makes the most sense.

The problem is that config doesn't define word multiplier anymore, instead symbol regexes are used and technically two regexes which are a little bit different can both represent words so it's hard to apply the exponent only to the word count

I had a proposal for "segments" which would serve as aliases for word regex, sentence regex, paragraph regex, entire comment regex. Perhaps we can rely on that as the main "word counter."

Ultimately I think we should strive to make the config user friendly/idiot proof but I think its impossible to cover every situation for abuse proactively. I think we should define "best practices" and hope that partners don't shoot themselves in the foot with conflicting/bad configs.

I agree with having segments as aliases, we can keep regexes but I don't think many partners will bother to use that

0x4007 · 2024-09-03T11:58:56Z

I've also changed the calculation to cut off at 2 decimal places because exponents tend to produce a lot of decimals. If you want I can also change it to round up or down, but that would require all the tests to be changed again.

Use a numbers library like bignumber to handle this.

cohow · 2024-09-03T14:18:59Z

I've also changed the calculation to cut off at 2 decimal places because exponents tend to produce a lot of decimals. If you want I can also change it to round up or down, but that would require all the tests to be changed again.

Use a numbers library like bignumber to handle this.

bignumber to handle the rounding or cutting off decimals? right now decimal.js is able to handle both actually so I'm not sure if another library would be needed.

0x4007 · 2024-09-03T15:26:35Z

Sure use decimal for all calculations.

src/parser/formatting-evaluator-module.ts

cohow · 2024-09-08T14:40:21Z

if I'm not wrong the last checks tests depend on #108 or one of the issues that were linked in it. LMK if you need anything else!

gentlementlegen · 2024-09-09T13:23:38Z

@cohow You mean that your pull-request test fixes depend on another pull-request?

cohow · 2024-09-09T13:35:51Z

@cohow You mean that your pull-request test fixes depend on another pull-request?

The last tests failed due issues with permit generation, which is not caused by this PR and after checking other PRS I believe PR #108 fixes that issue if I'm not wrong

gentlementlegen · 2024-09-09T13:43:58Z

After checking the logs, it seems that everything works as expected but since your changes also modified the results the permit urls got changed as well (for example check https://github.com/ubiquibot/conversation-rewards/actions/runs/10753678522/job/29832632935?pr=109#step:4:432) so you should just have to fix the result and every test should pass.

cohow · 2024-09-09T14:56:53Z

After checking the logs, it seems that everything works as expected but since your changes also modified the results the permit urls got changed as well (for example check https://github.com/ubiquibot/conversation-rewards/actions/runs/10753678522/job/29832632935?pr=109#step:4:432) so you should just have to fix the result and every test should pass.

Ok that does make sense considering mock results are pre made.. I'm not sure why but I thought permits were being generated on the spot. I'll push changes to fix those issues when when I'm back from my class.

gentlementlegen · 2024-09-10T06:36:05Z

@cohow Seems that the tests are still failing. I would advise running them locally or on your own repo to avoid having to wait for me revalidating workflows every time.

cohow · 2024-09-11T21:29:42Z

ok I'm really shocked how many attempts this took for me to fix, but I finally fixed the tests I believe, tested locally and they passed.

cohow · 2024-09-12T04:29:02Z

Seems like the permit URLs have changed, i’ll get a fix out asap.

cohow · 2024-09-12T15:49:41Z

Quite lost on why I keep getting 2 different test results between when I run it locally and Github actions

for example https://github.com/ubiquibot/conversation-rewards/actions/runs/10832843625/job/30060543240#step:4:580 Its expecting a 1.57 but receives 1.232 and when I change them I get the exact opposite

            "relevance": 1,
-           "reward": 1.232,
+           "reward": 1.57,

gentlementlegen · 2024-09-12T15:52:24Z

This is probably because you are using your own credentials so the status you see for the other user differs from when Ubiquibot checks the author association, changing the reward results. Also, be aware that there are two results: one comes from the JSON and one comes from the HTML file.

gentlementlegen · 2024-09-12T16:36:13Z

@cohow Fixed the tests for you. also, seems to work, here is my QA: https://github.com/Meniole/conversation-rewards/issues/12#issuecomment-2346757261
However I think some explanation should be added to the result so the user understand what's going on in the results. @0x4007 rfc

0x4007 · 2024-09-12T16:37:48Z

@cohow Fixed the tests for you. also, seems to work, here is my QA: Meniole#12 (comment) However I think some explanation should be added to the result so the user understand what's going on in the results. @0x4007 rfc

Not sure if its easy to tell to be honest. How is somebody expected to manually count all their words and complain that there is a discrepancy? We can add a small blurb in the details table if somebody complains.

gentlementlegen · 2024-09-12T16:41:59Z

Maybe something very simple like adding coeff: 0.85 in the output? That would also show what value was used from the configuration to avoid bad surprises.

0x4007 · 2024-09-12T17:18:54Z

I dont think its something people will notice. We can add if its a problem. I'm just concerned it will lead to more confusion. And if we add too much info it will look bad.

gentlementlegen

If the display is fine as it is then I am good with the code changes.

gentlementlegen and others added 16 commits June 11, 2024 11:50

Merge pull request ubiquity-os-marketplace#35 from ubiquibot/development

2a0252a

chore: merge development into main

Merge pull request ubiquity-os-marketplace#47 from ubiquibot/development

ebc6c7b

Merge develop into main

Merge pull request ubiquity-os-marketplace#53 from ubiquibot/development

d0c3197

feat: split reward between multiple assignees

Merge pull request ubiquity-os-marketplace#70 from ubiquibot/development

a68ad6b

Merge develop into main

Merge pull request ubiquity-os-marketplace#80 from ubiquibot/development

3938625

Merge develop into main

chore(main): release 1.3.0

2ade82a

Merge pull request ubiquity-os-marketplace#71 from ubiquibot/release-…

2ba580b

…please--branches--main chore(main): release 1.3.0

Merge pull request ubiquity-os-marketplace#89 from ubiquibot/development

21b54b2

Merge development into main

Merge pull request ubiquity-os-marketplace#90 from ubiquibot/development

59391f2

feat: using latest ChatGpt version and fixed truncated results

chore(main): release 1.3.1

4be2fde

Merge pull request ubiquity-os-marketplace#91 from ubiquibot/release-…

a1ccb45

…please--branches--main chore(main): release 1.3.1

Merge pull request ubiquity-os-marketplace#99 from ubiquibot/development

dd3141c

Merge develop into main

Merge pull request ubiquity-os-marketplace#105 from ubiquibot/develop…

75fa9a2

…ment Merge development into main

chore(main): release 1.4.0

00d65b6

Merge pull request ubiquity-os-marketplace#100 from ubiquibot/release…

7f49264

…-please--branches--main chore(main): release 1.4.0

refactor(verbosity): better handling for verbosity

4e19255

0x4007 requested a review from gentlementlegen August 31, 2024 16:47

gentlementlegen requested review from Keyrxng and whilefoo September 1, 2024 03:24

cohow added 3 commits September 2, 2024 17:00

fix: fixed decimal issues with evaluation

1e30cd2

fix: edited the mock results to match new calculation module

a0d007c

fix: include forgotten reward-split calculation changes

b171bb6

whilefoo suggested changes Sep 2, 2024

View reviewed changes

gentlementlegen reviewed Sep 7, 2024

View reviewed changes

src/parser/formatting-evaluator-module.ts Outdated Show resolved Hide resolved

fix: refactored extra calculation and fixed mock results

834910e

fix: test permit related mock test result fixes

0b6f245

fix: fix github comment and reward split tests

3cee864

Merge branch 'development' into word-count

ff5c7ed

cohow added 3 commits September 12, 2024 13:16

merge branch 'development' into word-count

35ea7a6

fix: fixed issues with permit generation and mock results

aa3affa

fix: fix github comment and reward split tests

4e24f40

chore: fixed test results

9a1fd36

gentlementlegen approved these changes Sep 12, 2024

View reviewed changes

0x4007 merged commit 5bb7895 into ubiquity-os-marketplace:development Sep 12, 2024
3 checks passed

ubiquity-os bot mentioned this pull request Sep 12, 2024

Scoring Algorithm Verbosity Update #94

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor(verbosity): better handling for verbosity #109

refactor(verbosity): better handling for verbosity #109

cohow commented Aug 31, 2024

gentlementlegen commented Sep 1, 2024

cohow commented Sep 1, 2024

gentlementlegen commented Sep 2, 2024

cohow commented Sep 2, 2024

whilefoo Sep 2, 2024

0x4007 Sep 2, 2024

whilefoo Sep 3, 2024

0x4007 Sep 3, 2024 •

edited

Loading

whilefoo Sep 3, 2024 •

edited

Loading

0x4007 commented Sep 3, 2024

cohow commented Sep 3, 2024

0x4007 commented Sep 3, 2024

cohow commented Sep 8, 2024

gentlementlegen commented Sep 9, 2024

cohow commented Sep 9, 2024

gentlementlegen commented Sep 9, 2024

cohow commented Sep 9, 2024

gentlementlegen commented Sep 10, 2024

cohow commented Sep 11, 2024

cohow commented Sep 12, 2024

cohow commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

0x4007 commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

0x4007 commented Sep 12, 2024 •

edited

Loading

gentlementlegen left a comment

refactor(verbosity): better handling for verbosity #109

refactor(verbosity): better handling for verbosity #109

Conversation

cohow commented Aug 31, 2024

gentlementlegen commented Sep 1, 2024

cohow commented Sep 1, 2024

gentlementlegen commented Sep 2, 2024

cohow commented Sep 2, 2024

whilefoo Sep 2, 2024

Choose a reason for hiding this comment

0x4007 Sep 2, 2024

Choose a reason for hiding this comment

whilefoo Sep 3, 2024

Choose a reason for hiding this comment

0x4007 Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

whilefoo Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

0x4007 commented Sep 3, 2024

cohow commented Sep 3, 2024

0x4007 commented Sep 3, 2024

cohow commented Sep 8, 2024

gentlementlegen commented Sep 9, 2024

cohow commented Sep 9, 2024

gentlementlegen commented Sep 9, 2024

cohow commented Sep 9, 2024

gentlementlegen commented Sep 10, 2024

cohow commented Sep 11, 2024

cohow commented Sep 12, 2024

cohow commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

0x4007 commented Sep 12, 2024

gentlementlegen commented Sep 12, 2024

0x4007 commented Sep 12, 2024 • edited Loading

gentlementlegen left a comment

Choose a reason for hiding this comment

0x4007 Sep 3, 2024 •

edited

Loading

whilefoo Sep 3, 2024 •

edited

Loading

0x4007 commented Sep 12, 2024 •

edited

Loading