Improve non-English matching #374

GioSensation · 2023-08-28T13:21:11Z

Reviewer: https://app.asana.com/0/0/1205354344050186/f
Asana: https://app.asana.com/0/0/1204235865964101/f

Description

Improves the accuracy of our algo on non-English forms. Note: the diff looks huge, but 2900 lines is just for one test case file. The PR itself is not that complicated.

Steps to test

Will add some automated tests, but the bulk of the testing will be done by running the script using autofill-compare.

Signed-off-by: Emanuele Feliziani <[email protected]>

src/DeviceInterface/initFormSubmissionsApi.js

GioSensation · 2023-08-28T13:40:33Z

src/Form/FormAnalyzer.js

-const conservativeSignupRegex = new RegExp(/sign.?up|join|register|enroll|newsletter|subscri(be|ption)|settings|preferences|profile|update/i)
-const strictSignupRegex = new RegExp(/sign.?up|join|register|(create|new).+account|enroll|settings|preferences|profile|update/i)
-const resetPasswordLink = new RegExp(/(forgot(ten)?|reset|don't remember) (your )?password|password forgotten/i)
-const loginProvidersRegex = new RegExp(/ with /i)


Moved them all to the matching configuration, so they're centralized and easier to translated.

GioSensation · 2023-08-28T13:49:51Z

src/Form/FormAnalyzer.js

@@ -162,6 +153,24 @@ class FormAnalyzer {
        })
    }

+    evaluateUrl () {


Just another signal.

GioSensation · 2023-08-28T13:50:56Z

src/Form/FormAnalyzer.js

@@ -207,12 +216,13 @@ class FormAnalyzer {
                strength: 5,
                signalType: `explicit: ${el.getAttribute('autocomplete')}`
            })
+            return


I've added these returns, otherwise certain elements could be counted multiple times.

GioSensation · 2023-08-28T13:57:28Z

src/Form/matching-configuration.js

-            email: {
-                type: 'email',
+            unknown: {
+                type: 'unknown',
                strategies: {
-                    cssSelector: 'email',
-                    ddgMatcher: 'email',
+                    ddgMatcher: 'unknown'
+                }
+            },
+            emailAddress: {
+                type: 'emailAddress',
+                strategies: {
+                    cssSelector: 'emailAddress',
+                    ddgMatcher: 'emailAddress',


This is simpler to read if you hide whitespace, but anyway all I'm doing is:

Introduce an unknown matcher to ensure we return early when certain signals are obvious (like captchas). This has a minor performance impact, but most importantly avoids false-matching with the vendor strategy when we already know it should be unknwon (vendor strategies don't have the concept of forceUnknown).

Rename email to emailAddress because that's how we call it in the rest of the codebase, so this inconsistency just added confusion. If we ever want to go in and simplify all this stuff to only have one name, this brings us closer.

GioSensation · 2023-08-28T13:58:30Z

src/Form/matching-configuration.js

-                birthdayYear: css.__secret_do_not_use.birthdayYear
-            }
-        },
+        cssSelector: {selectors},


Now exporting the selectors directly with simpler naming.

GioSensation · 2023-08-28T13:59:32Z

src/Form/matching-configuration.js

+                unknown: {
+                    match: 'search|filter|subject|title|captcha|mfa|2fa|two factor|one-time|otp' +
+                        // Italian
+                        '|cerca|filtr|oggetto|titolo|(due|più) fattori',


I'm copying this approach from the vendors. It's just breaking these strings with a + and adding comments. We can add more complexity later if we ever need it.

src/Form/matching-configuration.js

src/Form/matching.js

Signed-off-by: Emanuele Feliziani <[email protected]>

Signed-off-by: Emanuele Feliziani <[email protected]> # Conflicts: # src/Scanner.js

Signed-off-by: Emanuele Feliziani <[email protected]>

Signed-off-by: Emanuele Feliziani <[email protected]> # Conflicts: # dist/autofill-debug.js # dist/autofill.js # src/Scanner.js # swift-package/Resources/assets/autofill-debug.js # swift-package/Resources/assets/autofill.js

Signed-off-by: Emanuele Feliziani <[email protected]>

… ema/improve-non-english-matching Signed-off-by: Emanuele Feliziani <[email protected]> # Conflicts: # dist/autofill-debug.js # dist/autofill.js # src/Scanner.js # src/autofill-utils.js # swift-package/Resources/assets/autofill-debug.js # swift-package/Resources/assets/autofill.js

Signed-off-by: Emanuele Feliziani <[email protected]>

GioSensation · 2023-09-19T10:24:44Z

I've pushed a bunch of commits, but most were actually only from pulling from the other branch (already merged). Not much has actually changed in the core of this PR. I'll go ahead and merge.

@GioSensation

Task/Issue URL: https://app.asana.com/0/1205538042573894/1205538042573894 Autofill Release: https://github.com/duckduckgo/duckduckgo-autofill/releases/tag/8.4.0 ## Description Updates Autofill to version [8.4.0](https://github.com/duckduckgo/duckduckgo-autofill/releases/tag/8.4.0). ### Autofill 8.4.0 release notes ## What's Changed * Improve non-English matching by @GioSensation in duckduckgo/duckduckgo-autofill#374 **Full Changelog**: duckduckgo/duckduckgo-autofill@8.3.0...8.4.0 ## Steps to test This release has been tested during autofill development. For smoke test steps see [this task](https://app.asana.com/0/1198964220583541/1200583647142330/f). Co-authored-by: GioSensation <[email protected]>

@GioSensation

Task/Issue URL: https://app.asana.com/0/1205538042573894/1205538042573894 Autofill Release: https://github.com/duckduckgo/duckduckgo-autofill/releases/tag/8.4.0 ## Description Updates Autofill to version [8.4.0](https://github.com/duckduckgo/duckduckgo-autofill/releases/tag/8.4.0). ### Autofill 8.4.0 release notes ## What's Changed * Improve non-English matching by @GioSensation in duckduckgo/duckduckgo-autofill#374 **Full Changelog**: duckduckgo/duckduckgo-autofill@8.3.0...8.4.0 ## Steps to test This release has been tested during autofill development. For smoke test steps see [this task](https://app.asana.com/0/1198964220583541/1200583647142330/f). Co-authored-by: GioSensation <[email protected]>

GioSensation added 17 commits August 22, 2023 09:34

Fix CSS selector naming

f9f8fe5

Signed-off-by: Emanuele Feliziani <[email protected]>

Move more regexes to config and centralise access

8faf88f

Signed-off-by: Emanuele Feliziani <[email protected]>

Minor type fix

a1b7112

Signed-off-by: Emanuele Feliziani <[email protected]>

Improve and rename getText to getTextShallow

176445a

Signed-off-by: Emanuele Feliziani <[email protected]>

Avoid evaluating buttons twice in FormAnalyzer

eaef31e

Signed-off-by: Emanuele Feliziani <[email protected]>

Evaluate URL when analyzing the form

e3dcdde

Signed-off-by: Emanuele Feliziani <[email protected]>

Centralise safeUniversalSelector

bf79aa5

Signed-off-by: Emanuele Feliziani <[email protected]>

Fix form scanning when markup is broken

4d28c3e

Signed-off-by: Emanuele Feliziani <[email protected]>

Simpler regex handling (no toLowerCase)

2ad2ac2

Signed-off-by: Emanuele Feliziani <[email protected]>

Minor accuracy improvement

8dcdf18

Signed-off-by: Emanuele Feliziani <[email protected]>

Rename email strategy for consistency

56cf736

Signed-off-by: Emanuele Feliziani <[email protected]>

Add/update test cases

9b351b8

Signed-off-by: Emanuele Feliziani <[email protected]>

Add performance logging capability

3215447

Signed-off-by: Emanuele Feliziani <[email protected]>

Remove strict/conservative regex distinction

12b886d

Signed-off-by: Emanuele Feliziani <[email protected]>

Log when site is disabled remotely

1ee12ef

Signed-off-by: Emanuele Feliziani <[email protected]>

Add the unknown matcher

f479ca7

Signed-off-by: Emanuele Feliziani <[email protected]>

Update fixed test

c5c6e01

Signed-off-by: Emanuele Feliziani <[email protected]>

GioSensation self-assigned this Aug 28, 2023