Revise READ UI/UX tracking #157
Assignees
Labels
Comments
|
Hi Jack,
Please see in-line below.Pranams to AMMA
_____________________________
From: JackMostow <[email protected]>
Sent: Wednesday, October 25, 2017 2:32 PM
Subject: [RoboTutorLLC/RoboTutor] Revise READ UI/UX tracking (#157)
To: RoboTutorLLC/RoboTutor <[email protected]>
Cc: Subscribed <[email protected]>
(When and why) should we adjust narration speed in HEAR and ECHO?
1.1. To [10%] faster than kid's recent reading pace after filtering out hesitations longer than [1 second]?
[Vishnu Rajan Tejus] If kids hesitate for longer than 1 sec, shouldn’t narration speed decrease instead of increasing by 10%? Should narration speed be adjusted when/if RoboTutor detects through gaze-point awareness that kids attempt to “catch up” with narration speed (deceleration) or read ahead (acceleration)?
1.2. To follow kid's finger through the text?
[Vishnu Rajan Tejus] Using computer vision or tapping? Following the kid’s eyes as they read text through gaze-point awareness can also help, and following the kid’s finger through taps can augment graceful degradation.
Gaze-point awareness is also a way to track attention, coupled with emotion for tracking hesitation, perhaps even being used to pause narration when the kid looks away.
+: intuitive
+: trains kid to track position in text
+: lets kids rehear
-: lets kids skip ahead (unless RoboFinger refuses)
+: gives kid agency
-: line boundary awkward
?: how adapt smoothly to kid's pace?
?: what if kid skips line?
?: should dragging work the same for kid's reading?
Let READ vary mode by sentence (or utterance?) as in Project LISTEN's Reading Tutor:
2.1. HEAR reads aloud to the kid.
2.2. READ listens to the kid read aloud.
2.3. ECHO listens to the kid read the sentence, then rereads it fluently.
2.4. REVEAL listens to the kid read, revealing each word when heard.
2.5. SAY speaks the text with VSS but without displaying the text., like As_spoken_only in Reading Tutor
Why did/does READ lag several words behind reader, at least at first?
How can we get READ’s “blame the kid” left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.
4.1. Can we reduce false rejections by recognizing visual speech?
[Vishnu Rajan Tejus] Yes, by 30% at minimum when compared to auditory speech recognition with non-vision based beam-forming.
4.2. When should READ tolerate a skipped word?
[Vishnu Rajan Tejus] Probably not, unless READ has prior knowledge that the kid knows that particular word very well, and is fluent with reading at his/her level.
4.3. How should we change READ's UI to encourage fluent reading?
[Vishnu Rajan Tejus] How would we measure fluent reading? I thought of 2 examples:
1. Measure reading speed with eye-tracking and determine “comfortability” through RoboFeel.
2. Measure reading speed and accuracy with visual speech recognition, VSR (and/or ASR).
4.3.1. Replace underline with RoboFinger.
[Vishnu Rajan Tejus] I agree! Adding RoboFinger will bring about a sense of continuity of RoboTutor’s Persona.
4.3.2. When kid is speaking, move RoboFinger through text at kid's recent pace.
[Vishnu Rajan Tejus] For recent pace, do we average out the kid’s reading pace at his/her current level vs. previous (easier) levels?
4.3.3. When kid hesitates, stop moving Robofinger.
[Vishnu Rajan Tejus] Neural network training will (considerably) help tracking hesitation with RoboFeel.
There are around 30 subconscious facial muscles that move during hesitation that can be tracking through depth mapping e.x. eyebrows moving, lips slightly twirling, eyes darting, nose twitching, and subtle mouth movements.
4.3.4. When kid skips >1 word, move RoboFinger back to first skipped word and turn it red.
[Vishnu Rajan Tejus] Sounds great!
4.3.5. When kid hesitates, start tapping RoboFinger on first skipped word instead of audio icon.
+: audio icon is unintuitive
+: tapping RoboFinger is intuitive cue to tap
[Vishnu Rajan Tejus] These interactions with RoboFinger can also help build rapport between kids and RoboPersona (given the current pace of RoboTutor’s technological development, I’m beginning to wonder what would happen if I removed the ‘a’ at the end of RoboPersona ;-).
4.3.6. After [3] seconds, say the word and advance RoboFinger to the next word.
[Vishnu Rajan Tejus] Should there be preventive measures to ensure that kids can’t simply start READ and wait for it to cycle through words every 3 sec i.e. finishing the activity without required input from kids?
Best regards, — Vishnu
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
From: Vishnu Rajan Tejus [mailto:[email protected]]
Sent: Thursday, October 26, 2017 2:15 AM
To: RoboTutorLLC/RoboTutor <[email protected]>
Cc: Jack Mostow <[email protected]>; Author <[email protected]>
Subject: Re: [RoboTutorLLC/RoboTutor] Revise READ UI/UX tracking (#157)
Hi Jack,
Please see in-line below.Pranams to AMMA
…_____________________________
From: JackMostow <[email protected]<mailto:[email protected]>>
Sent: Wednesday, October 25, 2017 2:32 PM
Subject: [RoboTutorLLC/RoboTutor] Revise READ UI/UX tracking (#157)
To: RoboTutorLLC/RoboTutor <[email protected]<mailto:[email protected]>>
Cc: Subscribed <[email protected]<mailto:[email protected]>>
(When and why) should we adjust narration speed in HEAR and ECHO?
1.1. To [10%] faster than kid's recent reading pace after filtering out hesitations longer than [1 second]?
[Vishnu Rajan Tejus] If kids hesitate for longer than 1 sec, shouldn’t narration speed decrease instead of increasing by 10%?
[Jack Mostow] 10% faster than kid’s recent oral reading pace isn’t necessarily faster than previous narration speed. The idea is to nudge the kid to read slightly faster.
Should narration speed be adjusted when/if RoboTutor detects through gaze-point awareness that kids attempt to “catch up” with narration speed (deceleration) or read ahead (acceleration)?
[Jack Mostow] Interesting idea. How?
a. React immediately.
-: Gaze point is likely to flit here and there in the text, so using it without smoothing could make narration jump around.
+: Immediate reaction makes behavior more obvious, hence easier for kids to adapt to.
b. Smoothed tracking.
+: Avoid discontinuities.
-: Smoothing could make RoboTutor behavior less obvious than reacting immediately.
c. Delayed reaction, e.g. move to word after kid gazes at it for [1] second.
+: Clear reaction could train kid to exploit gaze as gesture.
-: Non-immediate reaction may be less obvious than immediate reaction.
So I expect to need usability testing and iterative refinement.
1.2. To follow kid's finger through the text?
[Vishnu Rajan Tejus] Using computer vision or tapping?
[Jack Mostow] Dragging.
Following the kid’s eyes as they read text through gaze-point awareness can also help, and following the kid’s finger through taps can augment graceful degradation.
[Jack Mostow] Beware of feedback loops in which graphical actions, e.g. moving RoboFinger, draw kid’s gaze.
Gaze-point awareness is also a way to track attention, coupled with emotion for tracking hesitation, perhaps even being used to pause narration when the kid looks away.
[Jack Mostow] I like this idea – worth trying! But where should narration resume when kid looks back?
a. At pause point
+: clear behavior
-: less responsive
b. At new gaze point
+: responsive
?: how accurate is gaze point?
-: confusing to jump around
c. At pause point, but move to new point if gaze lingers there for [1] second
+: intuitive
+: trains kid to track position in text
+: lets kids rehear
-: lets kids skip ahead (unless RoboFinger refuses)
+: gives kid agency
-: line boundary awkward
?: how adapt smoothly to kid's pace?
?: what if kid skips line?
?: should dragging work the same for kid's reading?
Let READ vary mode by sentence (or utterance?) as in Project LISTEN's Reading Tutor:
2.1. HEAR reads aloud to the kid.
2.2. READ listens to the kid read aloud.
2.3. ECHO listens to the kid read the sentence, then rereads it fluently.
2.4. REVEAL listens to the kid read, revealing each word when heard.
2.5. SAY speaks the text with VSS but without displaying the text., like As_spoken_only in Reading Tutor
Why did/does READ lag several words behind reader, at least at first?
How can we get READ’s “blame the kid” left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.
4.1. Can we reduce false rejections by recognizing visual speech?
[Vishnu Rajan Tejus] Yes, by 30% at minimum when compared to auditory speech recognition with non-vision based beam-forming.
4.2. When should READ tolerate a skipped word?
[Vishnu Rajan Tejus] Probably not, unless READ has prior knowledge that the kid knows that particular word very well, and is fluent with reading at his/her level.
[Jack Mostow] False rejection rate currently deters fluent reading. So tolerating a single skipped word should help.
4.3. How should we change READ's UI to encourage fluent reading?
[Vishnu Rajan Tejus] How would we measure fluent reading? I thought of 2 examples:
1. Measure reading speed with eye-tracking and determine “comfortability” through RoboFeel.
2. Measure reading speed and accuracy with visual speech recognition, VSR (and/or ASR).
4.3.1. Replace underline with RoboFinger.
[Vishnu Rajan Tejus] I agree! Adding RoboFinger will bring about a sense of continuity of RoboTutor’s Persona.
4.3.2. When kid is speaking, move RoboFinger through text at kid's recent pace.
[Vishnu Rajan Tejus] For recent pace, do we average out the kid’s reading pace at his/her current level vs. previous (easier) levels?
[Jack Mostow] I meant recently in the current story.
4.3.3. When kid hesitates, stop moving Robofinger.
[Vishnu Rajan Tejus] Neural network training will (considerably) help tracking hesitation with RoboFeel.
[Jack Mostow] If RoboFeel includes gaze, should we rename it RoboFace?
There are around 30 subconscious facial muscles that move during hesitation that can be tracking through depth mapping e.x. eyebrows moving, lips slightly twirling, eyes darting, nose twitching, and subtle mouth movements.
4.3.4. When kid skips >1 word, move RoboFinger back to first skipped word and turn it red.
[Vishnu Rajan Tejus] Sounds great!
[Jack Mostow] This is the sort of UI/UX change that requires usability testing and tweaking.
If first “skipped” word was falsely rejected, turning it red will deter fluent reading. It may be less disruptive to speak the “skipped” word and move RoboFinger to the word after it to recue it.
4.3.5. When kid hesitates, start tapping RoboFinger on first skipped word instead of audio icon.
+: audio icon is unintuitive
+: tapping RoboFinger is intuitive cue to tap
[Jack Mostow] Should further hesitation make RoboFinger speak the word and move on?
+: maintain flow
-: could keep moving even if kid leaves – so condition on gaze
[Vishnu Rajan Tejus] These interactions with RoboFinger can also help build rapport between kids and RoboPersona (given the current pace of RoboTutor’s technological development, I’m beginning to wonder what would happen if I removed the ‘a’ at the end of RoboPersona ;-).
[Jack Mostow] Cute.
4.3.6. After [3] seconds, say the word and advance RoboFinger to the next word.
[Vishnu Rajan Tejus] Should there be preventive measures to ensure that kids can’t simply start READ and wait for it to cycle through words every 3 sec i.e. finishing the activity without required input from kids?
[Jack Mostow] At least we should pause reading while kid looks away from screen.
If we trust gaze point, RoboTutor could wait to say a word till the kid looks at it, i.e. use gaze rather than tap as required input. But what if the kid doesn’t look at the next word? Maybe:
1. If RoboTutor is reading the sentence narration, say next word as soon as kid looks at it.
2. If kid gazes at a word for [2] seconds, move there and say it.
+: intuitive
-: may skip words
3. Same as 2, but only if doesn’t skip any words.
4. If gaze skips words, flash the first skipped word to draw kid’s gaze to it, then resume narration.
5. If gaze regresses to an earlier word, resume narration from that point. This may work great or terribly, I’m not sure which.
Best regards, — Vishnu
[Jack Mostow] General question: how reliable and accurate is gaze point? If both, we can count on it; if not, we must engineer responses to be robust to gaze point error.
Thanks. – Jack
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#157 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AL3GWRrkkl0JHh81WLjw0PYQrbxYR0jbks5swCNugaJpZM4QGsTw>.
|
|
It's time to revisit this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(When and why) should we adjust narration speed in HEAR and ECHO?
1.1. To [10%] faster than kid's recent reading pace after filtering out hesitations longer than [1 second]?
1.2. To follow kid's finger through the text?
+: intuitive
+: trains kid to track position in text
+: lets kids rehear
-: lets kids skip ahead (unless RoboFinger refuses)
+: gives kid agency
-: line boundary awkward
?: how adapt smoothly to kid's pace?
?: what if kid skips line?
?: should dragging work the same for kid's reading?
Let READ vary mode by sentence (or utterance?) as in Project LISTEN's Reading Tutor:
2.1. HEAR reads aloud to the kid.
2.2. READ listens to the kid read aloud.
2.3. ECHO listens to the kid read the sentence, then rereads it fluently.
2.4. REVEAL listens to the kid read, revealing each word when heard.
2.5. SAY speaks the text with VSS but without displaying the text., like As_spoken_only in Reading Tutor
Why did/does READ lag several words behind reader, at least at first?
How can we get READ’s “blame the kid” left-to-right policy not to encourage kids to, read, one, word, at, a, time, unlike Project LISTEN’s Reading Tutor, which used “chase the kid” to track the kid through the text.
4.1. Can we reduce false rejections by recognizing visual speech?
4.2. When should READ tolerate a skipped word?
4.3. How should we change READ's UI to encourage fluent reading?
4.3.1. Replace underline with RoboFinger.
4.3.2. When kid is speaking, move RoboFinger through text at kid's recent pace.
4.3.3. When kid hesitates, stop moving Robofinger.
4.3.4. When kid skips >1 word, move RoboFinger back to first skipped word and turn it red.
4.3.5. When kid hesitates, start tapping RoboFinger on first skipped word instead of audio icon.
+: audio icon is unintuitive
+: tapping RoboFinger is intuitive cue to tap
4.3.6. After [3] seconds, say the word and advance RoboFinger to the next word.
The text was updated successfully, but these errors were encountered: