Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scroll on 'main scrollable element' when possible #405

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

seanmcguire12
Copy link
Collaborator

@seanmcguire12 seanmcguire12 commented Jan 14, 2025

why

  • currently, we only scroll on the root DOM element. thus, we ignore elements that are inside a scrollable container
  • this is problematic, because there may be other elements that we need to scroll into visibility so that they are included in the list of candidate elements

examples

Zillow:

Screenshot 2025-01-14 at 3 02 33 PM

apartments.com

Screenshot 2025-01-14 at 3 03 46 PM

what changed

  • updated processAllOfDom to attempt scrolling on a large scrollable container, and fall back to the original approach (scrolling on the root DOM) if there are no large scrollable containers

important note

  • this PR should only impact processAllOfDom
  • this means that we will still need a future PR to extend this logic to processDom to enable act to take actions on elements that are positioned deep within a scrollable container
  • the thought process behind this decision was to control the scope of this PR

test plan

  • added evals for Zillow & apartments.com

Copy link

changeset-bot bot commented Jan 14, 2025

⚠️ No Changeset found

Latest commit: cdc58b6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@seanmcguire12 seanmcguire12 added act These changes pertain to the act function extract These changes pertain to the extract function observe These changes pertain to the observe function combination These changes affect multiple Stagehand functions text-extract These changes pertain to text-extract labels Jan 14, 2025
@kamath kamath self-requested a review January 15, 2025 01:08
Copy link
Contributor

@kamath kamath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing is we probably need one unified abstract class called StagehandContainer that implements methods like getHeight(), scrollTo(), etc.

great work here, but these if statements aren't really sustainable as we keep adding to this kind of workflow


await stagehand.close();
const listings = real_estate_listings.listings;
const expectedLength = 38;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure this is fixed? this doesn't seem sustainable

@@ -19,7 +19,10 @@ declare global {
}>;
debugDom: () => Promise<void>;
cleanupDebug: () => void;
scrollToHeight: (height: number) => Promise<void>;
scrollToHeight: (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we make this serialized json

let mainScrollable: HTMLElement = document.documentElement;
let maxScrollableArea = 0;

// Find the largest candidate
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so this is basically just getting literally the biggest element and marking that main? using this approach should totally be opt-in; i feel like this can get funky super fast

// Verify we can actually scroll that candidate
if (mainScrollable !== document.documentElement) {
if (!canElementScroll(mainScrollable)) {
console.log(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

future note: we should consolidate DOM logs to a window.log function

export async function scrollToHeight(height: number) {
window.scrollTo({ top: height, left: 0, behavior: "smooth" });
export async function scrollToHeight(
containerOrWindow: HTMLElement | Window,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's name this differently and make it a custom type as well -- something like mainContainer: StagehandContainer, where StagehandContainer is HTMLElement | Window.

that way you can just do something like mainContainer.scrollTo({ top: height, left: 0, behavior: "smooth" }) without all these if else statements

clearTimeout(scrollEndTimer);
scrollEndTimer = window.setTimeout(() => {
window.removeEventListener("scroll", handleScrollEnd);
if (containerOrWindow instanceof Window) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not be an if statement; this should be one method

window.addEventListener("scroll", handleScrollEnd, { passive: true });
handleScrollEnd();
if (containerOrWindow instanceof Window) {
containerOrWindow.addEventListener("scroll", handleScroll, {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also not be in an if statement

let viewportHeight: number;
let totalScrollHeight: number;

if (container instanceof Window) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not an if statement

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
act These changes pertain to the act function combination These changes affect multiple Stagehand functions extract These changes pertain to the extract function observe These changes pertain to the observe function text-extract These changes pertain to text-extract
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants