Fix MyHeritage with new design #115

GuyKh · 2024-03-20T14:58:11Z

Still work in progress.... DO NOT MERGE YET

Solves #110

Update from master

Merge from master

eljeffeg · 2024-03-21T03:58:01Z

If it helps, here was my work in progress that I haven't had time to finish.
myheritage.js.zip

GuyKh · 2024-03-21T07:37:15Z

@eljeffeg
The main problem from what I can tell is the fact that the relative URLs are coming from GraphQL queries:
https://familygraphql.myheritage.com/profile_header_data/
https://familygraphql.myheritage.com/profile_details_data/

e.g. for https://www.myheritage.com/profile-420551751-1500056/dov-kushnir:

{
    "data": {
        "profile": {
            "individual": {
                "family_groups": [
                    {
                        "type": "parent",
                        "is_parent_family": true,
                        "father": {
                            "relationship_description": "His father",
                            "relationship_type": "father",
                            "individual": {
                                "id": "individual-420551751-1500056",
                                "name": "DOV KUSHNIR",
                                "gender": "M",
                                "age_group": "A",
                                "lifespan": "Died: 1943",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500056\/dov-kushnir"
                            }
                        },
                        "mother": {
                            "relationship_description": "His mother",
                            "relationship_type": "mother",
                            "individual": {
                                "id": "individual-420551751-1500057",
                                "name": "GITL KUSHNIR (\u05dc\u05d1\u05d9\u05ea GUTMAN)",
                                "gender": "F",
                                "age_group": "A",
                                "lifespan": "Died: 1943",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500057\/gitl-kushnir-%D7%9C%D7%91%D7%99%D7%AA-gutman"
                            }
                        },
                        "siblings": [
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500066",
                                    "name": "ZEEV KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "1911 - 2011",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500066\/zeev-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500060",
                                    "name": "ROCHL KUSHNER",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500060\/rochl-kushner"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500061",
                                    "name": "TAYBL TILLIE GOLDMAN (\u05dc\u05d1\u05d9\u05ea KUSHNER)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500061\/taybl-tillie-goldman-%D7%9C%D7%91%D7%99%D7%AA-kushner"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500062",
                                    "name": "AVRAHAM KUSHNER",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500062\/avraham-kushner"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500063",
                                    "name": "BARUCH KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500063\/baruch-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500064",
                                    "name": "SHAMAI KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Died: 1943",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500064\/shamai-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His brother",
                                "relationship_type": "brother",
                                "individual": {
                                    "id": "individual-420551751-1500065",
                                    "name": "HILEL KUSHNIR",
                                    "gender": "M",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500065\/hilel-kushnir"
                                }
                            },
                            {
                                "relationship_description": "His sister",
                                "relationship_type": "sister",
                                "individual": {
                                    "id": "individual-420551751-1500067",
                                    "name": "HANA RABINOVICH (\u05dc\u05d1\u05d9\u05ea KUSHNIR)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "Deceased",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500067\/hana-rabinovich-%D7%9C%D7%91%D7%99%D7%AA-kushnir"
                                }
                            }
                        ],
                        "spouse": null,
                        "children": null
                    },
                    {
                        "type": "spouse",
                        "is_parent_family": false,
                        "father": null,
                        "mother": null,
                        "siblings": null,
                        "spouse": {
                            "relationship_description": "His wife",
                            "relationship_type": "wife",
                            "individual": {
                                "id": "individual-420551751-1500014",
                                "name": "\u05d7\u05de\u05d3\u05d4 \u05e7\u05d5\u05e9\u05e0\u05d9\u05e8",
                                "gender": "F",
                                "age_group": "A",
                                "lifespan": "Deceased",
                                "personal_photo": null,
                                "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500014\/%D7%97%D7%9E%D7%93%D7%94-%D7%A7%D7%95%D7%A9%D7%A0%D7%99%D7%A8"
                            }
                        },
                        "children": [
                            {
                                "relationship_description": "His son",
                                "relationship_type": "son",
                                "individual": {
                                    "id": "individual-420551751-1500005",
                                    "name": "<Private> <Private>",
                                    "gender": "U",
                                    "age_group": "A",
                                    "lifespan": null,
                                    "personal_photo": null,
                                    "link_in_profile_page": "https:\/\/www.myheritage.com\/profile-420551751-1500005\/private"
                                }
                            }
                        ]
                    }
                ],
                "event_facts": {
                    "data": [
                        {
                            "id": "event-420551751-1500016",
                            "type": "BIRT",
                            "title": "Birth",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": "\u05e4\u05d5\u05dc\u05d9\u05df",
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-420551751-1500015"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        },
                        {
                            "id": "event-420551751-1500017",
                            "type": "DEAT",
                            "title": "Death",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": null,
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-420551751-1500015"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        }
                    ]
                },
                "insights": {
                    "confirmed_record_matches_summary": null,
                    "consistency_issues_summary": null,
                    "relative_hints": null
                },
                "map_pins": {
                    "data": [
                        {
                            "location": {
                                "name": "Poland",
                                "point": {
                                    "lat": 51.919438,
                                    "lng": 19.145136
                                },
                                "bounds": {
                                    "north_east": {
                                        "lat": 54.835784,
                                        "lng": 24.1458931
                                    },
                                    "south_west": {
                                        "lat": 49.0020252,
                                        "lng": 14.1228641
                                    }
                                }
                            },
                            "facts": {
                                "data": [
                                    {
                                        "id": "event-420551751-1500016",
                                        "is_fact_of_relative": false,
                                        "is_family_fact": false,
                                        "title": "Birth",
                                        "date": null,
                                        "formatted_place": "\u05e4\u05d5\u05dc\u05d9\u05df",
                                        "individual": {
                                            "id": "individual-420551751-1500015"
                                        },
                                        "relative": null,
                                        "spouse": null
                                    }
                                ]
                            }
                        }
                    ]
                }
            },
            "site_membership": null,
            "user": null,
            "birthday_greeting": null,
            "anniversary_greeting": null
        }
    }
}

Unfortunately, the GraphQL request is sent with a bearer_token so it'll be quite impossible to reproduce it, and also the url (i.e. link_in_profile_page in the response) is nowhere to be seen in the HTML)

GuyKh · 2024-03-21T08:10:12Z

Ideally speaking - we need to tap on these responses and everything you need is there but I'm not sure how to fetch this data from JQuery/HTML

GuyKh · 2024-03-21T08:10:45Z

I found this code:

        function Ty(e) {
            var t = e.individual
              , n = e.type
              , r = e.relationship;
            e.familyType;
            return Boolean(t) && a().createElement("div", {
                className: Ti(["family_relative", n]),
                onClick: function() {
                    K().location.href = t.link_in_profile_page
                }
            }

It seems that they generate the link on the onClick event. But I have no idea how to fetch that outside

GuyKh · 2024-03-21T08:13:20Z

I fixed various things -
One thing that's missing, I think is relatives Birth/Death which is being ignored today. and the freaking URL (which is a blocker) :/

GuyKh · 2024-03-26T06:44:07Z

OK @eljeffeg , I think we're in a whole new realm here.
Using React DevTools I was able to find this React Component hierarchy:

As you can see Fragment key=family_0 is the whole "Immediate Family" list,
where for each sub-component (including the parent) - there's a component (Cy - but I guess the name may change) which contains a prop with all the data - name, relation and link.

The question now is... it seems that we can't just fetch this from the HTML itself (as the logic happens today) - and some React analysis is required.

Here's what Google Gemini said:

Q: I'm developing a chrome extension. To show some data, I need to query the react component tree and find a Fragment component by key and fetch it's props. How to do that?
A:
There are two main approaches to achieve this functionality in a Chrome extension:

Content scripts:

Content scripts are JavaScript programs that run directly within the webpage's DOM. They can access and modify the webpage's content.

Here's how you can use a content script to achieve your goal:
   - **Inject a script:**  Use the `chrome.tabs.executeScript` API to inject a content script into the webpage where you want to query the React component tree.

   - **Traverse the React tree:**  Within the content script, you can access the React component hierarchy using the browser's developer tools API (`chrome.devtools.inspectedWindow.evaluate`). This API allows you to execute code within the context of the webpage.

   - **Find Fragment by key:**  Use the `document.querySelector` function (or similar methods) to navigate the component tree based on the provided key. Keep in mind that directly querying React components by key might not always be reliable, as keys are internal implementation details.

   - **Fetch props:**  Once you have a reference to the Fragment component, you can access its props using the standard JavaScript property access syntax (e.g., `component.props.propName`).
Background script and devtools extension:

Background scripts run in the background process of your extension and can communicate with webpages and other scripts using message passing.

Devtools extensions leverage the Chrome DevTools Extension API to interact with the browser's developer tools.

Here's how you can use this approach:
    - **Background script communication:**  The devtools panel can communicate with a background script using the `runtime.onMessage` API.

    - **Content script injection:**  The background script can inject a content script similar to the first approach, but with the key information received from the devtools panel.

    - **Data retrieval and response:**  The content script retrieves the Fragment's props and sends them back to the background script using message passing.

    - **Display data in devtools panel:**  The background script receives the data from the content script and updates the devtools panel UI to display the retrieved props.
Important considerations:

Content scripts can be limited in their ability to access and modify certain parts of the webpage due to security restrictions.

Directly querying React components by key is not the most reliable approach as keys might not be unique or publicly exposed.

Consider using a library like react-devtools (if allowed by Chrome extension guidelines) to interact with the React component tree in a more robust way.

Remember that modifying the DOM or interacting with webpages requires careful consideration of security implications and potential conflicts with other extensions. Make sure your extension adheres to Chrome extension development guidelines.

eljeffeg · 2024-04-07T21:32:11Z

Yeah, I see what you mean with https://familygraphql.myheritage.com. That's suppose to be MyHeritage's API - wonder if it's still an accessible option. For the react, a content script probably wouldn't work for this part of SmartCopy, but a background script might. We do have a background script that runs from the popup in background.js, which can make xhr requests and such. Perhaps that would be a way to get the data.

eljeffeg · 2024-04-07T21:33:55Z

What you have is a lot better than the failure now, so I wonder if we should just commit that and let everyone know we're still working out the dates. We might be able to at least put the years in there, since that appears to be in the data.

GuyKh · 2024-04-08T05:17:35Z

@eljeffeg - the problem is not the dates rather than the urls. Without it, SmartCopy (atleast today's implementation) doesn't "recognize" the family members - which is, I think, a crucial part.

I'd definitely leave the dates out, but in my opinion, the URLs are a must....

GuyKh · 2024-04-08T05:28:16Z

In a perfect world, I'd somehow reuse the bearer token and call the GraphQL myself (rather than parsing the HTML) and fetch all the data from there.

However, couple of thoughts on this:

There's a MyHeritage rate limiter (unfortunately I've seen it wayyyy to many times) - so calling APIs a second time would just bring us closer to the limit
Not sure how trivial it is to fetch the bearer token since the extension doesn't monitor the network calls....
Perhaps some data is being fetched from different APIs or so. So a single (or even 2) call(s) to the GraphQL API won't be enough.

On the other hand, I have no clear idea about accessing React Component's state from within an Extension.

alvestrand · 2024-04-08T07:30:48Z

Commenting without too much knowledge ... this is about translating the list of family members into queries that fetch the information about the family members? Is the URL you need the one for the family member's page?

Wild thought - is it possible to just follow the relative links from within the page and intercept the queries so that you can figure out what the complete URL has become? (the interception could also return a 5xx code to avoid bumping the rate limiter)? Or is the result of following the relative link a different URL from the one you need?

GuyKh · 2024-04-08T08:28:54Z

this is about translating the list of family members into queries that fetch the information about the family members? Is the URL you need the one for the family member's page?

The problem here is how to get the links - the way this is implemented is through React Components, when the page is built - each family member in the list has a onClick event hander which sends the page to the relevant link (when the user clicks anywhere on the compnent).
This is unlike the previous (and standard) behavior where an object has a <a href... tag - which is easily "read" when parsing the HTML.

The trouble here is how to get these links... React BS...

GuyKh · 2024-05-15T07:15:51Z

See this
and this commit specifically

GuyKh · 2024-05-18T04:45:27Z

@eljeffeg as a temp solution. We can set the url as the current page url for all family members, so they'll be added but referenced to the source person and not the actual person. WDYT?

alvestrand · 2024-05-24T09:25:13Z

If we can add "referenced in " to the text of the URL in family members, that would make it clear what's happening.
I'd love to have my MyHeritage copy tool back soon!

alvestrand · 2024-07-17T09:39:19Z

any chance of getting this merged? I'm still hurting from the lack of MyHeritage SmartCopy.

GuyKh · 2024-07-17T10:22:03Z

Still hoping to get it working with actual URLs.

eljeffeg · 2024-08-04T16:58:54Z

As we update the parsers for particular sites, we may want to consider implementing a model based on beautifulsoup. https://beautiful-soup-4.readthedocs.io/en/latest/

GuyKh · 2024-08-04T17:03:47Z

@eljeffeg any references in the project?

Isn't beautiful soup is a python tool?

eljeffeg · 2024-08-04T18:38:39Z

Ha, you're right. I work with Python so much I forgot that this project is pretty much just JS.

GuyKh · 2024-09-29T07:52:07Z

OK. I give up. I'll set the url field as "" and let's see what happens

alvestrand · 2024-10-30T07:47:31Z

Is this ready to merge, then?

GuyKh · 2024-10-30T08:36:37Z

@alvestrand - this won't do, I'm afraid.
Still looking for a solution...

GuyKh · 2024-11-06T16:25:45Z

OK @eljeffeg , I think we're in a whole new realm here. Using React DevTools I was able to find this React Component hierarchy:

As you can see Fragment key=family_0 is the whole "Immediate Family" list, where for each sub-component (including the parent) - there's a component (Cy - but I guess the name may change) which contains a prop with all the data - name, relation and link.

The question now is... it seems that we can't just fetch this from the HTML itself (as the logic happens today) - and some React analysis is required.

Here's what Google Gemini said:

Q: I'm developing a chrome extension. To show some data, I need to query the react component tree and find a Fragment component by key and fetch it's props. How to do that?
A:
There are two main approaches to achieve this functionality in a Chrome extension:

Content scripts:
Content scripts are JavaScript programs that run directly within the webpage's DOM. They can access and modify the webpage's content.

Here's how you can use a content script to achieve your goal:
   - **Inject a script:**  Use the `chrome.tabs.executeScript` API to inject a content script into the webpage where you want to query the React component tree.

   - **Traverse the React tree:**  Within the content script, you can access the React component hierarchy using the browser's developer tools API (`chrome.devtools.inspectedWindow.evaluate`). This API allows you to execute code within the context of the webpage.

   - **Find Fragment by key:**  Use the `document.querySelector` function (or similar methods) to navigate the component tree based on the provided key. Keep in mind that directly querying React components by key might not always be reliable, as keys are internal implementation details.

   - **Fetch props:**  Once you have a reference to the Fragment component, you can access its props using the standard JavaScript property access syntax (e.g., `component.props.propName`).
Background script and devtools extension:

Background scripts run in the background process of your extension and can communicate with webpages and other scripts using message passing.

Devtools extensions leverage the Chrome DevTools Extension API to interact with the browser's developer tools.

Here's how you can use this approach:
    - **Background script communication:**  The devtools panel can communicate with a background script using the `runtime.onMessage` API.

    - **Content script injection:**  The background script can inject a content script similar to the first approach, but with the key information received from the devtools panel.

    - **Data retrieval and response:**  The content script retrieves the Fragment's props and sends them back to the background script using message passing.

    - **Display data in devtools panel:**  The background script receives the data from the content script and updates the devtools panel UI to display the retrieved props.
Important considerations:

Content scripts can be limited in their ability to access and modify certain parts of the webpage due to security restrictions.

Directly querying React components by key is not the most reliable approach as keys might not be unique or publicly exposed.

Consider using a library like react-devtools (if allowed by Chrome extension guidelines) to interact with the React component tree in a more robust way.

Remember that modifying the DOM or interacting with webpages requires careful consideration of security implications and potential conflicts with other extensions. Make sure your extension adheres to Chrome extension development guidelines.

Structure was change a bit:

Still no luck :(

GuyKh · 2024-11-06T16:33:36Z

I see that when loading - the page does a GraphQL Query:

This is for profile-171591352-1500891 and requires a Bearer Token

Request to **https://familygraphql.myheritage.com/profile_header_data**

Request

curl --location 'https://familygraphql.myheritage.com/profile_header_data/' \
--header 'accept: application/json, text/plain, */*' \
--header 'accept-language: en-US,en;q=0.9,fr;q=0.8,he;q=0.7' \
--header 'origin: https://www.myheritage.com' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
--form 'bearer_token="4.179cNOTTODAY"' \
--form 'query="\"{profile(id:\\\"profile-171591352-1500891-0\\\",lang:\\\"EN\\\"){name first_name last_name gender age_group age{text}personal_photo{...personal_photo_fragment}is_prefer_user can_current_user_view_discoveries can_current_user_manage_photos can_current_user_edit_personal_photo can_current_user_invite_individual tabs{name total counters}recent_individuals{data{...history_fragment}}favorite_individuals{data{...history_fragment}}individual{...individual_fragment}user{...user_fragment}site_membership{member_id site_id can_user_contact_member member_joined_date member_last_visit_date role_sentence{text}}tree{is_imported_using_family_search_sync}site{name}}}fragment history_fragment on Individual{id name gender age_group lifespan personal_photo{...personal_photo_fragment}tree_relationship{description}link_in_profile_page}fragment individual_fragment on Individual{id name first_name gender personal_photo{...personal_photo_fragment}is_privatized religious_name former_name namesake alternate_names birth_date{text}birth_place death_date{text}is_alive is_likely_deceased death_place burial_place cause_of_death is_cause_of_death_holocaust age{text}tree_relationship{description is_blood_relative is_biological_blood_relative blood_relative_description hour_glass_color_code is_path_to_self is_path_cannot_decide_if_related}can_edit link_in_tree link_in_pedigree_tree link_in_fan_view link_in_research_this_person link_template_in_edit_profile}fragment personal_photo_fragment on Photo{thumbnails(thumbnail_size:\\\"136x136c\\\"){url}}fragment user_fragment on User{name first_name crown_status country country_code birth_date{text}age{text}age_group_in_years show_age is_public is_privatized nickname created_time}\""' \
--form 'operation=""' \
--form 'variables=""' \
--form 'description="profile header data"'

Response

{
    "data": {
        "profile": {
            "name": "שושנה רוזנפלד (לבית ימפולסקי)",
            "first_name": "שושנה",
            "last_name": "ימפולסקי",
            "gender": "F",
            "age_group": "A",
            "age": null,
            "personal_photo": null,
            "is_prefer_user": false,
            "can_current_user_view_discoveries": false,
            "can_current_user_manage_photos": false,
            "can_current_user_edit_personal_photo": false,
            "can_current_user_invite_individual": false,
            "tabs": [
                {
                    "name": "details",
                    "total": 0,
                    "counters": null
                },
                {
                    "name": "biography",
                    "total": 0,
                    "counters": null
                },
                {
                    "name": "sources",
                    "total": 1,
                    "counters": null
                }
            ],
            "recent_individuals": {
                "data": [
                    {
                        "id": "individual-171591352-1500890",
                        "name": "אהרון רוזנפלד",
                        "gender": "M",
                        "age_group": "A",
                        "lifespan": "Deceased",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": "Great-grandfather's half brother's father-in-law (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1500890/%D7%90%D7%94%D7%A8%D7%95%D7%9F-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                    },
                    {
                        "id": "individual-171591352-1501337",
                        "name": "Efraim Rozenfld",
                        "gender": "M",
                        "age_group": "A",
                        "lifespan": "1876 - 1960",
                        "personal_photo": {
                            "thumbnails": [
                                {
                                    "url": "https://sites-cf.mhcache.com/t/591/352/171591352/501/501578_9778695jfa3598ge725b6u_V_111x111C/dcVCgF5LyBZt8E0Z66CxzH-FUV8=/.jpg"
                                }
                            ]
                        },
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1501337/efraim-rozenfld"
                    },
                    {
                        "id": "individual-171591352-1501958",
                        "name": "Shoshana Rosenfeld (לבית Yampolski)",
                        "gender": "F",
                        "age_group": "A",
                        "lifespan": "Deceased",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1501958/shoshana-rosenfeld-%D7%9C%D7%91%D7%99%D7%AA-yampolski"
                    },
                    {
                        "id": "individual-171591352-1000227",
                        "name": "בלה אלברט (לבית רוזנפלד)",
                        "gender": "F",
                        "age_group": "A",
                        "lifespan": "1874 - 1958",
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": "Great-grandfather's half brother's wife (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                    },
                    {
                        "id": "individual-171591352-1000626",
                        "name": "&lt;Private&gt; nagid",
                        "gender": "U",
                        "age_group": "A",
                        "lifespan": null,
                        "personal_photo": null,
                        "tree_relationship": {
                            "description": " (of site manager)"
                        },
                        "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000626/private-nagid"
                    }
                ]
            },
            "favorite_individuals": {
                "data": null
            },
            "individual": {
                "id": "individual-171591352-1500891",
                "name": "שושנה רוזנפלד (לבית ימפולסקי)",
                "first_name": "שושנה",
                "gender": "F",
                "personal_photo": null,
                "is_privatized": false,
                "religious_name": null,
                "former_name": null,
                "namesake": null,
                "alternate_names": null,
                "birth_date": null,
                "birth_place": null,
                "death_date": null,
                "is_alive": false,
                "is_likely_deceased": true,
                "death_place": null,
                "burial_place": null,
                "cause_of_death": null,
                "is_cause_of_death_holocaust": false,
                "age": null,
                "tree_relationship": {
                    "description": "Great-grandfather's half brother's mother-in-law (of site manager)",
                    "is_blood_relative": false,
                    "is_biological_blood_relative": false,
                    "blood_relative_description": null,
                    "hour_glass_color_code": null,
                    "is_path_to_self": false,
                    "is_path_cannot_decide_if_related": false
                },
                "can_edit": false,
                "link_in_tree": "https://www.myheritage.com/site-family-tree-171591352/lerner?rootIndividualID=1500891&familyTreeID=1",
                "link_in_pedigree_tree": "https://www.myheritage.com/pedigree-tree-171591352-1500891/lerner?familyTreeID=1",
                "link_in_fan_view": "https://www.myheritage.com/fan-view-171591352-1500891/lerner?familyTreeID=1",
                "link_in_research_this_person": "https://www.myheritage.com/research/individual-1500891/?s=171591352&rfr=profile_page",
                "link_template_in_edit_profile": null
            },
            "user": null,
            "site_membership": null,
            "tree": {
                "is_imported_using_family_search_sync": false
            },
            "site": {
                "name": "Lerner Web Site"
            }
        }
    }
}

Request to **https://familygraphql.myheritage.com/profile_details_data**

Request

curl --location 'https://familygraphql.myheritage.com/profile_details_data/' \
--header 'accept: application/json, text/plain, */*' \
--header 'accept-language: en-US,en;q=0.9,fr;q=0.8,he;q=0.7' \
--header 'origin: https://www.myheritage.com' \
--header 'referer: https://www.myheritage.com/' \
--header 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36' \
--form 'bearer_token="4.179c5NotToday"' \
--form 'query="\"{profile(id:\\\"profile-171591352-1500891-0\\\",lang:\\\"EN\\\"){individual{family_groups(relationship_prefix:\\\"auto\\\"){type is_parent_family father{...family_member_fragment}mother{...family_member_fragment}siblings(include_half_siblings:true){...family_member_fragment}spouse{...family_member_fragment}children{...family_member_fragment}}event_facts(hints:3){data{...fact_fragment}}insights{confirmed_record_matches_summary{...insight_summary_fragment}consistency_issues_summary{...insight_summary_fragment}relative_hints{...hint_fragment}}map_pins{data{...map_pin_fragment}}}site_membership{...site_membership_fragment}user{surname_research}birthday_greeting{...greeting_fragment}anniversary_greeting{...greeting_fragment}}}fragment fact_fragment on Fact{id type title is_family_fact is_fact_of_relative date{text}year formatted_age formatted_place cause_of_death content additional_content individual{id}relative{...fact_relative_fragment}spouse{...fact_relative_fragment}hint{...hint_fragment}citations{data{...citation_fragment}}notes{data{...note_fragment}}media{data{name link thumbnails(thumbnail_size:\\\"96x96c\\\"){url}}}}fragment citation_fragment on Citation{id page confidence event{id title}family_event{id title}date{text}formatted_text page_link{url name image}source{name smart_matching_site{id}image link}extended_citation{reference comment reason}smart_matching_individual{id name}}fragment note_fragment on Note{id type text subject body}fragment family_member_fragment on Relationship{relationship_description relationship_type individual{id name gender age_group lifespan personal_photo{...personal_photo_fragment}link_in_profile_page}}fragment personal_photo_fragment on Photo{thumbnails(thumbnail_size:\\\"136x136c\\\"){url}}fragment fact_relative_fragment on Individual{id name gender age_group personal_photo{...personal_photo_fragment}link_in_profile_page}fragment insight_summary_fragment on InsightSummary{type status count link is_accessible fields{id label value}}fragment hint_fragment on InsightHint{factor key modifier count first_source_name image}fragment map_pin_fragment on FactMapPin{location{name point{lat lng}bounds{north_east{lat lng}south_west{lat lng}}}facts{data{id is_fact_of_relative is_family_fact title date{text}formatted_place individual{id}relative{id name}spouse{name}}}}fragment sentence_fragment on StorySentence{text tokens{type text value link}}fragment site_membership_fragment on ProfileSiteMembership{member_id member_gender site_id site_creator_id role_sentence{...sentence_fragment}visit_sentence{...sentence_fragment}join_sentence{...sentence_fragment}request_sentence{...sentence_fragment}is_current_user_member_in_site can_user_contact_member can_user_contact_site_manager can_user_promote_member_to_site_manager can_user_demote_member_from_site_manager can_user_remind_member_to_visit can_user_change_member_email_for_remind_to_visit can_user_review_membership_request review_membership_request_link can_user_remove_member_from_site can_user_identify_member_in_tree can_user_edit_member_profile edit_member_profile_link can_user_edit_member_site_preferences edit_member_site_preferences_link can_user_edit_member_privacy_preferences edit_member_privacy_preferences_link can_user_change_member_email_and_password change_member_email_and_password_link can_user_view_member_public_profile view_member_public_profile_link can_user_associate_member_in_tree other_site_memberships{data{site_name site_link role}}}fragment greeting_fragment on ProfileGreeting{type date title label link}\""' \
--form 'operation=""' \
--form 'variables=""' \
--form 'description="profile details data"'

Response

{
    "data": {
        "profile": {
            "individual": {
                "family_groups": [
                    {
                        "type": "spouse",
                        "is_parent_family": false,
                        "father": null,
                        "mother": null,
                        "siblings": null,
                        "spouse": {
                            "relationship_description": "Her husband",
                            "relationship_type": "husband",
                            "individual": {
                                "id": "individual-171591352-1500890",
                                "name": "אהרון רוזנפלד",
                                "gender": "M",
                                "age_group": "A",
                                "lifespan": "Deceased",
                                "personal_photo": null,
                                "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1500890/%D7%90%D7%94%D7%A8%D7%95%D7%9F-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                            }
                        },
                        "children": [
                            {
                                "relationship_description": "Her daughter",
                                "relationship_type": "daughter",
                                "individual": {
                                    "id": "individual-171591352-1000227",
                                    "name": "בלה אלברט (לבית רוזנפלד)",
                                    "gender": "F",
                                    "age_group": "A",
                                    "lifespan": "1874 - 1958",
                                    "personal_photo": null,
                                    "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                                }
                            }
                        ]
                    }
                ],
                "event_facts": {
                    "data": [
                        {
                            "id": "event-171591352-1000251",
                            "type": "BIRT",
                            "title": "Birth of daughter:",
                            "is_family_fact": false,
                            "is_fact_of_relative": true,
                            "date": {
                                "text": "1874"
                            },
                            "year": "1874",
                            "formatted_age": null,
                            "formatted_place": "סבסטופול, רוסיה",
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-171591352-1500891"
                            },
                            "relative": {
                                "id": "individual-171591352-1000227",
                                "name": "בלה אלברט (לבית רוזנפלד)",
                                "gender": "F",
                                "age_group": "A",
                                "personal_photo": null,
                                "link_in_profile_page": "https://www.myheritage.com/profile-171591352-1000227/%D7%91%D7%9C%D7%94-%D7%90%D7%9C%D7%91%D7%A8%D7%98-%D7%9C%D7%91%D7%99%D7%AA-%D7%A8%D7%95%D7%96%D7%A0%D7%A4%D7%9C%D7%93"
                            },
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        },
                        {
                            "id": "event-171591352-1501837",
                            "type": "DEAT",
                            "title": "Death",
                            "is_family_fact": false,
                            "is_fact_of_relative": false,
                            "date": null,
                            "year": null,
                            "formatted_age": null,
                            "formatted_place": null,
                            "cause_of_death": null,
                            "content": null,
                            "additional_content": null,
                            "individual": {
                                "id": "individual-171591352-1500891"
                            },
                            "relative": null,
                            "spouse": null,
                            "hint": null,
                            "citations": {
                                "data": null
                            },
                            "notes": {
                                "data": null
                            },
                            "media": {
                                "data": null
                            }
                        }
                    ]
                },
                "insights": {
                    "confirmed_record_matches_summary": null,
                    "consistency_issues_summary": null,
                    "relative_hints": null
                },
                "map_pins": {
                    "data": [
                        {
                            "location": {
                                "name": "Russia",
                                "point": {
                                    "lat": 61.52401,
                                    "lng": 105.318756
                                },
                                "bounds": {
                                    "north_east": {
                                        "lat": 70,
                                        "lng": 179
                                    },
                                    "south_west": {
                                        "lat": 40,
                                        "lng": 27
                                    }
                                }
                            },
                            "facts": {
                                "data": [
                                    {
                                        "id": "event-171591352-1000251",
                                        "is_fact_of_relative": true,
                                        "is_family_fact": false,
                                        "title": "Birth of daughter:",
                                        "date": {
                                            "text": "1874"
                                        },
                                        "formatted_place": "סבסטופול, רוסיה",
                                        "individual": {
                                            "id": "individual-171591352-1500891"
                                        },
                                        "relative": {
                                            "id": "individual-171591352-1000227",
                                            "name": "בלה אלברט (לבית רוזנפלד)"
                                        },
                                        "spouse": null
                                    }
                                ]
                            }
                        }
                    ]
                }
            },
            "site_membership": null,
            "user": null,
            "birthday_greeting": null,
            "anniversary_greeting": null
        }
    }
}

This could be a potential to use it - but we need to extract the BearerToken from the page somehow

eljeffeg · 2024-11-11T01:23:12Z

Yeah, if you can use the background script, it's not too hard to call listener functions in that and wait for the response.

GuyKh · 2024-11-11T07:06:02Z

Yeah, if you can use the background script, it's not too hard to call listener functions in that and wait for the response.

I'd love to see an example

GuyKh and others added 8 commits March 29, 2021 10:54

Merge pull request #1 from jeffg2k/master

231b8c9

Update from master

Merge pull request #2 from jeffg2k/master

1f9a6bc

Merge from master

Merge branch 'jeffg2k:master' into master

2d5ac57

Merge branch 'eljeffeg:master' into master

258637f

Merge branch 'eljeffeg:master' into master

364b3a1

New MyHeritage - Get PersonName, Gender and Photo

8a6249d

New MyHeritage - Parse Birth/Death Dates

ac088c6

New MyHeritage - Parse family members (without URL)

e9afc29

New MyHeritage - Fix gender parsing and family

e81d694

GuyKh referenced this pull request in RobPavey/wikitree-sourcer May 16, 2024

mh: update for format change to profile pages

5afabaf

CyDefender mentioned this pull request May 17, 2024

New Billion Graves design update #117

Merged

nlof mentioned this pull request Aug 22, 2024

Newest update 4.11.0 broke most remaining SC functions #127

Closed

Merge branch 'eljeffeg:master' into fix-my-heritage

80cd55c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MyHeritage with new design #115

Fix MyHeritage with new design #115

GuyKh commented Mar 20, 2024 •

edited

Loading

eljeffeg commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024 •

edited

Loading

GuyKh commented Mar 26, 2024

eljeffeg commented Apr 7, 2024

eljeffeg commented Apr 7, 2024

GuyKh commented Apr 8, 2024

GuyKh commented Apr 8, 2024

alvestrand commented Apr 8, 2024

GuyKh commented Apr 8, 2024 •

edited

Loading

GuyKh commented May 15, 2024 •

edited

Loading

GuyKh commented May 18, 2024

alvestrand commented May 24, 2024

alvestrand commented Jul 17, 2024

GuyKh commented Jul 17, 2024

eljeffeg commented Aug 4, 2024

GuyKh commented Aug 4, 2024 •

edited

Loading

eljeffeg commented Aug 4, 2024

GuyKh commented Sep 29, 2024

alvestrand commented Oct 30, 2024

GuyKh commented Oct 30, 2024

GuyKh commented Nov 6, 2024

GuyKh commented Nov 6, 2024 •

edited

Loading

Request

Response

Request

Response

eljeffeg commented Nov 11, 2024

GuyKh commented Nov 11, 2024

Fix MyHeritage with new design #115

Are you sure you want to change the base?

Fix MyHeritage with new design #115

Conversation

GuyKh commented Mar 20, 2024 • edited Loading

eljeffeg commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024

GuyKh commented Mar 21, 2024 • edited Loading

GuyKh commented Mar 26, 2024

eljeffeg commented Apr 7, 2024

eljeffeg commented Apr 7, 2024

GuyKh commented Apr 8, 2024

GuyKh commented Apr 8, 2024

alvestrand commented Apr 8, 2024

GuyKh commented Apr 8, 2024 • edited Loading

GuyKh commented May 15, 2024 • edited Loading

GuyKh commented May 18, 2024

alvestrand commented May 24, 2024

alvestrand commented Jul 17, 2024

GuyKh commented Jul 17, 2024

eljeffeg commented Aug 4, 2024

GuyKh commented Aug 4, 2024 • edited Loading

eljeffeg commented Aug 4, 2024

GuyKh commented Sep 29, 2024

alvestrand commented Oct 30, 2024

GuyKh commented Oct 30, 2024

GuyKh commented Nov 6, 2024

GuyKh commented Nov 6, 2024 • edited Loading

Request

Response

Request

Response

eljeffeg commented Nov 11, 2024

GuyKh commented Nov 11, 2024

GuyKh commented Mar 20, 2024 •

edited

Loading

GuyKh commented Mar 21, 2024 •

edited

Loading

GuyKh commented Apr 8, 2024 •

edited

Loading

GuyKh commented May 15, 2024 •

edited

Loading

GuyKh commented Aug 4, 2024 •

edited

Loading

GuyKh commented Nov 6, 2024 •

edited

Loading