Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle Observable Collections #2

Open
hrbrmstr opened this issue Aug 20, 2022 · 0 comments
Open

Handle Observable Collections #2

hrbrmstr opened this issue Aug 20, 2022 · 0 comments
Assignees
@hrbrmstr
Labels
enhancement New feature or request

Comments

@hrbrmstr
Copy link
Owner

hrbrmstr commented Aug 20, 2022 •
edited
Loading

It appears that the collection info is also present in the embedded JSON:

Example Collections JSON 
            "initialCollection":
            {
                "id": "5610fa9dc3f1eab0",
                "type": "public",
                "slug": "tidy-js",
                "title": "tidy.js",
                "description": "Examples of using tidy.js to wrangle data",
                "update_time": "2021-02-24T22:18:57.641Z",
                "pinned": false,
                "ordered": false,
                "custom_thumbnail": null,
                "default_thumbnail": "42469726dbbe2a563c5e6e8e7e7dc36399a5d06ee2d97eec85c7c2b85fcef306",
                "thumbnail": "42469726dbbe2a563c5e6e8e7e7dc36399a5d06ee2d97eec85c7c2b85fcef306",
                "listing_count": 6,
                "owner":
                {
                    "id": "26ab57b8a43a5dda",
                    "github_login": "pbeshai",
                    "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                    "login": "pbeshai",
                    "name": "Peter Beshai",
                    "bio": "",
                    "home_url": "http://peterbeshai.com",
                    "type": "individual",
                    "tier": "basic"
                },
                "listings":
                [
                    {
                        "id": "a476ce385b75bc7f",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 166,
                        "publish_version": 166,
                        "title": "tidy.js - zero-filling yearly bar charts with complete()",
                        "update_time": "2021-03-23T15:30:14.190Z",
                        "publish_time": "2021-03-23T15:30:14.190Z",
                        "likes": 1,
                        "comment_count": 0,
                        "slug": "tidy-js-zero-filling-yearly-bar-charts-with-complete",
                        "thumbnail": "42469726dbbe2a563c5e6e8e7e7dc36399a5d06ee2d97eec85c7c2b85fcef306",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    },
                    {
                        "id": "8b770ac8cd0fe392",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 208,
                        "publish_version": 208,
                        "title": "tidy.js - converting data to percentages of a whole",
                        "update_time": "2021-05-07T19:19:58.215Z",
                        "publish_time": "2021-03-03T18:57:55.940Z",
                        "likes": 5,
                        "comment_count": 0,
                        "slug": "tidy-js-converting-data-to-percentages-of-a-whole",
                        "thumbnail": "0b420857867937133d983b7146fb758fa055cce1885d300f8c644ef5c9dcc575",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    },
                    {
                        "id": "13da1c9c6714778b",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 70,
                        "publish_version": 70,
                        "title": "tidy.js - groupBy: summarize while keeping reference to items",
                        "update_time": "2021-05-07T19:19:29.247Z",
                        "publish_time": "2021-02-05T23:32:50.409Z",
                        "likes": 4,
                        "comment_count": 0,
                        "slug": "tidy-js-groupby-summarize-while-keeping-reference-to-items",
                        "thumbnail": "0ca929f0063e5917a8d9ffa891e295910ddcfaa08fbf896a45a7302848dfe483",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    },
                    {
                        "id": "eb159cd244d1afa5",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 195,
                        "publish_version": 195,
                        "title": "tidyjs",
                        "update_time": "2021-09-24T00:47:53.194Z",
                        "publish_time": "2021-02-24T17:20:00.613Z",
                        "likes": 9,
                        "comment_count": 0,
                        "slug": "tidyjs",
                        "thumbnail": "432b20744a9e64498f7bd7cab85b79ea205e9ba919222cee61980d4d5e1e0123",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    },
                    {
                        "id": "bcaf8c209f1a283f",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 645,
                        "publish_version": 645,
                        "title": "tidy.js – Intro & Demo",
                        "update_time": "2021-02-24T17:25:21.441Z",
                        "publish_time": "2021-02-02T16:36:21.176Z",
                        "likes": 108,
                        "comment_count": 1,
                        "slug": "tidy-js-intro-demo",
                        "thumbnail": "f4e0b02c026359561c55c81f4e5974b0a789a15152645a2ea5bb0313759463fe",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    },
                    {
                        "id": "44ccf22ff3418cd3",
                        "type": "notebook_reader",
                        "publish_level": "public",
                        "version": 200,
                        "publish_version": 200,
                        "title": "tidy.js - prep data for Multi-Line Charts",
                        "update_time": "2021-05-07T19:21:13.073Z",
                        "publish_time": "2021-02-24T19:18:46.147Z",
                        "likes": 20,
                        "comment_count": 1,
                        "slug": "tidy-js-prep-data-for-multi-line-charts",
                        "thumbnail": "c99b91984da61c9affbe9775f87da41bbe970644855d4079f2092c7210533e34",
                        "default_thumbnail": null,
                        "fork_of": false,
                        "collection_count": 1,
                        "roles":
                        [],
                        "authors":
                        [
                            {
                                "id": "26ab57b8a43a5dda",
                                "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                                "name": "Peter Beshai",
                                "login": "pbeshai",
                                "bio": "",
                                "home_url": "http://peterbeshai.com",
                                "github_login": "pbeshai",
                                "tier": "basic",
                                "approved": true,
                                "description": ""
                            }
                        ],
                        "sharing": null,
                        "notebook_description": "",
                        "listing_type": "notebook",
                        "owner":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        },
                        "creator":
                        {
                            "type": "individual",
                            "id": "26ab57b8a43a5dda",
                            "github_login": "pbeshai",
                            "login": "pbeshai",
                            "name": "Peter Beshai",
                            "bio": "",
                            "home_url": "http://peterbeshai.com",
                            "avatar_url": "https://avatars.observableusercontent.com/avatar/db43a100f98fe00872d57f02fe3deb1fe4d5305239db9310ba983a9c7d6efb65",
                            "tier": "basic"
                        }
                    }
                ],
                "parent_collections":
                []
            }

It shouldn't be too hard to yank all of the notebooks and put them into one quarto folder

#!/bin/bash
# Turn an Observable Notebook Collection into a directory of Quarto projects
#
# Provide the ""@user/collection" as the only command line argument and
# this script will create directories below the current directory filled with 
# Quarto projects made with ohq2quarto
#
# System Requirements:
#   - htmlq      # https://github.com/mgdm/htmlq
#   - jq         # https://stedolan.github.io/jq/
#   - ohq2quarto # https://github.com/hrbrmstr/ohq2quarto

if [ -z "$1" ]
then  
  echo "Usage: col2quarto.sh @user/collection"
	exit
fi

curl --silent "https://observablehq.com/collection/$1" | \
  htmlq --text "script#__NEXT_DATA__" | \
	jq -r '.props.pageProps.collection.listings.results[] | "https://observablehq.com/@" + (.owner.login) + "/" + (.slug) ' | \
	while read -r OBS_URL
	do
	  SLUG="$(basename "${OBS_URL}")"
		ohq2quarto --ohq-ref "${OBS_URL}" --output-dir "${SLUG}"

	done
@hrbrmstr hrbrmstr self-assigned this Dec 24, 2022
@hrbrmstr hrbrmstr added the enhancement New feature or request label Dec 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hrbrmstr hrbrmstr

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hrbrmstr