Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rest_api: accept DltResource instances in resource list #1837

Closed
rudolfix opened this issue May 21, 2024 · 2 comments · Fixed by #1865
Closed

rest_api: accept DltResource instances in resource list #1837

rudolfix opened this issue May 21, 2024 · 2 comments · Fixed by #1865
Assignees

Comments

@rudolfix
Copy link
Collaborator

Source name

rest_api

Describe the data you'd like to see

We could combine instances of regular resources with a. Such config could look like this:

@dlt.resource(name="stories")
def read_stories(limit: int = 10):
    ...


@dlt.source(name="hacker_news_source", max_table_nesting=2)
def hacker_news_source(
    base_url: str = dlt.config.value,
) -> List[DltResource]:

    # source configuration
    source_config: RESTAPIConfig = {
        "client": {
            "base_url": base_url,
        },
        "resources": [
            {
                "name": "item",
                "table_name": "item",
                "endpoint": {
                    "path": "/item/{id}.json",
                    "data_selector": "$",
                    "paginator": "single_page",
                    "params": {
                        "id": {
                            "type": "resolve",
                            "field": "$",
                            "resource": "stories"
                        }
                    },
                },
            },
            read_stories(20)
        ],
    }

    return rest_api_source(source_config)

Above we add resource instance with the name "stories" and also attach it to a transformer that is fully declared

Are you a dlt user?

Yes, I'm already a dlt user.

Do you ready to contribute this extension?

Yes, I'm ready.

dlt destination

No response

Additional information

No response

@burnash
Copy link
Collaborator

burnash commented May 27, 2024

TODO: Update the docs after this feature is merged

@burnash burnash transferred this issue from dlt-hub/verified-sources Sep 18, 2024
@burnash
Copy link
Collaborator

burnash commented Sep 19, 2024

Relevant discussion in the community Slack https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1726574128329839

@willi-mueller willi-mueller linked a pull request Sep 24, 2024 that will close this issue
2 tasks
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Sep 25, 2024
@github-project-automation github-project-automation bot moved this from Planned to Ready for Deployment in Verified Sources Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready for Deployment
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants