Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamlit improvements #1060

Merged
merged 75 commits into from
Mar 21, 2024
Merged

Streamlit improvements #1060

merged 75 commits into from
Mar 21, 2024

Conversation

sultaniman
Copy link
Contributor

@sultaniman sultaniman commented Mar 6, 2024

This PR upgrades our streamlit app #459 which is launched when dlt pipeline PIPELINE_NAME show command is called.
It also includes better prompt to detect and install streamlit and additional packages if it is missing.

Also important thing for TODO is wait until streamlit.AppTest will merge the PR support for page transitions to add more thorough tests.
This PR introduces a new argument for pipeline subcommand --hot-reload which can be used for development purposes to allow hot reload of changing streamlit app code.
Also to pass pipelines_dir to streamlit app I used DLT_PIPELINES_DIR environment variable.

NOTE: I will remove old streamlit_helper.py once we merge this PR.

Light color mode

Screenshot 2024-03-19 at 10 57 33 Screenshot 2024-03-19 at 10 57 30 Screenshot 2024-03-19 at 10 57 25 Screenshot 2024-03-19 at 10 57 16

Dark color mode

Screenshot 2024-03-19 at 10 57 46 Screenshot 2024-03-19 at 10 57 38

Error states

We need to show users some useful information for example if the destination does not support sql client or other capabilities.

image image

@sultaniman sultaniman added the enhancement New feature or request label Mar 6, 2024
@sultaniman sultaniman self-assigned this Mar 6, 2024
Copy link

netlify bot commented Mar 6, 2024

Deploy Preview for dlt-hub-docs ready!

Name Link
🔨 Latest commit b8208ae
🔍 Latest deploy log https://app.netlify.com/sites/dlt-hub-docs/deploys/65fc2f73aa042600081a69f0
😎 Deploy Preview https://deploy-preview-1060--dlt-hub-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@sultaniman sultaniman marked this pull request as ready for review March 6, 2024 15:52
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few improvements in the UI before merging:

image
this is non-essential information. please move it to Load Info to the bottom. Also it would be cool to show the full pipeline state there but collapsed (hidden) by default

image
this is actually important. can we show it extended?

image
please do not show state view at all if state does not exist

image
order of the columns is for some reason random. it should be:
name | data_type | nullable

I'm happy we are able to test this

dlt/cli/pipeline_command.py Outdated Show resolved Hide resolved
@sultaniman sultaniman force-pushed the issue-459-streamlit branch 2 times, most recently from 94cc7c8 to 6d26612 Compare March 18, 2024 11:14
@sultaniman
Copy link
Contributor Author

Added sorting of table by column names.

Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to refactor this app a little because it got messy.

  1. old app code is still there, why?
  2. old app code was a single page app with single entry and init logic also it forced an interface for displaying pages which was good.
  3. new app duplicates init code, pages are not self contained etc.
  4. why dashboard is not a page?

I like that pages are broken into blocks etc. but top level structure is gone in this PR

dlt/cli/pipeline_command.py Outdated Show resolved Hide resolved
dlt/cli/pipeline_command.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/blocks/table_hints.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/dashboard.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/dashboard.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/dashboard.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/menu.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/menu.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/pages/load_info.py Outdated Show resolved Hide resolved
dlt/helpers/streamlit_app/pages/load_info.py Outdated Show resolved Hide resolved
@rudolfix
Copy link
Collaborator

@sultaniman now I see

NOTE: I will remove old streamlit_helper.py once we merge this PR.

forget my (1) bullet point, didn't see that

@sultaniman sultaniman force-pushed the issue-459-streamlit branch from 52ed0fc to f3dba23 Compare March 18, 2024 20:04
@sultaniman sultaniman requested a review from rudolfix March 19, 2024 14:27
@sultaniman sultaniman force-pushed the issue-459-streamlit branch 2 times, most recently from 9504dcb to 309fbce Compare March 20, 2024 08:07
@sultaniman sultaniman force-pushed the issue-459-streamlit branch from 88acdea to 0609daa Compare March 20, 2024 14:57
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just two small thing in review. and one bigger:

when displaying schemas the order is not fixed! should be
name | data type | nullability
and is reverse

dlt/cli/pipeline_command.py Outdated Show resolved Hide resolved
dlt/cli/pipeline_command.py Outdated Show resolved Hide resolved
@sultaniman
Copy link
Contributor Author

Also we're now manually enforcing the order of columns.

@sultaniman sultaniman force-pushed the issue-459-streamlit branch from 7f8b32b to b8208ae Compare March 21, 2024 13:00
@sultaniman sultaniman requested a review from rudolfix March 21, 2024 13:44
Copy link
Collaborator

@rudolfix rudolfix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! we should think how to eject this app from the lib so people can build their own stuff on top

def dummy_render(pipeline: dlt.Pipeline) -> None:
pass

old_args = sys.argv[:]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just FYI you should patch this value in context manager so it restores at the end of test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed it in the cli runner PR 🙂

@rudolfix rudolfix merged commit 3a815bc into devel Mar 21, 2024
41 of 47 checks passed
@rudolfix rudolfix deleted the issue-459-streamlit branch March 21, 2024 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants