Code

[FDS-1797] input graph api#1481

Code

Draft

afwillia wants to merge 137 commits intodevelopSage-Bionetworks/schematic:developfrom FDS-1797-input-graph-apiSage-Bionetworks/schematic:FDS-1797-input-graph-apiCopy head branch name to clipboard

+425-94

Commits

Commits on Mar 25, 2024

• WIP: add export_as_graph flag to convert CLI command.

  afwillia

  committed
  2527224

  Copy full SHA for 2527224

• Add convert CLI test for exporting graph as a pickle.

  afwillia

  committed
  70bfc19

  Copy full SHA for 70bfc19

• Add export_as_graph to help.py and update the CLI click docs.

  afwillia

  committed
  5b41aa5

  Copy full SHA for 5b41aa5

• Change export_as_graph to output_format and specify graph, jsonld, or all.

  afwillia

  committed
  76a5567

  Copy full SHA for 76a5567

• Add tests for CLI convert output_type options.

  afwillia

  committed
  670b5c7

  Copy full SHA for 670b5c7

• Change text of CLI convert output_type help.

  afwillia

  committed
  a87d2fe

  Copy full SHA for a87d2fe

• Test that jsonld is created correctly by convert CLI

  afwillia

  committed
  07dbd70

  Copy full SHA for 07dbd70

Commits on Mar 26, 2024

• Add type hints to convert, clean up formatting issues from sonarcloud scan.

  afwillia

  committed
  421c59a

  Copy full SHA for 421c59a

• Add convert CLI test for not specifying --output_type

  afwillia

  committed
  2f5abf1

  Copy full SHA for 2f5abf1

Commits on Apr 2, 2024

• Merge branch 'Sage-Bionetworks:develop' into FDS-1796-schema-convert-export-pickle

  afwillia

  authored
  119cf40

  Copy full SHA for 119cf40

Commits on Apr 3, 2024

• parameterize the convert CLI test.

  afwillia

  committed
  c6c5cb8

  Copy full SHA for c6c5cb8

• remove unused test code for the convert CLI.

  afwillia

  committed
  dd8b2cd

  Copy full SHA for dd8b2cd

• run black on the updated files.

  afwillia

  committed
  6ec81db

  Copy full SHA for 6ec81db

• Use logger.info and logger.error instead of click.echo.

  afwillia

  committed
  bbe6c98

  Copy full SHA for bbe6c98

Commits on Apr 9, 2024

• Trim pickle from end of filename, in addition to csv and jsonld.

  afwillia

  committed
  b99d9eb

  Copy full SHA for b99d9eb

• add output_path alias for output_jsonld

  afwillia

  committed
  1189107

  Copy full SHA for 1189107

Commits on Apr 10, 2024

• Add combination of arguments to convert CLI tests

  afwillia

  committed
  86f7af8

  Copy full SHA for 86f7af8

• add export_graph to schema_utils.py to dump a pickle file

  afwillia

  committed
  d3c3769

  Copy full SHA for d3c3769

• use export_graph util to save pickle file instead of handling that logic in the CLI function

  afwillia

  committed
  215cc86

  Copy full SHA for 215cc86

• add docstrings to graph_export

  afwillia

  committed
  242c885

  Copy full SHA for 242c885

• Update tests to run different combinations of output_jsonld and output_type

  afwillia

  committed
  a1cfcce

  Copy full SHA for a1cfcce

Commits on Apr 11, 2024

• Add data_model_graph_pickle to generator

  afwillia

  committed
  1c67b4a

  Copy full SHA for 1c67b4a

• Add ata_model_graph_pickle to metadata.py

  afwillia

  committed
  a986b39

  Copy full SHA for a986b39

• Add data_model_graph to attributes_explorer

  afwillia

  committed
  0390b69

  Copy full SHA for 0390b69

• Add data_model_graph_pickle to tangled_tree

  afwillia

  committed
  37bbf74

  Copy full SHA for 37bbf74

Commits on Apr 18, 2024

• Add convert CLI test case where output_jsonld is pickle and output_path is pickle. Also test both parameters are empty.

  afwillia

  committed
  441b865

  Copy full SHA for 441b865

• move tests into one function

  afwillia

  committed
  fe4fffb

  Copy full SHA for fe4fffb

Commits on Apr 19, 2024

• Merge branch 'develop' into FDS-1796-schema-convert-export-pickle

  afwillia

  authored
  8c64981

  Copy full SHA for 8c64981

• Run black on commands.py and schema_utils.py

  afwillia

  committed
  f20f88b

  Copy full SHA for f20f88b

• fix single-letter variable names for pylint error

  afwillia

  committed
  d56c965

  Copy full SHA for d56c965

Commits on May 7, 2024

• Merge branch 'develop' into FDS-1843-use-graph

  afwillia

  committed
  4596560

  Copy full SHA for 4596560

Commits on May 8, 2024

• Add data_model_graph argument to ManifestGenerator.create_manifests

  afwillia

  committed
  b5dd8e4

  Copy full SHA for b5dd8e4

• use pickle.load to read pickle file

  afwillia

  committed
  c8da900

  Copy full SHA for c8da900

• Add data model graph pickle to attributes_explorer

  afwillia

  committed
  601b5d0

  Copy full SHA for 601b5d0

• Add data model graph pickle to tangled_tree

  afwillia

  committed
  c6d01de

  Copy full SHA for c6d01de

Commits on Jun 5, 2024

• return 0 to resolve a pylint error.

  afwillia

  committed
  1f57cf1

  Copy full SHA for 1f57cf1

• Merge branch 'develop' into FDS-1796-schema-convert-export-pickle

  afwillia

  authored
  99e860b

  Copy full SHA for 99e860b

• remove extra function def from merge

  afwillia

  committed
  d89dc10

  Copy full SHA for d89dc10

• run black on utils/schema_utils.py

  afwillia

  committed
  c9786dc

  Copy full SHA for c9786dc

• Return 0 for all convert, but echo the filepath created

  afwillia

  committed
  38b0690

  Copy full SHA for 38b0690

• Merge branch 'develop' into FDS-1843-use-graph

  afwillia

  committed
  77979c7

  Copy full SHA for 77979c7

Commits on Jun 6, 2024

• Fix mix-up between parsed data model and graph data model

  afwillia

  committed
  9907b4e

  Copy full SHA for 9907b4e

• Add data model pickle parsing to tanlged_tree

  afwillia

  committed
  481ebd3

  Copy full SHA for 481ebd3

• Add data_model_pickle file to tests

  afwillia

  committed
  f63231a

  Copy full SHA for f63231a

• Add data model pickle file

  afwillia

  committed
  f3fdc41

  Copy full SHA for f3fdc41

• run black

  afwillia

  committed
  032771f

  Copy full SHA for 032771f

• fix a couple sonarcloud issues with the graph_data_model variable

  afwillia

  committed
  bf0e9ba

  Copy full SHA for bf0e9ba

• Add data_model_graph_pickle to metadata.py and tests

  afwillia

  committed
  bc59218

  Copy full SHA for bc59218

• run black on generator.py

  afwillia

  committed
  5973a75

  Copy full SHA for 5973a75

• remove type check from networkx

  afwillia

  committed
  bf1a449

  Copy full SHA for bf1a449

• fix pylint issues

  afwillia

  committed
  69bb182

  Copy full SHA for 69bb182

Commits on Jun 7, 2024

• remove pickle from metadata test because it wasn't created with data_model_labels=display_label

  afwillia

  committed
  d93aa9d

  Copy full SHA for d93aa9d

• create two metadata_model objects with different data_model_label graph files

  afwillia

  committed
  aef4f62

  Copy full SHA for aef4f62

• add display_label graph pickle

  afwillia

  committed
  08b2ae5

  Copy full SHA for 08b2ae5

Commits on Jun 26, 2024

• Merge branch 'develop' into FDS-1843-use-graph

  afwillia

  committed
  a088d63

  Copy full SHA for a088d63

Commits on Jun 28, 2024

• Add multiple combinations of jsonld and pickle to test_viz attribute_explorer

  afwillia

  committed
  68672c3

  Copy full SHA for 68672c3

• Fix missing variables in tangled_tree when jsonld and pickle are supplied

  afwillia

  committed
  de07c55

  Copy full SHA for de07c55

• Add combinations of jsonld and pickle to tangled_tree tests

  afwillia

  committed
  77f5ec4

  Copy full SHA for 77f5ec4

• add graph_data_model argument description

  afwillia

  committed
  5934ed5

  Copy full SHA for 5934ed5

• add data_model_graph_pickle description to MetadataModel

  afwillia

  committed
  51313bd

  Copy full SHA for 51313bd

Commits on Aug 26, 2024

• run black on test_viz.py

  afwillia

  committed
  e4e1368

  Copy full SHA for e4e1368

• Update schematic/manifest/generator.py

  Show description for 9a78a18
  

  afwillia

  and

  BryanFauble

  authored
  9a78a18

  Copy full SHA for 9a78a18

• Add read_pickle function to read a binary pickle file

  afwillia

  committed
  05b4fe0

  Copy full SHA for 05b4fe0

• Use read_pickle instead of importing pickle and opening file

  afwillia

  committed
  0964501

  Copy full SHA for 0964501

• Add note for pickle files that don't fit in memory. Not sure if this will ever be an issue

  afwillia

  committed
  697cc3e

  Copy full SHA for 697cc3e

• Use read_pickle to load the pickle file in metadata.py

  afwillia

  committed
  774dc91

  Copy full SHA for 774dc91

• read_pickle instead of import pickle in attributes_explorer

  afwillia

  committed
  bb7c450

  Copy full SHA for bb7c450

• set self.graph_data_model and update it if data_model_grapher is not provided

  afwillia

  committed
  8f5bc1a

  Copy full SHA for 8f5bc1a

• use if ... is not None instead of if not ...

  afwillia

  committed
  35c13dd

  Copy full SHA for 35c13dd

• use if ... is not None instead of if not ...

  afwillia

  committed
  f8c14d4

  Copy full SHA for f8c14d4

• add an input to read_pickle

  afwillia

  committed
  13c9ffc

  Copy full SHA for 13c9ffc

Commits on Aug 27, 2024

• run black on io_utils.py

  afwillia

  committed
  9205d6d

  Copy full SHA for 9205d6d

• use read_pickle to read pickle file. Set default data_model_labels

  afwillia

  committed
  172e1b1

  Copy full SHA for 172e1b1

• set default data_model_labels

  afwillia

  committed
  d4d3a30

  Copy full SHA for d4d3a30

• alter logic for setting graph_data_model from pickle or DataModelGraph object

  afwillia

  committed
  730b227

  Copy full SHA for 730b227

• fix pickle reading logic in tangled tree

  afwillia

  committed
  795e263

  Copy full SHA for 795e263

• fix logic checking for None parameters in attributes_explorer.py

  afwillia

  committed
  0ddb552

  Copy full SHA for 0ddb552

• import "import pickle" should be placed before "from schematic import LOADER"

  afwillia

  committed
  537af6f

  Copy full SHA for 537af6f

• Use literal for output_type parameter and remove unnecessary try/except statement

  afwillia

  committed
  dcb2f65

  Copy full SHA for dcb2f65

• use logger.exception instead of logger.error

  afwillia

  committed
  bb4ee3a

  Copy full SHA for bb4ee3a

• move return statement inside the output_type check block

  afwillia

  committed
  5504586

  Copy full SHA for 5504586

• run black

  afwillia

  committed
  b88ee86

  Copy full SHA for b88ee86

• import Literal from typing

  afwillia

  committed
  2bf450b

  Copy full SHA for 2bf450b

• add tests for export_graph

  afwillia

  committed
  7adaf44

  Copy full SHA for 7adaf44

• run black

  afwillia

  committed
  c2a06f6

  Copy full SHA for c2a06f6

• add graph_url spec to manifest/generate endpoint

  afwillia

  committed
  1e5bd7d

  Copy full SHA for 1e5bd7d

• add function to download temp pickle file from url. Add data_model_pickle parameter to manifest/generate

  afwillia

  committed
  c849f92

  Copy full SHA for c849f92

• use data_model_graph_pickle argument in manifest/generate endpoint

  afwillia

  committed
  8041b07

  Copy full SHA for 8041b07

• download temp pickle in manifest/generate endpoint

  afwillia

  committed
  0870fa1

  Copy full SHA for 0870fa1

• check that files are created in schema convert

  afwillia

  committed
  6802753

  Copy full SHA for 6802753

• update docstring and use if ... is not None instead of if not ...

  afwillia

  committed
  23b7e1d

  Copy full SHA for 23b7e1d

• use if ... is None instead of if ... is not None

  afwillia

  committed
  fa5e308

  Copy full SHA for fa5e308

Commits on Aug 28, 2024

• Merge branch 'FDS-1796-schema-convert-export-pickle' into FDS-1797-input-graph-api

  afwillia

  committed
  3dd6659

  Copy full SHA for 3dd6659

• Merge branch 'FDS-1843-use-graph' into FDS-1797-input-graph-api

  afwillia

  committed
  f28a0d7

  Copy full SHA for f28a0d7

• add graph_url to validate and submission endpoints

  afwillia

  committed
  afbe448

  Copy full SHA for afbe448

• add graph_url to validate and submission endpoints

  afwillia

  committed
  e4d0b7c

  Copy full SHA for e4d0b7c

• add pickled htan data model for testing

  afwillia

  committed
  a7d7366

  Copy full SHA for a7d7366

• remove data model graph parameter from validation and submission. It's only available for manifest generation. Also rework some logic with reading the graph pickle

  afwillia

  committed
  ee9cb9f

  Copy full SHA for ee9cb9f

• remove graph_url from validation and submission endpoints

  afwillia

  committed
  cec7aa6

  Copy full SHA for cec7aa6

• Merge branch 'develop' into FDS-1797-input-graph-api

  afwillia

  authored
  92b66fd

  Copy full SHA for 92b66fd

• run black on routes.py

  afwillia

  committed
  b52bfdc

  Copy full SHA for b52bfdc

• set graph_data_model to None before looking for graph_url

  afwillia

  committed
  8a64c39

  Copy full SHA for 8a64c39

• add DataModelParser back to tangled_tree.py

  afwillia

  committed
  5bd3fc0

  Copy full SHA for 5bd3fc0

Commits on Sep 10, 2024

• change logger.info to click.echo

  afwillia

  committed
  9c2edde

  Copy full SHA for 9c2edde

• Check if pickle file contains a networkx MultiDiGraph object

  afwillia

  committed
  58c1429

  Copy full SHA for 58c1429

• explain options to schema convert output_type

  afwillia

  committed
  487e31d

  Copy full SHA for 487e31d

• update data_model_graph_pickle docstring for ManifestGenerator

  afwillia

  committed
  e13f693

  Copy full SHA for e13f693

• don't return 0 at the end of schema convert

  afwillia

  committed
  3d5da40

  Copy full SHA for 3d5da40

• raise a valueError instead of logger.error for passing a bad filepath to schema convert

  afwillia

  committed
  a331725

  Copy full SHA for a331725

• return 0 at the end of schema convert to avoid error 'error: Return value expected [return-value]'

  afwillia

  committed
  3ed1177

  Copy full SHA for 3ed1177

• Raise an actual exception and valueError for bad filepath to schema convert

  afwillia

  committed
  b6728d3

  Copy full SHA for b6728d3

• add more information to schema convert when saving the graph as a pickle.

  afwillia

  committed
  b7460ea

  Copy full SHA for b7460ea

• add argument and return types to get_temp_pickle

  afwillia

  committed
  53e639b

  Copy full SHA for 53e639b

• run the black code formatter

  afwillia

  committed
  498851b

  Copy full SHA for 498851b

• turn a single line that pylint states is too long into two shorter lines of satisfactory length

  afwillia

  committed
  58aef40

  Copy full SHA for 58aef40

• swap the f in logger.echo to avoid 'Using an f-string that does not have any interpolated variables'

  afwillia

  committed
  03f7497

  Copy full SHA for 03f7497

• Merge branch 'develop' into FDS-1797-input-graph-api

  afwillia

  authored
  bc64305

  Copy full SHA for bc64305

• message where graph is saved in schema convert cli

  afwillia

  committed
  2a7ea55

  Copy full SHA for 2a7ea55

• check for FileNotFound error instead of generic exception

  afwillia

  committed
  275fbae

  Copy full SHA for 275fbae

• change test variables to camel case

  afwillia

  committed
  89cd4ce

  Copy full SHA for 89cd4ce

• Merge branch 'FDS-1797-input-graph-api' of https://github.com/Sage-Bionetworks/schematic into FDS-1797-input-graph-api

  afwillia

  committed
  0f4091a

  Copy full SHA for 0f4091a

• document schema convert tests

  afwillia

  committed
  3197abf

  Copy full SHA for 3197abf

• turn get_temp_ functions into one single function

  afwillia

  committed
  abd1630

  Copy full SHA for abd1630

• remove unhelpful comments and whitespace

  afwillia

  committed
  445aad5

  Copy full SHA for 445aad5

• clarify test cases for attributes explorer

  afwillia

  committed
  0c4fad7

  Copy full SHA for 0c4fad7

• run black

  afwillia

  committed
  f0b266d

  Copy full SHA for f0b266d

Commits on Oct 28, 2024

• Merge branch 'develop' into FDS-1797-input-graph-api

  afwillia

  authored
  559581c

  Copy full SHA for 559581c

• merge develop updates

  afwillia

  committed
  9808f2b

  Copy full SHA for 9808f2b

• simpify message when export graph as pickle.

  afwillia

  committed
  ec5e121

  Copy full SHA for ec5e121

• add tests for read_pickle

  afwillia

  committed
  5d59890

  Copy full SHA for 5d59890

• add error handling and messaging to read_pickle

  afwillia

  committed
  79ad3d8

  Copy full SHA for 79ad3d8

• add explanation to display label test

  afwillia

  committed
  763dbdd

  Copy full SHA for 763dbdd

• remove extra import and update docstring for create_manifests

  afwillia

  committed
  288431c

  Copy full SHA for 288431c

• add pickle file to test_create_manifests

  afwillia

  committed
  9ec9ddb

  Copy full SHA for 9ec9ddb

• run black

  afwillia

  committed
  86e1646

  Copy full SHA for 86e1646

• run black

  afwillia

  committed
  d20c1fc

  Copy full SHA for d20c1fc

• fix pylint issues

  afwillia

  committed
  5a42f8a

  Copy full SHA for 5a42f8a