Order focus exported columns to make it easier to read and parse #344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Varun Mittal <[email protected]>
mike-finopsorg
added a commit
that referenced
this pull request
Jun 6, 2024
* Added initial documentation to the README Signed-off-by: Varun Mittal <[email protected]> * Added more details in README about initial service design. Added contributing rules Signed-off-by: Mike Fuller ([email protected]) Signed-off-by: Varun Mittal <[email protected]> * Initial code import. Signed-off-by: Varun Mittal <[email protected]> * Added data generators. Signed-off-by: Varun Mittal <[email protected]> * Imported code into focus_converter_base. Signed-off-by: Varun Mittal <[email protected]> * Updated git repository address. Signed-off-by: Varun Mittal <[email protected]> * Lint: Fixed step 4 command in README.md. Signed-off-by: Varun Mittal <[email protected]> * Updated README.md to include windows powershell instructions. Signed-off-by: Varun Mittal <[email protected]> * Added file_names to base_config and added file_names to visualization layer. Signed-off-by: Varun Mittal <[email protected]> * Added jupyter notebook to show plan graphically. Signed-off-by: Varun Mittal <[email protected]> * Added sql validation while building plan. (#49) Signed-off-by: Varun Mittal <[email protected]> * Fixed test data generators to use if/else instead of match statement. (#43) Signed-off-by: Varun Mittal <[email protected]> * Used if-else for backward compatibility and added unittests for all python versions. (#36) Signed-off-by: Varun Mittal <[email protected]> * Added exception handler in convert function to allow cleanup of exporter processes. (#46) Signed-off-by: Varun Mittal <[email protected]> * Updated gcp ChargePeriodStart and ChargePeriodEnd to use date types as UTC datetime format. (#37) Signed-off-by: Varun Mittal <[email protected]> * Added config for chargetype for AWS. (#42) Signed-off-by: Varun Mittal <[email protected]> * Added gcp cost type config. (#48) Signed-off-by: Varun Mittal <[email protected]> * Fixed credits.amount by adding field schema field type in case it is missing. (#104) Signed-off-by: Varun Mittal <[email protected]> * Add datetime function to convert to ms for datetime conversion functions. (#107) Signed-off-by: Varun Mittal <[email protected]> * Added capability to specify column prefix in the source dataset. (#105) Signed-off-by: Varun Mittal <[email protected]> * Implemented lookup operation for converting column values using a reference dataset. (#117) Signed-off-by: Varun Mittal <[email protected]> * Fixed sql execution by creating unique sql context for each query. (#115) Signed-off-by: Varun Mittal <[email protected]> * Added value map function to allow values maps with defaults. (#112) Signed-off-by: Varun Mittal <[email protected]> * Added base function to create static value assignment. (#109) Signed-off-by: Varun Mittal <[email protected]> * Added aws provider conversion config. (#110) Signed-off-by: Varun Mittal <[email protected]> * Lookup fixes (#122) * Added test notebook glob pattern to .gitignore. * Added reference path check to differentiate between package file vs test file. Signed-off-by: Varun Mittal <[email protected]> * Implemented new unnest logic using pl.col.struct expression implementing aggregation logic for nested list values. (#121) Signed-off-by: Varun Mittal <[email protected]> * Added service category mappings for AWS. (#114) Signed-off-by: Varun Mittal <[email protected]> * Added gcp service category mapping using lookup operation. (#123) Signed-off-by: Varun Mittal <[email protected]> * Added documentation for different conversion operations supported with examples. (#124) Signed-off-by: Varun Mittal <[email protected]> * Updated main to allow running validator and added sample notebook. (#119) Signed-off-by: Varun Mittal <[email protected]> * Added focus column names to config validator. (#128) Signed-off-by: Varun Mittal <[email protected]> * Added focus column enum for SubAccountId. (#129) Signed-off-by: Varun Mittal <[email protected]> * Lint: Fixed black and isort errors. Also python 3.8 import errors. (#127) Signed-off-by: Varun Mittal <[email protected]> * Fixed Billing Currency focus enum. (#130) Signed-off-by: Varun Mittal <[email protected]> * Temporarily disable python 3.12 checks as pyarrow not compatible until 14.0 release. (#133) Signed-off-by: Varun Mittal <[email protected]> * Added azure dimension configs. (#132) Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Cory Syvenky <[email protected]> Add Azure provider field (#135) Signed-off-by: Cory Syvenky <[email protected]> Fix dimension order for provider (#136) Signed-off-by: Cory Syvenky <[email protected]> Add Azure ServiceCategory (#137) * Add Azure ServiceCategory Signed-off-by: Cory Syvenky <[email protected]> Added notebook cell for Azure plan visualization. (#138) Signed-off-by: Cory Syvenky <[email protected]> Updated README to include instructions for direct pip install. (#131) Signed-off-by: Varun Mittal <[email protected]> Added OCI Provider dimension (#85) (#139) Signed-off-by: Irena Jurica <[email protected]> Simple Typo fix in readme Signed-off-by: Mike Fuller <[email protected]> * Added OCI Region dimension (#86) (#148) Signed-off-by: ijurica <[email protected]> * Added OCI Billing Account Id dimension (#29) (#147) Signed-off-by: ijurica <[email protected]> * add resourcename field from azure config (#145) Signed-off-by: Cory Syvenky <[email protected]> * add billingcurrency field from azure config (#144) Signed-off-by: Cory Syvenky <[email protected]> * converted all date fields to datetime (#143) Signed-off-by: Cory Syvenky <[email protected]> * Added AvailibityZone, BillingCurrency, InvoiceIssuer, ResourceId, ServiceName for AWS. (#140) Signed-off-by: Varun Mittal <[email protected]> * Added image path parameter for explain command to save graph. (#149) --------- Signed-off-by: Varun Mittal <[email protected]> * Fixed datetime columns imported as str instead of datetime[ns]. (#150) Signed-off-by: Varun Mittal <[email protected]> * Added base functions for validating column names before starting conversion. (#151) Signed-off-by: Varun Mittal <[email protected]> * Added new conversion plan to add default null values if a column is missing in source dataset. (#158) Signed-off-by: Varun Mittal <[email protected]> * Added ignore_errors while reading cvs to allow reading even if the csv parsing fails. (#157) Signed-off-by: Varun Mittal <[email protected]> * Shuffle docs around (#156) Signed-off-by: Joshua Kwan <[email protected]> Co-authored-by: Varun Mittal <[email protected]> Co-authored-by: Mike Fuller <[email protected]> * Aws 1.0 transform plans along with more sql validations. (#154) Signed-off-by: Varun Mittal <[email protected]> * addition of an anonymized AWS CUR dataset (#155) Signed-off-by: Cory Syvenky <[email protected]> * Added multiple OCI focus v0.5 dimensions: AvailabilityZone (issue #82), BillingCurrency (issue #83), ResourceId (issue #56), ServiceName (issue #88), SubAccountId (issue #89) (#163) Signed-off-by: ijurica <[email protected]> * Added OCI BilledCost (issue #35) and EffectiveCost (issue #90) (#162) Signed-off-by: ijurica <[email protected]> * chore: fix a typo on aws category mapping filename (#160) Signed-off-by: Ivan Stoiev <[email protected]> * Added GCP configs for Provider,AvailabilityZone,BillingCurrency,ResourceId (#153) Signed-off-by: Varun Mittal <[email protected]> * Dependency fixes + readme edit (#165) Signed-off-by: Barak Freiman <[email protected]> * Create ChargePeriodStart FOCUS from Billing Start (#239) Signed-off-by: Cory Syvenky <[email protected]> * addition of Publisher field for Azure provider. (#237) Signed-off-by: Cory Syvenky <[email protected]> * addition of BilledCost field (#236) Signed-off-by: Cory Syvenky <[email protected]> * addition of Billing Account Name and Billing Account ID fields for Azure. (#235) Signed-off-by: Cory Syvenky <[email protected]> * aws:cur as a new provider for processing AWS CUR CSV (#240) Signed-off-by: spriharani <[email protected]> * Fixed python dependencies. (#164) Added jupyterlab as optional dependency and removed redundant dependencies. Signed-off-by: Varun Mittal <[email protected]> * Renamed aws:cur to aws-cur to avoid path issues on windows. (#241) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.0 → 0.7.1-dev1, Init pypi build (#242) Signed-off-by: Varun Mittal <[email protected]> * Hotfix: Temporarily change focus-validator version to support pip. (#243) Signed-off-by: Varun Mittal <[email protected]> * Handle lineItem/ResourceId if it is missing in source data (#246) Signed-off-by: spriharani <[email protected]> * fix(aws-cur): wrong aws cur column used on ChargeSubcategory calculation (#244) Signed-off-by: Ivan Stoiev <[email protected]> * fix(aws-cur): wrong invoice issuer field calculation (#245) Signed-off-by: Ivan Stoiev <[email protected]> * Added specific config checks for aws,azure and oci. Also added dtype re-enforcements for datatime/date. (#247) Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Ivan <[email protected]> * Added cli utility to export conversion rules to a csv/markdown per provider. (#249) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Bump version: 0.7.1-dev1 → 0.7.1-dev2 (#250) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev2 → 0.7.1-dev3 * Fix aws-cur - parse bill/PayerAccountId as string (#251) Signed-off-by: spriharani <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added focus_validation integration. (#254) Signed-off-by: Varun Mittal <[email protected]> * Updated conversion config to include feature request, column conversion logic and bug_report. (#255) Signed-off-by: Varun Mittal <[email protected]> * Added FOCUS columns to output types to produce empty columns if conversion plan not available yet. (#256) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Standardize AWS category mapping, similar to Azure and GCP (#259) Signed-off-by: Brian Wyka <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added converter function to compute start and end of datetime column. (#248) Added configs for OCI using new start/end function to compute BillingPeriodStart, BillingPeriodEnd. Fixes #30, #31, #32, #33. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion config for GCP SkuId, resoloving #232. (#260) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Adding a performance profiling tool and applying to existing tests (#266) Signed-off-by: Justin Ohrenberger <[email protected]> * Adding flake8 config for higher quality code. (#268) Signed-off-by: Justin Ohrenberger <[email protected]> * Removing Azure D011 as it appears to be a duplicate of D007 (only with an error - appears to be trying to convert SubscriptionID to BillingCurrency) * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of availability zone field (#273) Signed-off-by: Cory Syvenky <[email protected]> * addition of EffectiveCost field. (#271) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Azure usage quantity conversion yaml file (#276) Signed-off-by: David Flood <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of focus column ChargeDescription (#290) Signed-off-by: Cory Syvenky <[email protected]> * Changed orientation of generated UML diagram to left-right from top-down (#289) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of focus column PricingCategory (#295) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * addition of focus column ChargeFrequency (#291) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of ChargeSubcategory (#296) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Bug fix for effective cost (#292) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added definition for CommitmentDiscountName. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Azure commitment discount name conversion file (#300) Signed-off-by: David Flood <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of InvoiceIssuer FOCUS field. (#308) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of UsageUnit FOCUS field via Pricing Units mapping. (#309) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Addition of SkuPriceId FOCUS field, defaulting to null. (#310) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Reducing cyclomatic complexity (#306) Signed-off-by: Justin Ohrenberger <[email protected]> * Add docker image. (#311) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev3 → 0.7.1-dev4 Signed-off-by: Varun Mittal <[email protected]> * Upgraded polars(0.19.2 -> 0.20.6) to match sql aggregate functions. Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev4 → 0.7.1-dev5 Signed-off-by: Varun Mittal <[email protected]> * Add progress bar for each provider. (#314) Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> * Azure ResourceId conversion (#301) Signed-off-by: David Flood <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * remove-dupe: Remove duplicate Azure microsoft.security mapping (#315) Signed-off-by: Brian Wyka <[email protected]> * addition of CommitmentDiscountId (#298) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Gcp new column configs. (#317) * Added configs for PricingQuantity(#212), SkuPriceId(#227), UsageQuantity(#223), UsageUnit(#221) for GCP. Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of CommitmentDiscountType (#299) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * addition of CommitmentDiscountCategory (#297) Signed-off-by: Cory Syvenky <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added string conversion plans to support lower, upper, title case transforms. (#319) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fixed docker metadata tag to include current version tags. (#320) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev5 → 0.7.1-dev6 * addition of ChargeCategory column. (#322) Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Varun Mittal <[email protected]> Co-authored-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion configs for OCI. (#303) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added new plan to split string and get part using index. (#323) Signed-off-by: Varun Mittal <[email protected]> * Removed BilledCurrency definition. (#324) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added new AWS dimension configs for ResourceName(#51), ResourceName(#278), Publisher(#9). (#326) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fixed pie-chart import for progress/README.md Signed-off-by: Varun Mittal <[email protected]> * Added conversion configs for aws-cur provider(AWS Cur CSV provider). (#327) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added conversion step to export FOCUS data. (#328) Signed-off-by: Varun Mittal <[email protected]> * Added step to convert azure sample data to FOCUS and add to artifact. Signed-off-by: Varun Mittal <[email protected]> * Moved jupyter dependencies to optional, allowing to install --with=jupyter. (#329) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev6 → 0.7.1-dev7 * Added PricingCategory for AWS (#284). (#330) * Added PricingCategory for AWS (#284). Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added input data auto sening module to simplify source data inputs (#332) Added input data auto sening module to simplify source data inputs Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev7 → 0.7.1-dev8 * fix(aws): Region column source (#333) Signed-off-by: Ivan Stoiev <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * aws-category: Add ComputeSavingsPlans to aws category mapping (#334) Signed-off-by: Brian Wyka <[email protected]> * Adding Amazon Bedrock entry (#335) Signed-off-by: varangian-core <[email protected]> * Output specific parquet filename (#339) Signed-off-by: David Sschneider <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Fix multiprocessing queue and downgraded polars version temporarily until pyarrow compatibility is restored. (#341) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Added GCP missing dimensions for EffectiveCost(#168), ServiceName(#77), ResourceName(#52), PricingUnit(#217), ListCost(#205) (#340) Signed-off-by: Varun Mittal <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Order focus exported columns to make it easier to read and parse (#344) Signed-off-by: Varun Mittal <[email protected]> * Bump version: 0.7.1-dev8 → 0.7.1-dev9 Signed-off-by: Varun Mittal <[email protected]> * Added support for file CSV export (#349) * Added support for file CSV export Signed-off-by: Mike Fuller <[email protected]> * Fix import sorting Signed-off-by: Mike Fuller <[email protected]> * Fix up defaults for class to pass tests Signed-off-by: Mike Fuller <[email protected]> --------- Signed-off-by: Mike Fuller <[email protected]> * Commit from GitHub Actions (Export Conversion Rules) Signed-off-by: FinOps Converter Bot <[email protected]> * Updated column configs to align better with 1.0 (GA) spec Fixed tests to work with updated columns Signed-off-by: Mike Fuller <[email protected]> * Bump version to assist build Signed-off-by: Mike Fuller <[email protected]> * Bump version: 0.7.2-dev1 → 1.0.0-dev1 Signed-off-by: Mike Fuller <[email protected]> * Bump version: 1.0.0-dev1 → 1.0.0 Signed-off-by: Mike Fuller <[email protected]> --------- Signed-off-by: Varun Mittal <[email protected]> Signed-off-by: Mike Fuller ([email protected]) Signed-off-by: Mike Fuller <[email protected]> Signed-off-by: ijurica <[email protected]> Signed-off-by: Cory Syvenky <[email protected]> Signed-off-by: Joshua Kwan <[email protected]> Signed-off-by: Ivan Stoiev <[email protected]> Signed-off-by: Barak Freiman <[email protected]> Signed-off-by: spriharani <[email protected]> Signed-off-by: FinOps Converter Bot <[email protected]> Signed-off-by: spriharani <[email protected]> Signed-off-by: Brian Wyka <[email protected]> Signed-off-by: Justin Ohrenberger <[email protected]> Signed-off-by: David Flood <[email protected]> Signed-off-by: varangian-core <[email protected]> Signed-off-by: David Sschneider <[email protected]> Co-authored-by: Varun Mittal <[email protected]> Co-authored-by: Irena Jurica <[email protected]> Co-authored-by: Cory Syvenky <[email protected]> Co-authored-by: Joshua Kwan <[email protected]> Co-authored-by: Ivan <[email protected]> Co-authored-by: Barak <[email protected]> Co-authored-by: spriharani <[email protected]> Co-authored-by: FinOps Converter Bot <[email protected]> Co-authored-by: Brian Wyka <[email protected]> Co-authored-by: johrenberger <[email protected]> Co-authored-by: David Flood <[email protected]> Co-authored-by: David Flood <[email protected]> Co-authored-by: Akari <[email protected]> Co-authored-by: David Schneider (Ollion) <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.