Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scan estimation apiserver #559

Closed
wants to merge 2 commits into from
Closed

Conversation

fishkerez
Copy link
Contributor

@fishkerez fishkerez commented Aug 16, 2023

Description

Resolve #586

Discussion: #465

Adding the ability to post a new scan estimation request which will perform a cost estimation of a scan/assetScan according to the scan configuration (scan scope) and previous asset scans statistics.

Adds 2 new resources: ScanEstimation and AssetScanEstimation with corresponding API endpoints.
Also added 2 new tables in the DB.

2 new reconcile controllers for these new resources are added in the orchestrator PR: #560

The cost calculation implementation for AWS will be added soon.

Type of Change

[ ] Bug Fix
[* ] New Feature
[ ] Breaking Change
[ ] Refactor
[ ] Documentation
[ ] Other (please describe)

@fishkerez fishkerez requested a review from a team as a code owner August 16, 2023 12:06
@fishkerez fishkerez self-assigned this Aug 16, 2023
type: array
items:
type: string
nullable: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets add message and reason here too, that way if its Failed due to aborted, timeout or error we can tell the difference

Comment on lines +2807 to +2830
state:
description: The lifecycle state of this scan estimation.
type: string
enum:
- Pending
- Discovered
- InProgress
- Aborted
- Failed
- Done
stateMessage:
description: Human-readable message indicating details about the last state transition.
type: string
stateReason:
description: Machine-readable, UpperCamelCase text indicating the reason for the condition's last transition.
type: string
enum:
- Aborted
- TimedOut
- OneOrMoreAssetFailedToEstimate
- DiscoveryFailed
- Unexpected
- NothingToEstimate
- Success
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets collapse this into its own ScanEstimationState object like what we're doing with the AssetScanEstimationState. This is the approach we're planning to take across all objects soon once someone is able to pick up the refactor work: https://github.com/openclarity/vmclarity/pull/389/files

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should think about a general State object that will be used across all resources instead of defining a State for each object

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have a general state object, however we still want to have enums for state reason and state that are different for each type. It is possible to do that in openapi schema using "allOf" but the generator doesn't handle it well.

type: string
readOnly: true
scanEstimation:
$ref: '#/components/schemas/ScanEstimation'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this one for ScanEstimation.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not? It is returned in case of conflict in POST/PATCH

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no conflict case for a ScanEstimation as far as I know

description: Total cost of the scan ($)
type: number
format: float
recipe:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think recipe is the correct terminology here, costBreakdown would be easier to understand

summary:
$ref: '#/components/schemas/ScanEstimationSummary'
revision:
type: integer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're missing the relationship to the AssetScanEstimations which let us $expand it something like:

assetScanEstimations:
  description: AssetScanEstimations which make up this ScanEstimation
  type: array
  items:
     $ref: '#/components/schemas/AssetScanEstimationRelationship'

Copy link
Contributor Author

@fishkerez fishkerez Aug 29, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this if we already have assetIds? we can get the assetScanEstimation from the asset id and the scanEstimation id, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to be able to GET /scanEstimation/<uuid>?$expand=AssetScans to get a single response that has all the estimations for all the assets. Otherwise we have to do first query the scanEstimation and then make N requests to get all the assetScanEstimations.

Fields: odatasql.Schema{
"id": odatasql.FieldMeta{FieldType: odatasql.PrimitiveFieldType},
},
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this to exist, AssetScanEstimation should just have:

			"scanEstimation": odatasql.FieldMeta{
				FieldType:            odatasql.RelationshipFieldType,
				RelationshipSchema:   scanEstimationSchemaName,
				RelationshipProperty: "id",
			},

assetScanTemplate:
$ref: '#/components/schemas/AssetScanTemplate'
revision:
type: integer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add StartTime/EndTime here so that we can use them for timeout purposes.

return sendError(ctx, http.StatusNotFound, fmt.Sprintf("asset scan estimation was not found. assetScanEstimationID=%v: %v", assetScanEstimationID, err))
}
return sendError(ctx, http.StatusInternalServerError, fmt.Sprintf("failed to get asset scan estimation. assetScanEstimationID=%v: %v", assetScanEstimationID, err))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for this the DB layer will check this for you as part of the Update function

return sendError(ctx, http.StatusNotFound, fmt.Sprintf("asset scan estimation was not found. assetScanEstimationID=%v: %v", assetScanEstimationID, err))
}
return sendError(ctx, http.StatusInternalServerError, fmt.Sprintf("failed to get asset scan estimation. assetScanEstimationID=%v: %v", assetScanEstimationID, err))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto, just need to move the ErrRecordNotFound error handling to the block below here.

var validationErr *common.BadRequestError
var conflictErr *common.BadRequestError
var preconditionFailedErr *databaseTypes.PreconditionFailedError
switch true {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for "true" here switch in go can be used like:

switch {
case condition1:
case condition2:
}

@fishkerez
Copy link
Contributor Author

Moved to #603

@fishkerez fishkerez closed this Aug 29, 2023
@chrisgacsal chrisgacsal deleted the estimation-apiserver branch October 12, 2023 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants