-
Notifications
You must be signed in to change notification settings - Fork 234
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
apps/api testing script fixes #3284
Comments
In case it helps, here's how I'm currently running tests. To be clear, this is just the first way I tried that worked and there may well be better approaches. The key reason for the complexity in my approach is that the default Docker Compose configuration doesn't run the API image from a shared volume, meaning any changes you make in your local
|
When I run the above against commit 1bf1582 (head of develop just before #3283 was merged), it runs and passes the tests for auth and formats, then hangs on the 3rd through last tests in inputs, and if I comment those out it passes all tests for models, PFTs, ping, and the first three tests of runs before throwing a series of errors about missing IDs. Comparing URL patterns in the passing and failing tests of |
This seems to be correct in our case. When we don't employ usage of ID parameters, it does not even returns a response. However, we can still get a status response with usage of ID Parameters. I guess the main issue lies due to aging of our test cases and no being able to line up with our Latest API changes |
Correct me if I'm wrong. In the auth section, I don't think we require authorisation to GET our models via : res <- httr::GET(
"http://pecan.localhost/api/models/",
# Test Failed
) However, Authentication is required on using : res <- httr::GET(
"http://localhost:8000/api/models/",
# Test Passed
) |
I still think that the below test is useless since no timeout/endpoint has been set for such responses within our test.inputs.R test_that("Calling /api/inputs/{input_id} with invalid parameters returns Status 404", {
res <- httr::GET(
"http://localhost:8000/api/inputs/0",
httr::authenticate("carya", "illinois")
)
# No Response. Stops the testing Permanently
expect_equal(res$status, 404)
}) Another potential issue may be lack of eror handeling and if(!is.null(model_id)) {
url <- paste0(url, "&model_id=", model_id)
}
if(!is.null(site_id)) {
url <- paste0(url, "&site_id=", site_id)
}
if(!is.null(format_id)) {
url <- paste0(url, "&format_id=", format_id)
}
if(!is.null(host_id)) {
url <- paste0(url, "&host_id=", host_id)
} I'm still not a 100% sure about this idea so please correct me if I'm not going thoroughly. |
Status Update for test_that("Calling /api/inputs/{input_id}?filename={filename} with valid parameters returns Status 200", {
res <- httr::GET(
paste0("http://localhost:8000/api/inputs/295?filename=fraction.plantation"),
httr::authenticate("carya", "illinois")
)
# No response. Stops the testing Permanently
expect_equal(res$status, 200)
})
test_that("Calling /api/inputs/{input_id}?filename={filename} with invalid parameters returns Status 404", {
res <- httr::GET(
"http://localhost:8000/api/inputs/295?filename=random",
httr::authenticate("carya", "illinois")
)
# No Response. Stops the testing Permanently
expect_equal(res$status, 400)
})
test_that("Calling /api/inputs/{input_id} with valid parameters returns Status 200", {
res <- httr::GET(
paste0("http://localhost:8000/api/inputs/", 99000000003),
httr::authenticate("carya", "illinois")
)
# No Response. Stops the testing Permanently
expect_equal(res$status, 200)
}) Are the related endpoints correctly created and linked? If not, it would be natural to return No Response. If yes, there may be any underlying bug in our |
Just a preview from our Swagger Documentations about responses. Making sure the current endpoints and documentations are up to date. ![]() ![]() EDIT : Another Point to be Noted : No endpoint such as ![]() Maybe these lines from our download.input.R from expr = {
url <- paste0(server$url, "/api/inputs/", input_id)
if(! is.null(filename)) {
url <- paste0(url, "?filename=", filename)
}
if(! is.null(server$username) && ! is.null(server$password)){
res <- httr::GET(
url,
httr::authenticate(server$username, server$password)
)
}
else{
res <- httr::GET(url)
}
}
# Rest of Code |
This is the test case I've been looking into for now in apps/api/tests/test.runs.R: test_that("Calling /api/runs/{run_id}/graph/{year}/{yvar}/ with valid inputs returns Status 200", {
res <- httr::GET(
"http://localhost:8000/api/runs/99000000282/graph/2002/GPP",
httr::authenticate("carya", "illinois")
)
# No Response
expect_equal(res$status, 200)
}) I was looking within our SwaggerUI link and found its utilisation URL as follow : Two points I wanted clarity on :
|
Any Updates/Reviews @infotroph and @mdietze ?? |
Final briefing of all failing test cases with error logs/responses can be viewed in this Started with debugging the corresponding |
Final summary of total passing results {excluding failing(commented out) tests suites} : | F W S OK | Context
✔ | 3 | Testing authentication for API [96.2s]
✔ | 4 | Testing all formats related endpoints [109.1s]
✔ | 2 | Testing all inputs related endpoints [127.1s]
✔ | 4 | Testing all models endpoints [13.6s]
✔ | 4 | Testing all PFTs endpoints [13.7s]
✔ | 1 | Testing the /api/ping endpoint
✔ | 3 | Testing all runs endpoints [7.3s]
✔ | 4 | Testing all sites endpoints [4.8s]
✔ | 1 | Testing the /api/status endpoint [0.4s]
✔ | 5 | Testing all workflows endpoints [10.3s]
✔ | 5 | Testing all workflows endpoints [10.3s]
⠏ | 0 | testthat
══ Results ═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════
Duration: 389.7 s
[ FAIL 0 | WARN 0 | SKIP 0 | PASS 31 ] |
Sorry for the long wait. It’s interesting that the suite takes so long even with failing tests commented out — are the long run times in the first three tests related to the auth issues you mentioned? |
I don't think so. It's just that my host machine has been bearing too much of a load recently and is underperformed. When it's only running docker, than it quickly finishes the test suites in about a minute or two. |
If you decide to fix this, I can assign you a high end VM at my companies end for you to work faster, would that work for you ? @Sweetdevil144 |
Yeah sure. No Problem with that. However, I think it isn't much big of a problem. I'd been running some high usage processes alongside |
@GandalfGwaihir That's a generous offer! I trust that you
|
@infotroph , yeah no worries, also I confirm that this is an individual offer not related to PEcAn organisation! @Sweetdevil144 ping me on slack with your system config, we're take it on personal DM from here |
Assigned a VM , @Sweetdevil144 can you put in a timeline to solve this issue please ? |
Hey, thanks a lot @GandalfGwaihir . It'll surely help a lot. As I estimate right now, this issue will be resolved by the end of this month. Although, I'll try to wrap it up as soon as possible. |
#3283 changed URLs inside
test_pecanapi.sh
fromlocalhost:8000
topecan.localhost
, but that script is intended for use inside the Docker image (where localhost is the right hostname to use). However it appears the script wasn't fully working before that change, so let's use this issue to plan the next updates before rolling back any of #3283.Summary originally posted by @Sweetdevil144 in #3283 (comment) :
I think we would be better off with Partially reverting our changes from
pecan.localhost
tolocalhost:8000
with following points in consideration :apps/api/tests/
folder,pecan.localhost
works the same aslocalhost:8000
, only locally however. Can you confirm whether this is true again @infotroph ?GET
at all and take infinite time causing system barriers to exist.Below is a description of all test cases which have been causing problems with current Repository changes. (I plan to revert changes from
pecan.localhost
back tolocalhost:8000
and then confirm it once again whether these runs are successful or not) :apps/api/tests/test.auth.R
: Following test casesapps/api/tests/test.inputs.R
: Following test casesapps/api/tests/test.runs.R
: Following test casesapps/api/tests/test.status.R
: Following test casesapps/api/tests/test.workflows.R
: Following test casesThe text was updated successfully, but these errors were encountered: