ptd is asked to stop measurement twice #287
Labels
bug
Something isn't working
Comments
|
I see similarly spurious requests to stop PTD when starting a new server: |
|
I see the same with the latest unmerged PR from OctoML: |
|
@arjunsuresh It's a good hint, but I'm afraid we have no time to debug which PR causes the workflow to fail. |
|
I can do it, but I need to get to the power analyzer :) |
|
@psyhtest Can you please confirm the commit hash on both the server and SUT machines for the power-dev repository where this error is obtained? |
|
I'm getting the same error when I revert the server to 2.1 stage and client to the latest. This is expected as Once I update the server repository to latest commit there are no errors. |
|
This is with the r3.0 branch which is same as r2.1 branch for the server/client codes. I'm getting the same error of double "stop" but it is not affecting the power measurements. So, this issue has nothing to do with any of the reverted PRs. In fact the same error can be seen on Krai 2.1 submission logs like this ptd-server 2023-02-11 02:23:46,090 [INFO] Sending to ptd: 'Stop' ptd-server 2023-02-11 02:23:46,111 [INFO] Sending to ptd: 'Stop' 02-11-2023 02:23:47.038: Response to client sent: Error: no measurement to stop |
|
@psyhtest The response to client sent message is also there in old logs- it is in ptd_logs.txt and not server.log. |
|
Having said that this issue is certainly a bug and needs to be resolved. |
arjunsuresh
added a commit
to GATEOverflow/power-dev
that referenced
this issue
Apr 12, 2023
|
@psyhtest This issue can be closed right? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm running into an error with both the server and the client on 1a16663 (i.e. the very last change before updating the supported PTD versions).
Server
After the ranging run, everything is fine - we change the Amp & Volt ranges accordingly.
Then we start testing.
After 13 minutes, a 'Stop' command is sent to PTD.
And then another 'Stop' command is sent to PTD, whose response 'Error: no measurement to stop' is sent to the client.
I believe it is the above error that the client receives, while it expects an integer (see below).We set the Volt & Amp ranges back to Auto.
Why stopping PTD again and shutting connection down?
Client
ValueError: invalid literal for int() with base 10: 'Error'
client 2023-02-10 06:45:15,488 [INFO] Sending command to the server: 'session,2023-02-10_06-18-14,done' client 2023-02-10 06:45:17,375 [INFO] Got response: 'OK' client 2023-02-10 06:45:17,375 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/ptd_logs.txt' client 2023-02-10 06:45:17,379 [INFO] Received '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/ptd_logs.txt' client 2023-02-10 06:45:17,379 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/ranging/spl.txt' client 2023-02-10 06:45:17,381 [INFO] Received '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/ranging/spl.txt' client 2023-02-10 06:45:17,381 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/server.json' client 2023-02-10 06:45:17,382 [INFO] Received '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/server.json' client 2023-02-10 06:45:17,382 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/server.log' client 2023-02-10 06:45:17,384 [INFO] Received '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/power/server.log' client 2023-02-10 06:45:17,384 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/run_1/spl.txt' client 2023-02-10 06:45:17,386 [INFO] Received '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/run_1/spl.txt' client 2023-02-10 06:45:17,386 [INFO] Fetching file '/home/krai/CK_REPOS/ck-mlperf/program/mlperf-power-client/tmp/2023-02-10_06-18-14/run_1/ptd_out.txt' Traceback (most recent call last): File "/home/krai/CK_TOOLS/mlperf-power-r3.0/power/ptd_client_server/client.py", line 27, in client.main() File "/home/krai/CK_TOOLS/mlperf-power-r3.0/power/ptd_client_server/../ptd_client_server/lib/client.py", line 357, in main command.download(f"download,{session},{fname}", os.path.join(out_dir, fname)) File "/home/krai/CK_TOOLS/mlperf-power-r3.0/power/ptd_client_server/../ptd_client_server/lib/client.py", line 63, in download self._server.recv_file(fname) File "/home/krai/CK_TOOLS/mlperf-power-r3.0/power/ptd_client_server/../ptd_client_server/lib/common.py", line 110, in recv_file chunk_len = int(line, 10) ValueError: invalid literal for int() with base 10: 'Error'The text was updated successfully, but these errors were encountered: