Exceeding CPU limit when pushing eosio.system contract #8289
Comments
|
The max-transaction-time of 3000000 ms you have specified exceeds the maximum block time of 500 ms. No transaction taking that long can be incorporated into a block. You could try increasing |
|
@jgiszczak I already tried that. When I increase all max cpus in the genesis file to some absurd values (like minutes), the transaction takes forever to execute and the node gives up with timeout error. So there is something bad going on when executing that transaction (put system contract), looks like a deadlock to me. |
|
As I said, increasing those limits to beyond the block time is useless. Any transaction taking longer than block time will fail with the timeout you are seeing, always. The transaction is expiring in the queue after repeated failures to be incorporated into a block. Try building the 1.8.3 release of the system contract rather than the 1.8.x branch. What version of CDT did you use to build the system contract? |
|
Testing demonstrates that changing |
|
I can confirm that changing those values works, but the issue that we're facing is that no matter if we set these values to something unreasonable like 20 seconds per block or we use mainnet values (0.2s for max block CPU usage and 0.15s for max transaction CPU usage), it doesn't guarantee us that this transaction won't exceed the deadline. We counteract this by repeatedly submitting it until it succeeds and ultimately it does, usually requiring only about ~0.01-0.03s of CPU time. There must be some reason why this transaction either exceeds 0.15s deadline or gets executed under just a fraction of that time. We use eosio.cdt 1.6.3 as advised in release notes. Using v1.8.3 of system contract doesn't seem to resolve the issue. |
|
Reopening. I'm getting three independent reports of consistent transaction timing failures in the 1.8 branch, ranging all the way from 1.8.0 all the way up to 1.8.6. |
|
I am experiencing this same issue when deploying EOS Version: 1.8.9 Not sure exactly when it appeared but it is preventing local testnet deployment. |
|
I was able to solve this problem by adding The default value is 30ms which may not be enough on a local, single node testnet |
|
Note |
|
Correct, the original poster said.
So when he fixes the The newer versions of EOSIO defaulted the http-max-response-time to 30 to prevent RPC node abuse. As you have to use the RPC node to set the contract, the imposed limit caused issues on some hardware in a local testnet. Please at least attempt the above setting in a testnet before declaring it does not fix the described issue. Hopefully this information helps someone as it seemed to help me running into the same issues. I have seen some form of these errors constantly when making changes that did not include increasing allowed http response time. I suppose it is possible the original commenter was using outdated eosio.contracts that didnt work with his EOS version, but I had this issue with multiple versions of eosio.contracts. |
|
Thanks for the additional info. |
|
Thanks @netuoso. I tried your solution but it doesn't seem to make much impact for me. I'm using eosio 1.8.9, eosio.cdt 1.6.3 and eosio.contracts 1.8.3. I've set Here is the output from |
|
Apologies, it helped me for deploying a few times but now I am re-encountering the issue. Update: I am using an underpowered laptop and running the testnet in a docker instance via docker-compose. After some digging, it seems the issue I was experiencing with I was able to make my machine immediately start working by increasing the available RAM to the docker process as well as setting a CPU niceness of -18. https://en.wikipedia.org/wiki/Nice_(Unix) TL:DR; if you are experiencing this issue, try to upgrade your hardware or ensure your virtual process has sufficient resources to deploy the contract within the timespan of a single block. |
|
Appears to have been a system resource issue. |
|
We're having these issues when running this on CI and to be honest I wouldn't call it underpowered. We currently run builds on I tried changing the niceness, but it still consistently takes at least 2 attempts to submit the contract. My machine also doesn't seem to be under much load while running it. |
|
I prepared a Docker image and a simple script to reproduce this issue. The idea behind the script is that we're trying to bootstrap EOS testnet until it succeeds. Sometimes 1st attempt to deploy I ran these tests against an r5.large EC2 instance running Ubuntu 18.04 and on my Mid-2015 MacBook Pro (Core i7, 16 GB RAM). My results are here if anyone wants to dig in: My Docker Desktop config: |
|
This maybe the time-out issue on keosd. Can you try to following (sign the transaction manually without invoking keosd)? |
|
Interesting. So if I understand correctly, you suggest that the transaction itself might be already expired by the time it's signed and ready to be submitted? That would make sense, looking at it again the logs mention that the deadline, not CPU usage is exceeded. I adjusted the bootstrapping script of my Docker image accordingly and ran it again. I added test results here: https://1drv.ms/x/s!ApF6BBqdJeSZkaQ4u3tTOs4cEswMEQ?e=PntAbv What's interesting about the results, that 98 out of 100 builds were successful at first try. The 2 ones that failed had the error mentioned in the 1st message in this ticket: I attached the log files here: https://1drv.ms/u/s!ApF6BBqdJeSZka8r05JuZtH-4R2YiQ?e=ETt3ac It's as if there were 2 cases:
Our code is available here: ulamlabs/eos-testnet#2 |
|
Can you add |
|
It appears to be enabled: When I submit it manually, the response does include the traceback but it's not really helpful. The weird part is that all runs have the same config and it doesn't matter how many times you try to submit it, it fails with this exception. If I kill the container, remove all its data and spin it up again, it will probably be fine. In fact, I spinned up another container based on the same image and it bootstrapped just fine. Not sure how useful this is, but here is also broken nodeworking node |
|
to resolve increase the default 30 ms for configuration variable http-max-response-time-ms used by the http plugin in keosd. default is not enough to set the contract for eosio.system using cleos set contract on resource constrained environment. |
Hey,
when I run custom testnet
nodeoswith custom genesis file, I randomly get below error when pushingeosio.systemcontract.I use eos
1.8.5fromeostudio/eos:v1.8.5docker image andeosio.contractsfromrelease/1.8.xbranch plus following genesis and config files:{ "initial_timestamp": "2018-03-02T12:00:00.000", "initial_key": "EOS55oVd4ZyC2QMiAMzs1WFYzEYU1GKAAALPCa45Dzs9wrUWo3hjD", "initial_configuration": { "max_block_net_usage": 1048576, "target_block_net_usage_pct": 1000, "max_transaction_net_usage": 524288, "base_per_transaction_net_usage": 12, "net_usage_leeway": 500, "context_free_discount_net_usage_num": 20, "context_free_discount_net_usage_den": 100, "max_block_cpu_usage": 100000, "target_block_cpu_usage_pct": 500, "max_transaction_cpu_usage": 50000, "min_transaction_cpu_usage": 100, "max_transaction_lifetime": 3600, "deferred_trx_expiration_window": 600, "max_transaction_delay": 3888000, "max_inline_action_size": 4096, "max_inline_action_depth": 4, "max_authority_depth": 6 } }To easily reproduce it I prepared a repo with Dockerfile: https://github.com/ulamlabs/eos-regtest.
Just build the image and run it couple of times.
What I tried?
Increasing max CPU time in genesis file - then the pushing the system contract takes seconds and errors with
Please increase the expiration time of your transaction!.Thanks for helping.
The text was updated successfully, but these errors were encountered: