-
Notifications
You must be signed in to change notification settings - Fork 687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Forecasting beyond 10 days #63
Comments
Hello, I'd like to ask how you made the prediction within ten days. I entered 20 time data, but starting from the third data, the prediction is compared with the original data. I want to predict the time after 20 data. What should I do? If you know, please tell me, thank you. |
Take a look at |
If you check out the #32 and #56 I expect this is your same issue. It has to do with where the inputs are extracted from the example data. |
Thank you. I think I understand a little. I will make changes in an aligned way. Thank you very much for your advice. |
Hi, I'm sorry to bother you again. I operate on the data in a left aligned and right aligned manner. I use data from 1 to 20, and the training and evaluation steps are both 4. Left aligned, the predicted data are 3, 4, 5, and 6. Right aligned, the predicted results are 17, 18, 19, and 20. If I want to predict 21, 22, 23, and 24, what should I do? Thank you in advance |
You need to generate a targets_template of the correct size. It can be filled with nans, but needs to be bigger than what you're currently passing in. You'll also need to generate the matching forcings. |
Thank you very much for your help. I will make adjustments as you said and express my thanks again. |
Timo, when you say it will generate worse error than climatology, does the error ever settle into a climatological plateau or does it grow forever? |
Hi, I have a doubt, if i have my input as 1/1/24 06:00 and 1/1/24 12:00 and my targets are from 1/1/24 18:00 to 1/2/24 12:00 (TARGET SIZE OF FOUR) and i have the forcings of the following, can i iteratively use the outputs and successfully predict so on for a year? |
Are you saying you want to do a one year forecast? That is possible, but
you need to create a target that is one year long. You can concatenate many
targets together to accomplish this. If there is no training, then it
doesn’t matter what they are as long as they are the right dimensions.
…On Tue, Sep 10, 2024 at 01:06 Rohit P Suresh ***@***.***> wrote:
Hi, I have a doubt, if i have my input as 1/1/24 06:00 and 1/1/24 12:00
and my targets are from 1/1/24 18:00 to 1/2/24 12:00 (TARGET SIZE OF FOUR)
and i have the forcings of the following, can i iteratively use the outputs
and successfully predict so on for a year?
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BEAXI3RD37YH4UZY2CZHJITZV2SBRAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZZHE2TOMJVGY>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
Correct, but I'll also have to generate the corresponding forcings which will take a huge amount of time, right? |
No. It’s very quick. The model creates all necessary forcing during the
extract_input_target_forcing function. So if you specify a forecast of 40
steps (10 days), it makes all the needed forcings then.
Be aware that Graphcast has been shown to go unstable after 45 days-ish.
…On Wed, Sep 11, 2024 at 04:13 Rohit P Suresh ***@***.***> wrote:
Correct, but I'll also have to generate the corresponding forcings which
will take a huge amount of time, right?
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BEAXI3XNJRVU4S24V44HVITZWAQT7AVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNBTGM2DENRQGY>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
I have managed to generate the forcings quickly but when i try to run my prediction model for 4 steps, the kernel always crashes. Is there something I can do to optimise my code? |
2024-09-23 11:07:11.281222: E external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an instruction is taking > 1s: %add.675 = s32[3114720]{0} add(s32[3114720]{0} %broadcast.2435, s32[3114720]{0} %constant.620), metadata={op_name="jit()/jit(main)/while/body/mesh2grid_gnn/_process/mesh2grid_gnn/_process_step/mesh2grid_gnn/add" source_file="/data4/home/rohitsuresh/graphcast/graphcast/typed_graph_net.py" source_line=142} this is the error i get |
Which grid and vertical resolution model are you using? What kind of GPU
are you trying to run it on?
…On Mon, Sep 23, 2024 at 06:22 Rohit P Suresh ***@***.***> wrote:
2024-09-23 11:07:11.281222: E
external/xla/xla/service/slow_operation_alarm.cc:65] Constant folding an
instruction is taking > 1s:
%add.675 = s32[3114720]{0} add(s32[3114720]{0} %broadcast.2435,
s32[3114720]{0} %constant.620),
metadata={op_name="jit()/jit(main)/while/body/mesh2grid_gnn/_process/mesh2grid_gnn/_process_step/mesh2grid_gnn/add"
source_file="/data4/home/rohitsuresh/graphcast/graphcast/typed_graph_net.py"
source_line=142}
this is the error i get
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BEAXI3ROYLCWBZH3DL2YE4LZYAIZXAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRYGIZTCOJWGA>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
Its an nvidia a100 with 40gb ram and Im using the GraphCast_operational-ERA5-HRES_1979-2021-resolution_0.25-pressure_levels_13-mesh_2to6-precipitation_output_only.npz |
Hi, i was able to run it finally, but it took 20 minutes to make the predictions for 4 steps. Is this normal or can it be made faster? |
That's still very slow. With an a100 you should be able to do 4 steps in
30-40 seconds. Do you have the cuda-enabled version of jax in your
activated environment? I bet this is the problem and it is failing back to
CPU.
https://jax.readthedocs.io/en/latest/installation.html
…On Mon, Sep 23, 2024 at 19:35 Rohit P Suresh ***@***.***> wrote:
Hi, i was able to run it finally, but it took 20 minutes to make the
predictions for 4 steps. Is this normal or can it be made faster?
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BEAXI3U4KBGGNZNA35QVFQTZYDFYZAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRZHE4TCNRVGQ>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
jax definitely uses my gpu. When i run gpustat, it shows that I'm currently
using the gpu.
…On Tue, Sep 24, 2024 at 11:08 AM Tre ***@***.***> wrote:
That's still very slow. With an a100 you should be able to do 4 steps in
30-40 seconds. Do you have the cuda-enabled version of jax in your
activated environment? I bet this is the problem and it is failing back to
CPU.
https://jax.readthedocs.io/en/latest/installation.html
On Mon, Sep 23, 2024 at 19:35 Rohit P Suresh ***@***.***>
wrote:
> Hi, i was able to run it finally, but it took 20 minutes to make the
> predictions for 4 steps. Is this normal or can it be made faster?
>
> —
> Reply to this email directly, view it on GitHub
> <
#63 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/BEAXI3U4KBGGNZNA35QVFQTZYDFYZAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRZHE4TCNRVGQ>
> .
> You are receiving this because you modified the open/close state.Message
> ID: ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWGHF3ZNE6AEDOAZL54CKOLZYD3E5AVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQGIZTINBXGY>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
If that’s the case, then I do not know what the problem is. Maybe try a
verbose log file for more details. Good luck.
On Tue, Sep 24, 2024 at 03:13 Rohit P Suresh ***@***.***>
wrote:
… jax definitely uses my gpu. When i run gpustat, it shows that I'm
currently
using the gpu.
On Tue, Sep 24, 2024 at 11:08 AM Tre ***@***.***> wrote:
> That's still very slow. With an a100 you should be able to do 4 steps in
> 30-40 seconds. Do you have the cuda-enabled version of jax in your
> activated environment? I bet this is the problem and it is failing back
to
> CPU.
>
> https://jax.readthedocs.io/en/latest/installation.html
>
>
> On Mon, Sep 23, 2024 at 19:35 Rohit P Suresh ***@***.***>
> wrote:
>
> > Hi, i was able to run it finally, but it took 20 minutes to make the
> > predictions for 4 steps. Is this normal or can it be made faster?
> >
> > —
> > Reply to this email directly, view it on GitHub
> > <
>
#63 (comment)>,
>
> > or unsubscribe
> > <
>
https://github.com/notifications/unsubscribe-auth/BEAXI3U4KBGGNZNA35QVFQTZYDFYZAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRZHE4TCNRVGQ>
>
> > .
> > You are receiving this because you modified the open/close
state.Message
> > ID: ***@***.***>
> >
>
> —
> Reply to this email directly, view it on GitHub
> <
#63 (comment)>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AWGHF3ZNE6AEDOAZL54CKOLZYD3E5AVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQGIZTINBXGY>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
—
Reply to this email directly, view it on GitHub
<#63 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/BEAXI3XARXI3BLI3WUAE5ETZYE3MFAVCNFSM6AAAAABDSOJJNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQHA2TENBUGM>
.
You are receiving this because you modified the open/close state.Message
ID: ***@***.***>
|
There's seems to be some default setting that won't allow a forecast beyond 10 days. I've tried identifying where in the code that might be and have not been able to find the spot.
Anyone successfully do this or know where to point me for fixing it?
The text was updated successfully, but these errors were encountered: