Performance within PSaaS V7

[BadgerOnABike]

Rob and Wayne were having a discussion regarding performance in PSaaS v7, this will be logged here.

[BadgerOnABike]

NUMA Node Conversation

This is in chronological order

---

Hi Robert
Here is the info on my 4-core production machine (the dev machine is the same except it has 4 sockets ( 2 cores per socket, 8CPUs, 1 thread per core)

Looks like all cpu are on the same NUMA node.

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
Stepping: 0
CPU MHz: 2596.992
BogoMIPS: 5193.98
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 35840K
NUMA node0 CPU(s): 0-3

Cheers
Wayne

---

Hi Wayne,

+Brett, again.

How did you generate this?

The E5-2690V4 is a 14 core machine, hyperthreaded. It’s actually surprisingly similar to mine – E5-2697 V3. Your report is interesting, so I had to try it on my machine too.

My machine is reporting one processor group, but 2 NUMA nodes (since there are two physical processors). Without knowing what you generated this on, I would guess that each physical processor would report the same as mine. But, if you are running any virtualization software between the hardware and OS, that could be fibbing to you. This is what we were referring to: virtualization of any sort tends to get in our way unless you know it isn’t fibbing to you. We need to make sure you run on only 1 NUMA node, and you don’t really want it to migrate your OS to something else either, when we’re playing with core affinity.

--Rob

---

Here is the response from IT (the image matches what I see using lscpu – i.e. 2 cores per socket = 4 sockets for 8 CPU)

I spoke to Jarrad about this. He stated that the servers are NUMA compliant and that you shouldn’t have any issues if the vCPU’s are on different cores, which these are.

Wayne

---

Hi Wayne,

We have run Prometheus and PSaaS across NUMA boundaries without issues but the performance really degrades, which is why we want to lock it to one specific (hardware) NUMA node, and why you have to be careful about the virtualization.

--Rob

---

IT are willing to try adjusting settings to ensure single NUMA node – so I’ll do some runs before and after and let you know what I find.

Perhaps if there is a new version of PSaaS on its way soon I should wait for that?

Cheers

Wayne

[BadgerOnABike]

Conversation on Performance

This is in chronological order

---

Hi Wayne,

+Brett because this covers some of what he and I have covered lately.

Comments inline quoted.

Hi Rob
I tried an example using 2021.03.03 that was taking 20 or so minutes to process that I thought might be influenced by the perimeter reaching the 6km buffered boundary with a 7km buffered boundary (perimeter not reaching the boundary) but this did not significantly change the result.
What I did see was that in this example it was the total_fuel_consumed grid that was taking most of the time (with the statistics perhaps taking a couple of minutes).

I then looked to play around with the hardware settings
[“settings”][“hardware”][“cores”] == CORES
[“settings”][“hardware”][“processes”] == PROCESSES
[“project”][“scenarios”][“scenarios”][0][“threadCount”] == THREADCOUNT

In the examples to date the settings here have been CORES=4, PROCESSES=2, THREADCOUNT=7

Changing the CORES=8 – does indeed reduce the time (from around 21 minutes to 13 minutes).

CORES is the number of cores that the current PSaaS process will use. Currently, it is used to determine how many concurrent scenarios will run, but also the # of threads that a scenario export operation can use at a time, which explains why you found greater performance.

Playing with PROCESSES and THREADCOUNT made little difference (not that I’m sure what good settings would be).

THREADCOUNT is specific to a scenario. Prometheus uses this to determine how many threads that a specific scenario can use at once, for simulating. We may want to consider deprecating this with Prometheus. You can find some gain there, so long as the number is kept relatively low. But in PSaaS, where we can leverage coarser-grained multithreading, this is best left at 1 and instead of multithreading inside a specific simulation, multithread by running more than one simulation at once. If you are running PSaaS to only run 1 simulation at a time, this option may be still useful to you.
PROCESSES is instructing Manager how many instances of PSaaS to run at once. It may be appropriate to also review this setting since we’ve added a lot of intelligence into Manager since we first did this. PROCESSES isn’t even used by PSaaS.

The specific math for the maximum number of cores that a process will use, regardless of what you tell it is:
(number_processors * 3 + 3) / 4
Excel shows this result:

cores	max	OS
8	6	2
12	9	3
16	12	4
32	24	8
64	48	16

Manager has similar logic built into it, but this equation is the maximum that any PSaaS process will consume. How did this come about? Basically, Win10 (and final versions of Win7) has abandoned traditional thread prioritization and management for something that they think provides a more responsive user experience, and it’s caused us grief. If PSaaS creates lots of low priority threads, those are capable of impacting higher priority threads and even processes. So we purposely set aside “OS” # of cores for OS tasks. If we don’t do that, then Manager becomes unresponsive, even though it’s a separate higher priority process (Java). We were seeing MQTT network packets drop… And we were seeing other negative behaviour out of Win10 system tasks too. Setting aside this number of cores keeps Win10 and the rest of the PSaaS infrastructure a whole lot happier. In PSaaS, we have to go a step further when we set up affinity and set aside one core to just deal with MQTT traffic – again, Win10 is ignoring standard thread priorities. We haven’t experimented with this on Linux, though.

I do note that with CORES=4 the highest cpu is around 300% i.e. it looks like one core is waiting for the others.
Here are some screenshots from the Ubuntu htop command while running with CORES=4 on a 8-core machine

This one is early on – which seem to be the main processing (before outputs are created) – here one process (top) appears to waiting on the others and just one of the children is runs at 100% while the others run at 20-30%. I believe the 152% of the parent process represents the sum of all processes.

Later in the processing when outputs are being generated we get this – where there are 3 cores in use – some cores with multiple processes – but max CPU reaches 300% no more

Here you see CPU=1 does not seem to the overly used – this is how it roughtly looks the whole time the job is executing.

I then ran the same job through Prometheus 2021.03.00 – and I get this sort of behaviour

Where 6 out of 8 logical processors are used at 100% (but with close to 100% utilization ??) – this could be due to how these processes can turbo boost the frequency (but perhaps not using all cores). Or it could be a similar thing to the linux example where ‘one’ core is not being overly used.

We are about to go a step further in Prometheus, since it will attempt to use the maximum of the above cores, as mentioned. However, once you get past about 8 threads for simulating, thread synchronization incurs too much overhead. Brett and I have discussed this at length.

There seems to be (at least) 2 ways where we want to run PSaaS:

1. Batch-style where we have potentially many instances running at once on the same hardware
2. In a real-time environment where we are okay with unused hardware resources.

We have a command-line option for 1) to set PSaaS process affinity, which applies to all operations (simulating and generating stat’s). That works well and we can maintain very high processor utilization (again keeping a few cores idle for the OS, network, Java, and other system tasks).
In the fall, we tested PSaaS operationally where we discovered usage 2) where although we want to limit simulating to the most of 8 cores, we may want to use more cores (possibly as many as the physical processor has) to generate statistics. And we may want to opt to have a different affinity mask for “simulating” cores vs. “stat’s generating” cores, again to further increase through-put. We could do this for 1), but it would only make sense when a system has a lot of processor power and is limited in RAM.

Brett decided that the 2nd mode – where “stat’s generating” cores could have a different affinity mask from the simulating cores, isn’t of interest to him right now, particularly in V6.

I’m not sure what controls how many cores, etc that Prometheus 2021.03.00 uses (I think you said that it will use all that are available).

I also observed similar behaviour when running with CORES=8 on an 8 core linux system – 1 core is not utilized and the max CPU% reported is 700%.

I believe that Linux is smarter than Windows on all aspects of thread management. But/and/so we may want to disable the max core limitations that you’ve witnessed for your Linux runs, and then you can let us know if there are any ill side effects. But please let me know, since it may not be a bad idea, even on Linux, to keep some cores around for work other than PSaaS.

Not sure what this all means but thought you might find it interesting.

Cheers
Wayne

---

Hi Rob
So if I had a combination like this on linux what is the best settings

• 8 core machine
• Somewhere between 1 and 20 psaas jobs to run at a time. (4 would be typical)

As I now the number of jobs in a batch before I start I guess I can alter the settings per batch.
Would you recommend (when there are > 8 psaas jobs) running one core per psaas (with 8 psaas running at once)? Would this be faster than running one psaas at a time on 8 cores?(which is what I currently do).
Similar for 4 jobs would two cores per psaas be best? – I think in this case each psaas process would only effectively use 1 core even if you specify 2?

I think you might need to rethink how this works on linux – but at the moment I’m more interested in sorting out the very long running jobs that pop up on occasion (the typical job is not very long running).

Cheers
Wayne

---

Hi Wayne,

Plan for 8 cores (ideally) per PSaaS instance. We’ve run with 5-6 and it still runs good, and then as stated, performance drops off past about 10-12 threads. If you want to run 4 PSaaS instances, then plan for 32 cores. Then you need to look at RAM. We’ve been running those 8 instances on machines with 256GB+ RAM.

We did experiment with fewer (like you were stating), and you spend more time loading and saving, rather than running jobs. And we’ve tried to stick to about 400-600 scenarios per FGMJ too, any more than that and the jobs get very large. Any fewer and the starting and ending I/O seem to consume too much time.

PSaaS Manager will let you do some configurations. Have you looked at that recently? We use 1 Manager per machine, then our own “manager of managers” to coordinate execution across many machines.

Linux: I believe you, but right now everybody here is still living with V6 on Windows so that’s where I’ve been focusing.

--Rob

---

Rob,

Wow- 400-600 scenarios! Could I ask what changes between scenarios? I’m currently only running one scenario but had been thinking about 3-4 to show some potential differences.

As to Manager – I’m still not convinced on its GUI nature – however if I am going to run multiple psaas together perhaps I will need to look at it but I suspect with only 8 cores its not worth too much effort.

I’m not sure how high I can push the CPU number on our VMware server – I could look at a deploy onto azure and see what happens.
Are NUMA nodes an issue on azure as well?

Cheers
Wayne

---

Hi Wayne,

This may help, since it talks about some details in configuring virtual machines and NUMA nodes, and I think it’d apply to Azure too. When we were talking to Oracle about hosting some of our work on their infrastructure, we all came to the agreement that we had to be as close to the hardware as possible, so they gave us some bare metal machines to run on for experimentation.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn282282(v=ws.11)

Last year with the risk modelling, I think we did a total of 3.2 million scenarios? Here, we were modelling a potential ignition every 250m under hydro-electric power lines across all of Alberta. And for each location, we’d have 2 or 3 seasons, 100 weather streams per season. That multiplies out pretty quickly.

For real-time modelling I was expecting to do the following:

• 1 Ignition point
• 1 main fire perimeter for the choice weather stream
• Another 50 fire perimeters to provide a probability gradient based on other weather predictions
• Possibly perturb ignition details (dx, dy, dt) or specifics on the fuel grid, which would multiply the set again. But for each fire, at least ~50 scenarios in one run.

--Rob