TPV rules for efficient scheduling

[bgruening]

• what is needed for a global Galaxy Pulsar network
• how can TPV help here
• data local job scheduling

[bgruening]

@pauldg, @nuwang, @cat-bro

[jlqfab]

@jlqfab is interested

[pauldg]

We've already discussed quite a bit on this topic during the cofest I'll add my notes here and we can schedule a follow up based on that if we feel like we need to. Perhaps good to loop in @jlqfab anyways. What time still suits for you?

[jlqfab]

I'm available tomorrow - times that suit @cat-bro and @nuwang should suit me

[pauldg]

TPV could enable some smarter decisions in terms of the (pulsar) destination to pick through a ranking function that takes into account metrics describing the state of the destination, preference tagging, ...
These metrics include: latency, load, ...
These metrics need to be available and updated in real time for every destination.

• Send a job or configure a cron job that calculates latency and other metrics on the destination
• Dump metrics to a (dedicated) influxdb and have job runners and TPV be able to read this
• A test setup with some different pulsar nodes and object stores
• Helper functions in TPV that do some (basic and/or smart) scheduling based on the metrics