Reinforcement Learning for QuantumControl.jl

[Fe-r-oz]

Hi @goerz , Nice to meet you! QuantumControl.jl is awesome. I would be more than excited to contribute to the development of Reinforcement Learning Functionality for QuantumControl.jl. Really nice to see Krotov and Grape already implemented!

Please let me know how to get started so that we can incorporate ReinforcementLearning.jl . In the meantime, I will become more familiar with the workflow and the package.

Please let me know ideas for beginners that you might have in mind for introducing RL functionality. I will excited to look into it and discuss further.

Best Regards,
Feroz

[goerz]

You would start by creating a new package (maybe QuantumControlReinforcementLearning, or QuantumControlRL or QCReinforcementLearning, or maybe you can come up with a good name) that follows the same structure as, e.g., GRAPE.jl. You can start it under your own account, but once it gets to a point where it might be ready for registration, we can move it to the JuliaQuantumControl org.

Some of the key points:

• Use a "workspace object" where you pre-allocate any internal data used in the optimization. In general, make it as similar to the existing control methods as possible.
• Have a result object similar to the result objects in the existing package
• Implement a optimize_rl function and define the method for QuantumControl.optimize so that optimize(problem; method=QuantumControlReinforcementLearning) delegates to optimize_rl (or whatever name you choose)
• All simulation of dynamics must be via QuantumPropagators.jl.

So getting familiar with QuantumPropagators and the general structure explained in the Glossary would be an important prerequisite.

Another important question is at what level you would want to optimize controls. If it's the values of the control field at each point in time ("piecewise constant") you can closely follow the structure of the GRAPE package. If it is general parameters of analytic control functions, it would be closer to the (unfinished/WIP) ParameterizedQuantumControl.jl package

Did you have a particular quantum control paper in mind that describes the method you'd want to implement?

[Fe-r-oz]

Thanks for your comments.

Your documentation is so clear, I like reading it.

As for the paper, I have this in mind:

• Soft-Actor-Critic (SAC): https://arxiv.org/pdf/1801.01290
  -- https://github.com/haarnoja/sac
  -- https://github.com/pranz24/pytorch-soft-actor-critic

• As for Quantum Control Paper, 'Sample-efficient Model-based Reinforcement Learning for Quantum Control': https://arxiv.org/pdf/2304.09718
  -- SAC (as base for comparison) -- (Model Free)
  -- SAC equipped with ODE model (authors work) -- (Model-based)

Another important question is at what level you would want to optimize controls.

Authors: In practice, we solve for a piece-wise constant version of the dynamics represented by N fixed steps of ∆t = T /N of the final time T . Taken from (Page 4, Section C: Discretization)

Please let me know your insights about the feasibility of implementation. Looking forward to your comments.

[goerz]

That looks good! I've only glanced at the paper, but I'm familiar with a lot of the authors' other work, so it should be a good fit, conceptually. I'll have to read it in detail, though. Right now, I'm preparing for a workshop, though, so I won't have time to get into it at the moment. Let me get back to you in a couple of weeks.