Welcome to q-transformer Discussions!

[lucidrains]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[lucidrains]

can anyone working in robotics comment on how significant this work is. I'll prioritize accordingly, thanks!

[ramkumarkoppu]

I am hoping to use Q-transformer for offline RL pretraing if my future project needs it

[justindujardin]

I don't work in robotics, but I'll definitely try this out with Mathy when you think it's ready. I dropped the trained (A3C/AlphaZero) mathy agent examples a while ago in favor of swarm planning, but I'm interested in bringing them back to check out transformer-based agents that can work with large discrete action spaces.

[justindujardin]

Mujoco isn't bad since DeepMind made it free 👍

fyi: there are other envs in gym that have more complex action spaces, like car racing and bipedal walker

[lucidrains]

ohh nice, I'll check out bipedal walker!

[justindujardin]

@lucidrains How did your exploration turn out? Do you think this could be useful for Mathy? I added a PPO example to demonstrate how I currently train models for Mathy environments. It would be excellent to finally find an architecture that can solve Mathy's "hard" poly simplification environments, but the action space appears too much for PPO (and friends) to handle. 😓

mathy_ppo.mp4

[lucidrains]

@justindujardin hey! yeah i'll get around to it over here. if you can understand the code there, maybe you can help pick up the slack and try it out?

truth is, i'm having a lot of trouble finding a knowledgeable community outside of the big communities like LLMs, vision, and audio somewhat. being met with crickets when it comes to RL, robotics, molecules, and of course, anything that touches bio / medicine. there is no open source advantage without a community and participation, and i'm not sure what to do about these niche fields, as much as i think they matter.

[lucidrains]

@justindujardin have you read the Alpha Geometry paper?

[justindujardin]

@justindujardin hey! yeah i'll get around to it over here. if you can understand the code there, maybe you can help pick up the slack and try it out?

The trouble I have in contributing here is that I'm not sure how Mathy should integrate with LMs yet. Currently, it interfaces with models similar to any other Gym environment: it outputs observations in a floating-point format and only uses text as the input for the math parser that builds an expression tree.

For Mathy w/language models, I think perhaps interfacing with DSPy could be interesting, using LMs to handle the conversion of natural language inputs into ASCII-math expressions that Mathy can read and conversion of output states into natural language explanations based on the applied rule and affected expressions.

One other idea with Mathy is that you could use it to generate large datasets of step-by-step solutions to problems and use that dataset to fine-tune an LLM. Perhaps it overlaps with RLHF more in that kind of use case? I'm pretty ignorant here since I haven't spent much time with LLMs outside of daily ChatGPT use.

truth is, i'm having a lot of trouble finding a knowledgeable community outside of the big communities like LLMs, vision, and audio somewhat. being met with crickets when it comes to RL, robotics, molecules, and of course, anything that touches bio / medicine.

I feel this has been the case for a long time. I don't have much of an audience, but I've tried reaching out to RL people/companies for years now, and it seems like I'm the only one interested in math tutoring with RL 🤷 (or my limited ruleset makes the project appear too basic, or I didn't build a flashy web demo, or something else.)

have you read the Alpha Geometry paper?

I saw it come by, but I still need to read it. I've been working with VAEs and Diffusion models lately, and my time for reading research papers is limited. Was it all hype or did it inspire hope? It's on my list for sure 📖

[lucidrains]

@justindujardin Alpha Geometry inspires a bit of hope, but not for the pure neural approach. They basically used a small transformer to cut down the search space for the main symbolic search. So a bit of a mixed bag of emotions. Yannic has a great video as usual

[lucidrains]

@justindujardin yea, one thing for sure, this idea of placing a cherry on top of the cake did not really transpire in the way LeCun imagined. and its hard to know whether the problem is with the cherry or the cake. i think its possibly both, but maybe it is too early to say

[lucidrains]

The trouble I have in contributing here is that I'm not sure how Mathy should integrate with LMs yet. Currently, it interfaces with models similar to any other Gym environment: it outputs observations in a floating-point format and only uses text as the input for the math parser that builds an expression tree.

yea no worries, i'll try a few things once i circle back to this, maybe put up a simple arithmetic demo (if it works at all)