0.9.46 | Wish 🪄
What's Changed
🎉 Exciting New Features
⚡️ Spark UI/UX 2.0 for AWS EMR
It's finally here! Mage now comes with a completely revamped custom Spark UI/UX for our AWS EMR users! This is a huge update that comes with a complete overhaul of every element possible for managing your Spark cluster!
Check out the PR for more screenshots and get started today (docs coming soon)!
by @tommydangerous in #3997
🌊 Streaming: ActiveMQ Source
🔥 @shrutimantri is on fire! In another community PR, she adds streaming support for ActiveMQ as a source! If you're an ActiveMQ user, give it a shot today!
by @shrutimantri in #3978
📁 Download Files & Pipelines via the UI
Another community PR, this one from @PopaRares, allows you to download files and pipelines via the right-click menu in the Mage UI.
This will be a game changer for collabrative projects and importing/exporting data from Mage!
by @PopaRares in #3813
✨ Streaming: NATS JetStream Source
A big shoutout to community member @mfreeman451 for adding the NATS JetStream message broker as a Streaming Source in Mage!
by @mfreeman451 in #3985
🎏 Data integration: Kafka destination
@Luishfs is back at it with another destination— this one for data integration. You can now write DI outputs to a Kafka topic. We can't wait to see what y'all cook up with this one!
🛟 Auto-save triggers in code
Last, but certainly not least, Mage is now able to auto-save triggers as code. That means (when enabled) you can update triggers and have them auto-save to your Mage project. WThis should help you keep track of your trigger changes across projects.
by @tommydangerous in #4009
🐛 Bug Fixes
- Add
pyarrow-hotfixtorequirements.txtby @wangxiaoyou1993 in #3993 - Fix workspace user management fetching by @dy46 in #3992
- Handle hidden block positions when split view by @tommydangerous in #4005
- Don’t show SSH tunnel option unless kernel is PySpark by @tommydangerous in #4006
- Fix Monoco Editor for base path by @dy46 in #4012
- Update OAuth sign on and fix OAuth sign on with
REQUIRE_USER_PERMISSIONSby @dy46 in #4007 - Clarification when applying bookmark to all streams by @johnson-mage in #4036 and #4049
- Allow using block configuration when
run_pipeline_in_one_processis true by @wangxiaoyou1993 in #4046 - Only show Backfills in vertical nav for standard (python) pipelines by @johnson-mage in #4050
- Fix files not being selected in notebook by @tommydangerous in #4051
💅 Enhancements & Polish
- Added application/gzip support to API source by @Luishfs in #3990
- Re-enqueue the job if queue is empty by @wangxiaoyou1993 in #3996
- Add better error message for API source by @wangxiaoyou1993 in #4016
- Pass envFrom to job pod by @wangxiaoyou1993 in #4031
New Contributors
- @mfreeman451 made their first contribution in #3985
