Version 0.14.8

Version 0.14.8

Version 0.14.7

Version 0.14.7

Version 0.14.6

Highlights:

• Adding metrics sink: WebSink (#1615)
• Dry-run (#1629)
• Statically link libunwind (#1631)
• Added a topology status to the list all topologies ui. (#1646)
• Support connection buffer metrics between stmgr and instances (#1695)
• Refactor of runtime management control flow (#1618)

Release notes:

• ISpoutDelegate and IBoltDelegate to implement IUpdatable (#1749)
• Real time I/O redirection (#1726)
• Modifying configs at submit if numContainers is off (#1743)
• Adding --help option to Heron client (#1739)
• Modifying configs at submit if numContainers is off (#1743)
• Fixing bug in WebSink (#1714)
• Adding --help option to Heron client (#1739)
• Slightly increase Python unit test assertion delta. (#1736)
• Remove special case CentOS build files. (#1725)
• Eclipse config typo test~>tests (#1733)
• Fixing bug in WebSink (#1714)
• CentOS 7 file path preparation (#1713)
• Change SystemConfig to use SystemConfigKey instead of constants (#1717)
• add testing class for compatibility (#1719)
• Use resources label as much as possible (#1710)
• Change YarnKeys to enum (#1707)
• use raw Map in storm api (#1715)
• Support connection buffer metrics between stmgr and instances (#1695)
• add accessrule for eclipse to mititage ide display error (#1711)
• Rename Misc class to TokenSub class (#1708)
• Removing SANDBOX_ from the AuroraField and aurora properties (#1709)
• Move Command class from spi.common to scheduler (#1705)
• Change Misc pattern substitution logic to match any Key value (#1706)
• Removing sandbox enums that have duplicate concepts (#1702)
• Adding unit test for AuroraScheduler.createAuroraProperties(..) (#1704)
• Upgrade node.js (#1703)
• Remove scheduler SM adaptor for submission dry-run (#1697)
• Temporarily disable MultiSpoutsMultiTasks test. (#1700)
• Remove defaults.yaml in favor of enum Key approach (#1696)
• Change dryrun to not include format args if dry run is false (#1694)
• Replaces keys.yaml and Keys with Key enum (#1693)
• Clean up config visiblity and dead code (#1689)
• ignore the null value config in the map
• Standardize Java formatter string. (#1688)
• Display error message for NetworkErrorCode (#1678)
• added null check when heron reads the config file (#1682)
• Regexify Java header check file to relax header checking (#1679)
• Change AuroraController to an interface (#1672)
• Avoid notifying when no one is waiting on proceedLock.
• Remove notify in SlaveLooper.wakeUp. It is useless and affects performance (#1657)
• Fix aggregate topology metric dot colors for firefox and safari (#1655)
• cleanup glog build file and protobuf third party build file (#1636)
• Check at launch that configured container count same as packing plan (#1630)
• Consolidate more common failure tests in CommonPackingTests (#1625)
• Refactor of runtime management control flow (#1618)
• Refactor ByteAmount checking args method (#1621)
• Remove unused c.t.h.spi.common.Jars (#1620)
• using guava Preconditions in UpdateTopologyManager (#1613)
• Move Scheduler related spi util classes into schedulers package (#1617)
• Uses consistent scale-down algorithm for both IRepacking classes (#1606)
• Wait for a new pplan properly (#1605)
• Submission control flow improvment (#1571)
• Merge 0.14.5 release back to master (#1611)
• Use enum for package type (#1609)
• Change scale down to try to free containers (#1584)
• Updated SCP upload's sample.yaml to reflect config changes
• Refactor duplicated code in SubmitterMain and SchedulerConfig (#1599)
• Implemented seperate SSH/SCP config parameters for options and connection string
• S3Uploader support for AWS profiles and Default Credential Provider Chain (#1589)
• Adding new BytesAmount class for use by Resource (#1586)
• Move taskId/componentIndex management into PackingPlanBuilder (#1582)
• Use BaseMemPool in tuple-cache (#1585)
• Missing instruction: need to install heron-tools (#1583)
• Fix fd leak in http_unittest (#1581)
• Add more tests for PackingPlanBuilder (#1580)
• Add a scoring-based approach to modifying packing plan (#1572)
• Follow symbolic links when searching status files. (#1578)
• Fixed yarn scheduler doc typos (#1575)
• Broke out packing builder and utils into separate packages (#1565)
• Remove unnecessary ack() in bolt (#1570)
• Add contribution guideline (#1564)
• Clean up some logging found while testing (#1563)
• Colocate TMaster and Scheduler (#951)

Version 0.14.5

Highlights:

• Manual scaling using heron update
• Faster memory allocation in tuple cache (#1532)
• Optimize YARN container allocation (#1522)
• Send back pressure related metrics to TMaster (#1503)
• Limit the count of exceptions inside a MetricPublisherPublishMessage (#1481)

Release notes:

• Minor fix of link checker, fix broken links (#1562)
• Fix string formatting bug (#1554)
• Refactor test_runner to use status classes instead of success/fail (#1553)
• Use std::vector for fields grouping indices (#1551)
• Reduce verbosity of zookeeper logging (#1547)
• Using common python logging for int tests (#1550)
• Add tuple-serialization-time-ns metrics back (#1549)
• Remove failure logging for cleanup and duplicate cleanup method call (#1548)
• Change local integration tests to use test status objects instead of True/False (#1546)
• Use pipe to properly block clients when server hasn't entered event loop. (#1541)
• Clean up ZK lock resources upon kill (#1543)
• Reference symlinks when building topology tar, fixes #1538 (#1539)
• Catch error when killing processes if process doesn't exist (#1540)
• Move methods from PackingUtils into PackingPlanBuilder (#1529)
• Remove numAdjustments from ResourceCompliantRRPacking (#1531)
• Change scaling int tests to constantly emit a stream of tuples throughout scaling event (#1524)
• Change std::list to std::deque for tuple cache (#1532)
• IPacking and IRepacking now throws PackingException (#1530)
• Add PackingPlanBuilder (#1521)
• Reduce logging during packing plan change (#1528)
• Optimize YARN container allocation (#1522)
• Check for null before Slave stops instance (#1527)
• fix minor typo (#1526)
• Add the binpacking package to the CentOS 5 build (#1519)
• Remove unused packing plan code (#1520)
• Adding CLI documentation for the update command (#1518)
• Adding UpdatableComponent for instances that need to be aware of runtime changes to plan (#1510)
• Refactor usage of PhysicalPlanHelper, dedup simulator code (#1516)
• Add version in tracker and UI (#1507)
• add TupleUtils class (#1505)
• Minor cleanup of Slave class (#1506)
• Change integration test harness to support scaling tests (#1500)
• Send back pressure related metrics to TMaster (#1503)
• combine log.warning of tunnelHost into exception message (#1498)
• Refactor integration tests to share more code (#1494)
• Use argparse in executor and make chmod not fail in any case (#1493)
• Add the ability to store state as json in http_server for testing (#1492)
• add warning log of tunnelHost before throw exception (#1495)
• Add support for running integration tests in local mode (#1491)
• count bolt should ack (#1486)
• Add jvm arguments for gc log rotation (#1490)
• Refactor integration test to idiomatic python (#1488)
• Repack implementation for the Resource Compliant RR algorithm (#1471)
• Simplify how to run aurora int tests locally (#1487)
• Do not let external processes block on IO buffers in ShellUtils.runSyncProcess (#1477)
• More configurable Aurora (#1420)
• Limit the count of exceptions inside a MetricPublisherPublishMessage (#1481)
• When getting config of a topology, tracker should always return valid json object. (#1483)
• Add try and catch block when handling exceptions (#1482)
• Avoid duplicated exception wrapper during reportError() (#1480)
• Make metrics mgr fail fast when unexpected errors happen. (#1473)
• Only process update requests if there is a parallelism change (#1478)
• Using static imports for Assert and Mockito in statemgr tests (#1476)
• Make Python topology adapt to GPR #1421. (#1468)
• Prevent concurrent scaling events from occuring (#1459)

Version 0.14.4

Highlights

• Stream manager protobuffer optimizations
• Throughput increase by 4-5x compared to previous versions of Heron
• Add more metrics for stream manager queues
• Bug fixes based on feedback and doc fixes

Release Notes

• Change location of local lock to be consistent depth with other state objects (#1472)
• Remove usage of NullFoo objects (#1469)
• Implement IScalable scheduler for the YARN scheduler (#1431)
• Using static imports for Mock and Assert in scheduler-core tests (#1470)
• Adding distributed locking support (#1463)
• Add back the "import StringIO" to javaobj.py (#1464)
• Add unit tests to verify yarn launcher config construction (#1460)
• Clean the java example topologies (#1461)
• Moved methods to packing utils (#1458)
• Deactivate topology before scaling (#1412)
• Added padding constraints during repack (#1441)
• Add support for SSH sock proxying to scheduler (#1456)
• Add driver memory config for yarn scheduler (#1454)
• Update the compiling doc web page. (#1444)
• Add missing storm api class IMetricsConsumer (#1445)
• Define tunnel to be an empty list in the init method of StateManager stop init from being a abstract method
• Fix wrong key type in LocalScheduler.addContainers (#1426)
• Stream manager performance improvement (#1421)
• Add info logging for failed connections, this is the normal state either when using a tunnel or a recoverable state if multiple Zk nodes are declared
• Turn off Javadoc Checks for tests/.* (#1434)
• Changing ZkStateManager and StateManager to accept a set of host, port pairs rather than a single host port
• Supports to submit a topology to the specific yarn queue. Added YarnContext, default config in conf/yarn/scheduler and doc.
• Must delete packing plan before updating it (#1432)
• './scripts/setup-intellji.sh' will check if 'scripts/compile/env_exec.sh' exists (#1429)
• Permit the ability to scale down (#1433)
• Merge pull request #1404 from vidit-bhatia/vibhatia/tshootjavatopo
• Use C++ auto keyword when possible (#1415)
• Add metrics for stream manager queue size (#1397)
• Ability to run just one test in local_test_runner (#1407)
• Adding exception handling to scheduler responses (#1406)
• Use an enum for the TMaster commands instead of a string (#1409)
• local state manager to check for path before trying to read file (#1408)
• Support for scale down (#1394)
• Persist scheduled container size in packing plan (#1398)
• Added the steps for debug java topologies
• Remove double quote. (#1403)
• Include heron-py-instance in heron-core/bin for CentOS 5 (#1402)
• Improve Python program metrics collector, add test. (#1392)
• Use container id provided by packing plan (#1395)
• Support more s3Uploader options
• Change executor to not assume contiguous container id set (#1387)
• Removing PackingPlan.getInstanceDistribution (#1386)
• Update YARN scheduler doc for the 0.14.3 version
• java.lang.Object cannot be formatted using %d, should cast to integer
• Replacing colon-delimited Instance.id with typed InstanceId object (#1382)
• Proper check of the existence of SSH tunnel attribute (#1380)
• Speed up local test runner and enhancements for scaling (#1381)
• Repacking to use globally unique task ids (#1377)
• Adding more debugging info to integration test topologies (#1378)
• AuroraSchedler implements IScalable (#1360)
• Add integration test for scaling up (#1375)
• Add ubuntu16.04-based docker containers
• Refactor local test runner (#1366)
• Update heron cli to support update command for scaling (#1363)
• Change python state manager home resolution (#1367)
• Repacking Implementation for the First Fit Decreasing algorithm (#1364)

Version 0.14.3

Highlights

• Support for Kafka spout
• Support for Mesos/Marathon scheduler
• Support for Bazel 0.3.1
• Several bug fixes, and doc fixes

Release Notes

• Update doc to use Bazel 0.3.1.
• RuntimeManager changes required for scaling (#1334)
• Simplifying FirstFitDecreasingPackingTest and other cleanup (#1359)
• Adding LocalScheduler scaling impl with UpdateTopologyManager (#1333)
• Change hashCode() call on numbers to be java 1.7 compliant (#1352)
• python development tools installation added to the documention
• Add new python config loader class for state manager (#1344)
• Add checkstyle for kafka and storm code
• Fix and improve unit test.
• Remove PackingPlan.resource, fixes #1330 (#1332)
• Improve TMaster return message when metrics is not available (#1336)
• Add kafka spout for heron
• modify heron-api and heron-storm artifactId (#1341)
• Use executor id as int type in the YARN scheduler (#1342)
• Merge pull request #1339 from mycFelix/setup-intellij
• Modify the directory from contrib/kafka-spout to contrib
• make contrib/kafka-spout to be source code by setup-intellij.sh
• Adding new Scheduler interface methods for scaling (#1321)
• Change PackingPlan Resource class to be immutable (#1326)
• Make system config global (singleton as a module) (#1300)
• Remove duplicate invocation of Packing algorithm (#1316)
• Global metrics for PyHeron (#1319)
• Change executor to load packing plan from StateManager (#1313)
• Delete back-pressure message to avoid memory leak (#1318)
• Add more default metrics for Python topology (#1309)
• Use Optional instead of null
• Add global metrics module.
• Check whether the topology dir exists before collecting topologies (#1312)
• Adding annotations to specify interface usage (#1304)
• Allow exception to propogate to the caller
• kafka spout patching (#1299)
• add option to pass any additional classpath while launching a topology (#1245)
• Refactor state manager locations to prevent #1303 (#1307)
• Repacking Interface to be used for autoscaling (#1285)
• Fix broken CuratorStateManagerTest
• adding packing plan to state manager initTree() (#1305)
• Refactor HeronExecutor signal handling (#1302)
• Improving logging and retries of local int test runner (#1296)
• Clean up executor logging in LocalScheduler (#1288)
• Fix checkstyle errors and improve exception handling
• Kill and request new container(s) on restart
• verify topology when submitting to simulator (#1298)
• Fix code block rendering. (#1287)
• Update Launcher to set PackingPlan in StateManager (#1284)
• Adding packing plan methods to SchedulerStateManagerAdaptor (#1282)
• Added Streamparse copyrights to PyHeron API. (#1281)
• Move cli, explorer, tracker and ui and tools' common files to /tools (#1276)
• Add the ability to store PackingPlan in the state manager (#1274)
• Add NotCompatibleError for invalid streamparse apis (#1278)
• Adding PackingPlan protobuf object (#1219)
• Remove duplicate code from state manager tests (#1270)
• Separate a new page for Python topologies. (#1271)
• Change the name of the directory streamparse to pyheron (#1267)
• Fix flaky SpoutInstanceTest (#1263)
• Heron cli cleanup (#1260)
• Clean up compiling docs (#795)
• Added example topologies in python (#1198)
• Add address argument to heron-ui (#1256)
• Add python integration tests script (#1249)
• Keep argument value as list (#1257)
• Add python integration test topologies (#1248)
• Added python integration test core modules (#1247)
• Modified test_runner for python topology integration tests (#1246)
• Added docs for Python topology support (#1231)
• Fix heron executor log problem (#1252)
• Some fixes to python topologies (#1250)
• Added single-thread Heron Instance in python (#1201)
• Add Spout Instance and Bolt Instance for Python topologies (#1239)
• Update the heron scheduler document (#1241)
• Changed some debugging messages (#1240)
• Python topology submission/execution support (#1238)
• Streamparse compatible Python topology API (#1236)
• Added custom serializer support (#1237)
• When installing heron client, do not modify .heronrc if exists already. (#1235)
• make travis test compilation to use --config=ubuntu (#1234)
• Changed heron.options from system property to environment variable (#1233)
• Allow overriding docker build environment variables (#1227)
• Bump version number in config. (#1229)
• add marathon scheduler for heron (#1161)
• add support for heronrc file (#1119)
• Website content change (#1226)
• Adding support for python topology to executor, spi, scheduler
• Update Python's common/util modules (#1185)
• Import Log object directly.
• Improve PackingPlanTest (#1214)
• Added arg for physical plan helper (#1185)
• Added metricsmgr_cli, stmgr_cli for python topology

Version 0.14.2

Highlights

• Resource compliant round robin algorithm
• First fit decreasing packing algorithm
• SCP package uploader
• Experimental Native Mesos Scheduler support
• Support for Bazel 0.3.0
• Publishing of SPI JAR packages for Heron core development
• Upgrade PEX to 1.1.11
• Eclipse IDE support

Release Notes

• Improving logging for cli, scheduler, tracker and UI
• tar and curl only show more info when verbose.
• Set PEX_ROOT for executor environment
• Dedup and organize launcher
• Properly shutdown heron-tracker and heron-ui
• Modified pex_loader to avoid duplicates in sys.path
• Add reflectasm as a require dependency for kryo
• Native mesos scheduler documentation for running locally
• Make custom grouping more generic
• Common network and utility modules for python topology support
• Added pex_loader.py for python topology support
• Made ComponentObjectSpec optional for backward compatibility
• Modified topology.proto to support for Python topology
• Made Config message in topology.proto generic (
• Pex without interpreter cache disabled
• heron executor refactor and unit tests
• Add double quote to support spaces in envionment variables
• Improve exception handling of heron-explorer
• Correct custom-metrics-sink.md doc.
• Added brief description about the internal architecture
• Move scheduler specific computation outside packing
• Sort containers by id in the UI so they are easier to find
• Add java_tests to make adding java unit tests easier
• Change AuroraController.appendAuroraCommandOptions to private
• Creating main method to make it easy to inspect local state manager contents
• Fix NPE for MetricsManager.DefaultExceptionHandler
• Set correct response type HTTP header for tracker
• Allow passing properties to SchedulerMain
• Add utils method schedulerCommandArgs
• Allow specify env variables when spwaning a new process in java
• Update aws dependency version
• Add extension when generating filename
• Print verbose test failure output when test failed
• Website qs: update export path from install
• Fix HeronMasterDriverTest.createsContextConfigForExecutorId

Version 0.14.1

Highlights

• Experimental support for Heron on YARN using Apache REEF
• Move to bazel 0.2.3 for compiling and packaging Heron
• Export service provider interface for development of custom schedulers/uploaders/packing algorithms/metrics managers
• Several bug fixes and improved documentation

Release Notes

• Move to bazel 0.2.3
• Implement Heron on YARN using Apache REEF
• Detailed documentation on setting up Heron with local Aurora cluster
• Export heron-spi.jar for custom development of packing algorithms/uploaders/schedulers
• Make task index never negative in simulator's FieldsGrouping
• Fix simulator SpoutInstance OutQueueFullCount metric
• Allow display of single node topologies
• Add the command to install libtool for Mac OS X developers
• Add how to release to Maven central repo
• Remove duplicate install code
• Limit the maximum size of heron tuple set
• Addition of docker image for heron distribution and dist build scripts
• Update custom-metrics-sink documentation
• Fixed bugs in RoundRobinPacking algorithm
• Update custom-metrics-sink documentation
• Several documentation enhancements and fixes
• Increase the minimal queue size for Communicator
• Additions to getting-started-troubleshooting.md
• Remove localzk from heron_tracker.yaml
• Add the command to install libtool for Mac OS X developers
• Capture bazel build info into files, print info when return code is not zero for travis ci
• Suppress unzip warnings during install time
• Improve error messaging when role or env is missing

Version 0.14.0

This release focuses on additional documentation, documentation cleanup and cleanups

• Documentation for distributed deployment and schedulers
• Documentation for simulator mode
• Documentation about writing a topology
• Read statemgr connection string as host:port in tracker config
• Fix broken links on codebase
• Enable link checker in travis CI
• Fix yaml reading in utils.py
• Revert adding generics to Storm APIs
• Remove flaky cpp unit test "time_now_unittest"
• Slurm scheduler for HPC community
• Website cleanups

Version 0.13.7

This release contains bug fixes a blocker bug that allows for launching topologies using their tar balls

• Add serialVersionUID for TestWordTopology
• Make cluster nav bar full width
• Add missing values for Config runtime to avoid NPE
• Add TestWordSpout
• Fix import for heron cli
• Travis CI to test that website links are valid
• Mark stmgr cpp tests flaky, fix idea copyright settings