Skip to content
This repository has been archived by the owner on Jun 16, 2023. It is now read-only.

compile bug fix [0.7.1thrift8 ] #383

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

compile bug fix [0.7.1thrift8 ] #383

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
232acf5
remove task heartbeat when one task has been reassign
longdafeng Dec 3, 2013
d7f670d
update release binary
longdafeng Dec 3, 2013
deff7d9
fix sort supervisor sequence error
longdafeng Dec 3, 2013
0717729
update 0.7.1 binary
longdafeng Dec 4, 2013
2cbf48e
dos2unix
Dec 18, 2013
c609784
LICENSE fixed
Jan 24, 2014
bedb4bd
change to libthrift8
jxysoft Oct 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion LICENSE.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
</head>

<body lang="EN-US">
<p>Copyright (c) Nathan Marz. All rights reserved.</p>
<p>Copyright (c) Alibaba. All rights reserved.</p>
<p align=center><b>Eclipse Public License - v 1.0</b></p>

<p>THE ACCOMPANYING PROGRAM IS PROVIDED UNDER THE TERMS OF THIS ECLIPSE
Expand Down
136 changes: 69 additions & 67 deletions changehistory
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,68 +1,70 @@
# Release 0.7.1
In this version, it will follow storm 0.7.1 interface, so the topology running
in storm 0.7.1 can run in jstorm without any change.

Stability
1. add setting "zmq.max.queue.msg" for zeromq
2. communication between worker and tasks without zeromq
3. Add catch exception operation
3.1. in supervisor SyncProcess/SyncSupervisor
3.2. add catch exception and report_error in spout's open and bolt's prepare
3.3. in all IO operation
3.4. in all serialize/deserialize
3.5. in all ZK operation
3.6 in topology upload/download function
3.7 during initialization zeromq
4. do assignmen/reassignment operation in one thread to avoid competition
5. redesign nimbus 's topology assign algorithm, make the logic simple much.
6. redesign supervisor's sync assignment algorithm, make the logic simple much
7. reduce zookeeper load
7.1 redesign nimbus monitor logic, it will just scan tasks' hearbeat, frequency is 10s
7.2 nimbus cancel watch on supervisor
7.3 supervisor heartbeat frequence change to 10s
7.4 supervisor syncSupervisor/syncProcess frequence change to 10s
7.5 supervisor scan /$(ZKROOT)/assignment only once in one monitor loop
7.6 task hearbeat change to 10s
8 create task pid file before connection zk, this is very import when zk is unstable.


Performance tuning
1. reduce once memory copy when deserialize tuple, improve performance huge.
2. split executor thread as two thread, one handing receive tuples, one sending tuples, improve performance much
3. redeisign sample code, it will sampling every 5 seconds, not every 20 tuple once, improve performance much
3. simplify the ack's logic, make acker more effeciency
4. Communication between worker and tasks won't use zeromq, just memory share in process
5. in worker's Drainer/virtualportdispatch thread, spout/bolt recv/send thread,
the thread will sleep 1 ms when there is not tuple in one loop
6. communication between worker and tasks without zeromq
7. sampling frequence change to 5s, not every 20 tuple once.

Enhancement:
1. add IFailValueSpout interface
2. Redesign sampling code, collection statics model become more common.
Add sending/recving tps statics, statics is more precise.
3. Atomatically do deactivate action when kill/rebalance topology,
and the wait time is 2 * MSG_TIMEOUT
4. fix nongrouping bug, random.nextInt will generate value less than 0.
5. Sleep one setting time(default is 1 minute) after finish spout open,
which is used to wait other task finish initialization.
6. Add check component name when submit topology, forbidding the component
which name start with "__"
7. change the zk's node /$(ZKROOT)/storm to /$(ZKROOT)/topology
8. abstract topology check logic from generating real topology function
9. when supervisor is down and topology do rebalance, the alive task under down
supervisor is unavailable.
10. add close connection operation after finish download topology binary
11. automatically create all local dirtorie, such as
/$(LOCALDIR)/supervisor/localstate
12. when killing worker, add "kill and sleep " operation before "kill -9" operation
13. when generate real topology binary,
13.1. configuration priority different.
component configuration > topology configuration > system configuration
13.2. skip the output stream which target component doesn't exist.
13.3. skip the component whose parallism is 0.
13.4. component's parallism is less than 0, throw exception.
14. skip ack/fail when inputstream setting is empty
15. add topology name to the log
16. fix ui select option error, default is 10 minutes
# Release 0.7.1
In this version, it will follow storm 0.7.1 interface, so the topology running
in storm 0.7.1 can run in jstorm without any change.

Stability
1. add setting "zmq.max.queue.msg" for zeromq
2. communication between worker and tasks without zeromq
3. Add catch exception operation
3.1. in supervisor SyncProcess/SyncSupervisor
3.2. add catch exception and report_error in spout's open and bolt's prepare
3.3. in all IO operation
3.4. in all serialize/deserialize
3.5. in all ZK operation
3.6 in topology upload/download function
3.7 during initialization zeromq
4. do assignmen/reassignment operation in one thread to avoid competition
5. redesign nimbus 's topology assign algorithm, make the logic simple much.
6. redesign supervisor's sync assignment algorithm, make the logic simple much
7. reduce zookeeper load
7.1 redesign nimbus monitor logic, it will just scan tasks' hearbeat, frequency is 10s
7.2 nimbus cancel watch on supervisor
7.3 supervisor heartbeat frequence change to 10s
7.4 supervisor syncSupervisor/syncProcess frequence change to 10s
7.5 supervisor scan /$(ZKROOT)/assignment only once in one monitor loop
7.6 task hearbeat change to 10s
8 create task pid file before connection zk, this is very import when zk is unstable.
9.equally assign worker to supervisors
10. reset hearbeat timeout for one reassign task


Performance tuning
1. reduce once memory copy when deserialize tuple, improve performance huge.
2. split executor thread as two thread, one handing receive tuples, one sending tuples, improve performance much
3. redeisign sample code, it will sampling every 5 seconds, not every 20 tuple once, improve performance much
3. simplify the ack's logic, make acker more effeciency
4. Communication between worker and tasks won't use zeromq, just memory share in process
5. in worker's Drainer/virtualportdispatch thread, spout/bolt recv/send thread,
the thread will sleep 1 ms when there is not tuple in one loop
6. communication between worker and tasks without zeromq
7. sampling frequence change to 5s, not every 20 tuple once.

Enhancement:
1. add IFailValueSpout interface
2. Redesign sampling code, collection statics model become more common.
Add sending/recving tps statics, statics is more precise.
3. Atomatically do deactivate action when kill/rebalance topology,
and the wait time is 2 * MSG_TIMEOUT
4. fix nongrouping bug, random.nextInt will generate value less than 0.
5. Sleep one setting time(default is 1 minute) after finish spout open,
which is used to wait other task finish initialization.
6. Add check component name when submit topology, forbidding the component
which name start with "__"
7. change the zk's node /$(ZKROOT)/storm to /$(ZKROOT)/topology
8. abstract topology check logic from generating real topology function
9. when supervisor is down and topology do rebalance, the alive task under down
supervisor is unavailable.
10. add close connection operation after finish download topology binary
11. automatically create all local dirtorie, such as
/$(LOCALDIR)/supervisor/localstate
12. when killing worker, add "kill and sleep " operation before "kill -9" operation
13. when generate real topology binary,
13.1. configuration priority different.
component configuration > topology configuration > system configuration
13.2. skip the output stream which target component doesn't exist.
13.3. skip the component whose parallism is 0.
13.4. component's parallism is less than 0, throw exception.
14. skip ack/fail when inputstream setting is empty
15. add topology name to the log
16. fix ui select option error, default is 10 minutes
17. supervisor can display all worker's status
12 changes: 6 additions & 6 deletions deploy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,17 +13,17 @@ this will build one tar

Deploy

1. Deployment for nimbus or supervisor is same as storm. so please install jzmq and zmq.
1. Deployment for nimbus or supervisor is same as storm. so please install jzmq and zmq firstly.
2. Deployment for client,
2.1 please copy configuration to ~/.jstorm/storm.yaml
2.2 touch $JSTORM_HOME/RELEASE
2.1 cp $JSTORM_HOME/conf/storm.yaml ~/.jstorm/storm.yaml
2.2 cd $JSTORM_HOME && touch $JSTORM_HOME/RELEASE
3. Deployment for web ui.
3.1 copy configuration to ~/.jstorm/storm.yaml
3.1 cp $JSTORM_HOME/conf/storm.yaml ~/.jstorm/storm.yaml
3.2 downlaod tomcat 7.x
3.3 Extrade tar of tomcat 7.x
3.3 Extract tar of tomcat 7.x
3.4 put jstorm-ui-0.7.1.war to Tomcat's webapps directory

open one webbrowser and go to $jstorm-ui-ip:8080/jstorm-ui-0.7.1
open one webbrowser and go to http://$jstorm-ui-ip:8080/jstorm-ui-0.7.1

3.5 you also can change the default page as jstorm-ui, please google how to do it.

Expand Down
Loading