Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

细节性能优化2 #80

Open
muyannian opened this issue Jul 25, 2014 · 2 comments
Open

细节性能优化2 #80

muyannian opened this issue Jul 25, 2014 · 2 comments
Labels
enhancement

Comments

@muyannian
Copy link
Contributor

2014-07-25 11:53:11 SolrCore [INFO] getSearcher:r_rpt_cps_luna_item@2014052@1406212981800:/disk1/taobao/mdrill/higo/adhoc/shard_0_48/tablelist/r_rpt_cps_luna_item/solr/data/2014052
2014-07-25 11:53:11 ReadOnlyDirectory [INFO] ####/disk1/taobao/mdrill/higo/adhoc/shard_0_48/tablelist/r_rpt_cps_luna_item/solr/data/2014052,/group/tbdp-etao-adhoc/p4padhoc/tablelist/r_rpt_cps_luna_item/index/2014052/part-00048
2014-07-25 11:53:11 SolrCore [INFO] >>>>>LinkFSDirectory readOnlyOpen add links /disk7/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014052
2014-07-25 11:53:11 ReadOnlyDirectory [INFO] ####remakeSearch readonly ####[/disk7/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014052]
2014-07-25 11:53:13 SolrIndexSearcher [INFO] Opening Searcher@43b5e597 partion_r_rpt_cps_luna_item@2014052@1406212981800
2014-07-25 11:53:15 SolrCore [INFO] SolrIndexSearcher clear:rpt_p4padhoc_product@201407@1406201495517,17@16
2014-07-25 11:53:15 FSEditLog [INFO] Number of transactions: 131 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 1 Number of syncs: 131
2014-07-25 11:53:15 LogUpdateProcessor [INFO] {add=[(null), (null), (null), (null), (null), (null), (null)]} 0 35
2014-07-25 11:53:16 RealTimeDirectoryData [INFO] flushDocList start
2014-07-25 11:53:16 RealTimeDirectoryData [INFO] ####create buffer####
2014-07-25 11:53:16 RealTimeDirectoryData [INFO] flushDocList end
2014-07-25 11:53:16 RealTimeDirectoryThread [INFO] flushDocList timetake:0
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@830e918a
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@86bd1973
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@6bd62922
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@5b0968d4
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@56629b06
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@240b3460
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@924440f
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@25525a35
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryResultKeyCache@a6b4be4
2014-07-25 11:53:18 GrobalCache [INFO] field value cache FQ lru removed key org.apache.solr.search.SolrIndexSearcher$QueryKey@240b3466
2014-07-25 11:53:18 GrobalCache [INFO] ####fieldvaluecache FQ####179@184mb,size=2855,mem=3mb,key org.apache.solr.search.SolrIndexSearcher$QueryKey@199c7cea
2014-07-25 11:53:18 UnInvertedField [INFO] ajustBase 16,baseAdvanceDocs=154@7392202@29568809,org.apache.lucene.index.ReadOnlySegmentReader
2014-07-25 11:53:18 MakeUnivertedFieldBySigment [INFO] makeInit start:settle_uv ,this.baseAdvanceDocs=154
2014-07-25 11:53:18 DocValuesReader [INFO] 80>DocValuesReader [tispos=2080905642, tisposTxt=4134833, tisposVal=354401600, maxsize=29568808, maxtm=2, useBits=2, lst=1, sizeOfStrings=22]
2014-07-25 11:53:18 RamDocValueFill [INFO] file timetaken:0,13,97
2014-07-25 11:53:18 MakeUnivertedFieldBySigment [INFO] ####makeInit end####:settle_uv,3@110@1@0@2
2014-07-25 11:53:18 GrobalCache [INFO] ####fieldvaluecache####1073@1105mb,size=60,mem=31mb,key seg@true@settle_uv@29568809@[email protected]/disk7/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014052_32_2822444266_8962955101_1404023044000_20140629142404@_b
2014-07-25 11:53:18 UnInvertedField [INFO] ####timetaken####:118@2@0,{field=settle_uv,memSize=31.75551414489746mb,nullTermNum=3,indexbyte=29569024@29568811,,termValueLong=4096@4,bits.size=154}
2014-07-25 11:53:18 UnInvertedField [INFO] ####timetaken all####:120,{field=settle_uv,memSize=31.75551414489746mb,nullTermNum=3,indexbyte=29569024@29568811,,termValueLong=4096@4,bits.size=154}
2014-07-25 11:53:18 UnInvertedField [INFO] ajustBase 16,baseAdvanceDocs=154@7392202@29568809,org.apache.lucene.index.ReadOnlySegmentReader
2014-07-25 11:53:18 MakeUnivertedFieldBySigment [INFO] makeInit start:item_coll_num ,this.baseAdvanceDocs=154
2014-07-25 11:53:18 DocValuesReader [INFO] 70>DocValuesReader [tispos=1836962868, tisposTxt=4131708, tisposVal=354138760, maxsize=29568808, maxtm=30, useBits=5, lst=1, sizeOfStrings=22]
2014-07-25 11:53:18 RamDocValueFill [INFO] file timetaken:0,9,196
2014-07-25 11:53:18 MakeUnivertedFieldBySigment [INFO] ####makeInit end####:item_coll_num,3@205@2@0@2
2014-07-25 11:53:18 GrobalCache [INFO] field value cache lru removed key seg@true@shop_ipv@25618803@[email protected]/disk5/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014072_32_4117678148_7561825158_1406156495000_20140724070135@_b
2014-07-25 11:53:18 GrobalCache [INFO] field value cache lru removed key seg@true@shop_ipv@10753130@[email protected]/disk4/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014073_32_130129740_3375991876_1406234743000_20140725044543@b
2014-07-25 11:53:18 GrobalCache [INFO] ####fieldvaluecache####1066@1105mb,size=59,mem=31mb,key seg@true@item_coll_num@29568809@[email protected]/disk7/taobao/mdrill/higo/r_rpt_cps_luna_item/shard_0_48/2014052_32_2822444266_8962955101_1404023044000_20140629142404@b
2014-07-25 11:53:18 UnInvertedField [INFO] ####timetaken####:214@2@0,{field=item_coll_num,memSize=31.75551414489746mb,nullTermNum=31,indexbyte=29569024@29568811,,termValueLong=4096@32,bits.size=154}
2014-07-25 11:53:18 UnInvertedField [INFO] ####timetaken all####:216,{field=item_coll_num,memSize=31.75551414489746mb,nullTermNum=31,indexbyte=29569024@29568811,,termValueLong=4096@32,bits.size=154}
2014-07-25 11:53:18 ShardGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-25 11:53:18 ShardGroupByTermNumCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-25 11:53:18 MdrillParseGroupby [INFO] ##baseDocs.size## 154@154
2014-07-25 11:53:18 MergerGroupByGroupbyRowCompare [INFO] ####SortType [sortFieldNum=-1, typeNum=0, typeEnum=index]
2014-07-25 11:53:18 MdrillUtils [INFO] fieldValueCache.size=528,size=1
2014-07-25 11:53:18 MdrillGroupBy [INFO] ##FacetCross## time taken ,total:410,init:408,makeGroups:1,transGroupValue:0,groups.size:1
2014-07-25 11:53:18 FacetComponent [INFO] ####task####411,0
2014-07-25 11:53:18 SolrCore [INFO] [r_rpt_cps_luna_item] status=0 QTime=7439 webapp=/solr path=/select params={{echoParams=explicit,rows=0,facet.cross.join=@,facet.cross=true,mdrill.crc.key.set=a8a7258c-7cbf-4aa7-83d3-b6dead606eb5,facet=true,facet.cross.offset=0,facet.cross.sort.fl=higoempty_sort_s,indexpartion=2014052,version=2,facet.cross.sort.desc=true,showlog=1,facet.cross.limit=10010,facet.field=higoempty_groupby_l,fq=[Ljava.lang.String;@192dede6,facet.cross.sort.cp=string,higo_ms_depth=2,wt=javabin,rows=0,mlogtime=20140725115311,start=0,facet.sort=index,q=:,facet.cross.sort.tp=index,maxshards=64,facet.cross.fl=[Ljava.lang.String;@165da3f4,mdrill.isRestat=false,mergeservers=10.246.14.55:51270/solr/r_rpt_cps_luna_item,172.24.195.155:51271/solr/r_rpt_cps_luna_item,10.246.45.21:51272/solr/r_rpt_cps_luna_item,10.246.45.22:51273/solr/r_rpt_cps_luna_item,10.246.45.23:51274/solr/r_rpt_cps_luna_item,10.246.45.24:51275/solr/r_rpt_cps_luna_item,10.246.45.41:51276/solr/r_rpt_cps_luna_item,10.246.45.42:51277/solr/r_rpt_cps_luna_item,10.246.45.43:51278/solr/r_rpt_cps_luna_item,10.246.45.44:51279/solr/r_rpt_cps_luna_item,172.24.195.154:51280/solr/r_rpt_cps_luna_item,,isShard=true}}
2014-07-25 11:53:19 LogUpdateProcessor [INFO] {add=[(null), (null), (null), (null), (null), (null), (null), (null), ... (39 adds)]} 0 2
2014-07-25 11:53:22 RealTimeDirectoryData [INFO] flushDocList start
2014-07-25 11:53:22 RealTimeDirectoryData [INFO] ####create buffer####
2014-07-25 11:53:22 RealTimeDirectoryData [INFO] flushDocList end
2014-07-25 11:53:22 RealTimeDirectoryThread [INFO] flushDocList timetake:0
@muyannian
Copy link
Contributor Author

仅仅查询一个 154 个docid的查询 ，竟然用了7秒

@muyannian
Copy link
Contributor Author

从这段日子中可以看出，时间消耗并非在fieldvaluecache上，而是在于fq的查询上

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@muyannian