script.out

  0%|          | 0/381793 [00:00<?, ?it/s]
  3%|▎         | 11019/381793 [00:00<00:03, 110188.53it/s]
  6%|▌         | 22201/381793 [00:00<00:03, 110670.79it/s]
  9%|▉         | 33728/381793 [00:00<00:03, 112010.64it/s]
 12%|█▏        | 45539/381793 [00:00<00:02, 113771.41it/s]
 15%|█▌        | 57639/381793 [00:00<00:02, 115846.81it/s]
 18%|█▊        | 69741/381793 [00:00<00:02, 117349.64it/s]
 21%|██▏       | 81947/381793 [00:00<00:02, 118722.01it/s]
 25%|██▍       | 93878/381793 [00:00<00:02, 118895.04it/s]
 28%|██▊       | 106236/381793 [00:00<00:02, 120258.23it/s]
 31%|███       | 118218/381793 [00:01<00:02, 120124.91it/s]
 34%|███▍      | 130761/381793 [00:01<00:02, 121665.45it/s]
 38%|███▊      | 143199/381793 [00:01<00:01, 122464.05it/s]
 41%|████      | 155600/381793 [00:01<00:01, 122922.50it/s]
 44%|████▍     | 167968/381793 [00:01<00:01, 123145.48it/s]
 47%|████▋     | 180517/381793 [00:01<00:01, 123837.35it/s]
 51%|█████     | 192853/381793 [00:01<00:01, 122004.25it/s]
 54%|█████▍    | 205347/381793 [00:01<00:01, 122868.69it/s]
 57%|█████▋    | 218052/381793 [00:01<00:01, 124091.68it/s]
 60%|██████    | 230452/381793 [00:01<00:01, 122861.76it/s]
 64%|██████▎   | 243080/381793 [00:02<00:01, 123867.07it/s]
 67%|██████▋   | 255466/381793 [00:02<00:01, 123000.84it/s]
 70%|███████   | 267982/381793 [00:02<00:00, 123639.80it/s]
 73%|███████▎  | 280348/381793 [00:02<00:00, 123408.12it/s]
 77%|███████▋  | 292999/381793 [00:02<00:00, 124319.92it/s]
 80%|████████  | 305538/381793 [00:02<00:00, 124638.17it/s]
 83%|████████▎ | 318004/381793 [00:02<00:00, 123736.12it/s]
 87%|████████▋ | 330664/381793 [00:02<00:00, 124579.87it/s]
 90%|████████▉ | 343393/381793 [00:02<00:00, 125380.48it/s]
 93%|█████████▎| 355935/381793 [00:02<00:00, 124691.43it/s]
 96%|█████████▋| 368408/381793 [00:03<00:00, 119480.97it/s]
100%|█████████▉| 380934/381793 [00:03<00:00, 121156.85it/s]
100%|██████████| 381793/381793 [00:03<00:00, 121704.86it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6232/150000 [00:00<00:02, 62318.13it/s]
  8%|▊         | 12576/150000 [00:00<00:02, 62647.82it/s]
 13%|█▎        | 19147/150000 [00:00<00:02, 63534.82it/s]
 17%|█▋        | 25671/150000 [00:00<00:01, 64036.11it/s]
 21%|██▏       | 32128/150000 [00:00<00:01, 64194.91it/s]
 26%|██▌       | 38753/150000 [00:00<00:01, 64797.08it/s]
 30%|███       | 45498/150000 [00:00<00:01, 65568.61it/s]
 35%|███▍      | 52253/150000 [00:00<00:01, 66150.36it/s]
 39%|███▉      | 59088/150000 [00:00<00:01, 66794.09it/s]
 44%|████▍     | 65925/150000 [00:01<00:01, 67257.41it/s]
 49%|████▊     | 72834/150000 [00:01<00:01, 67795.20it/s]
 53%|█████▎    | 79575/150000 [00:01<00:01, 67676.48it/s]
 58%|█████▊    | 86389/150000 [00:01<00:00, 67812.48it/s]
 62%|██████▏   | 93157/150000 [00:01<00:00, 67771.10it/s]
 67%|██████▋   | 99936/150000 [00:01<00:00, 67775.38it/s]
 71%|███████   | 106698/150000 [00:01<00:00, 67726.87it/s]
 76%|███████▌  | 113522/150000 [00:01<00:00, 67878.08it/s]
 80%|████████  | 120296/150000 [00:01<00:00, 67554.33it/s]
 85%|████████▍ | 127161/150000 [00:01<00:00, 67878.16it/s]
 89%|████████▉ | 133943/150000 [00:02<00:00, 67452.41it/s]
 94%|█████████▍| 140685/150000 [00:02<00:00, 67429.24it/s]
 98%|█████████▊| 147426/150000 [00:02<00:00, 67195.39it/s]
100%|██████████| 150000/150000 [00:02<00:00, 66862.56it/s]

  0%|          | 0/167 [00:00<?, ?it/s]loading database updates...
loading word vectors...

WARNING-: Scrapper will use  30 parallel workers and batch size 90 

Ready to perform all updates from 2020-05-18 to 2020-05-28 

Updating Database Using Cisco Ranklist Of 2020-05-18 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 14979 new domains found

Scrapping urls...
scrapping will be done in 167 phase  where 90 urls attempt in each phase

  1%|          | 1/167 [00:32<1:28:33, 32.01s/it]
  1%|          | 2/167 [01:05<1:28:57, 32.35s/it]
  2%|▏         | 3/167 [01:41<1:31:35, 33.51s/it]
  2%|▏         | 4/167 [01:51<1:12:21, 26.63s/it]
  3%|▎         | 5/167 [02:09<1:04:40, 23.95s/it]
  4%|▎         | 6/167 [02:23<56:00, 20.88s/it]  
  4%|▍         | 7/167 [02:56<1:05:25, 24.53s/it]
  5%|▍         | 8/167 [03:35<1:16:48, 28.98s/it]
  5%|▌         | 9/167 [04:07<1:18:29, 29.81s/it]
  6%|▌         | 10/167 [04:31<1:13:32, 28.10s/it]
  7%|▋         | 11/167 [05:07<1:19:17, 30.50s/it]
  7%|▋         | 12/167 [05:30<1:13:09, 28.32s/it]
  8%|▊         | 13/167 [06:14<1:24:11, 32.80s/it]
  8%|▊         | 14/167 [06:54<1:29:25, 35.07s/it]
  9%|▉         | 15/167 [07:33<1:31:33, 36.14s/it]
 10%|▉         | 16/167 [08:13<1:33:46, 37.26s/it]
 10%|█         | 17/167 [08:59<1:39:56, 39.98s/it]
 11%|█         | 18/167 [09:38<1:38:16, 39.58s/it]
 11%|█▏        | 19/167 [10:13<1:34:35, 38.35s/it]
 12%|█▏        | 20/167 [11:02<1:41:30, 41.43s/it]
 13%|█▎        | 21/167 [11:41<1:39:20, 40.83s/it]
 13%|█▎        | 22/167 [12:32<1:45:47, 43.78s/it]
 14%|█▍        | 23/167 [13:08<1:39:31, 41.47s/it]
 14%|█▍        | 24/167 [13:49<1:38:23, 41.28s/it]
 15%|█▍        | 25/167 [14:33<1:39:49, 42.18s/it]
 16%|█▌        | 26/167 [15:04<1:30:56, 38.70s/it]
 16%|█▌        | 27/167 [15:49<1:34:55, 40.69s/it]
 17%|█▋        | 28/167 [16:34<1:37:05, 41.91s/it]
 17%|█▋        | 29/167 [17:18<1:38:08, 42.67s/it]
 18%|█▊        | 30/167 [17:51<1:30:44, 39.74s/it]
 19%|█▊        | 31/167 [18:31<1:30:29, 39.92s/it]
 19%|█▉        | 32/167 [19:13<1:30:48, 40.36s/it]
 20%|█▉        | 33/167 [19:55<1:31:28, 40.96s/it]
 20%|██        | 34/167 [20:45<1:37:05, 43.80s/it]
 21%|██        | 35/167 [21:20<1:30:00, 40.91s/it]
 22%|██▏       | 36/167 [22:01<1:29:44, 41.10s/it]
 22%|██▏       | 37/167 [22:20<1:14:48, 34.53s/it]
 23%|██▎       | 38/167 [22:57<1:15:46, 35.25s/it]
 23%|██▎       | 39/167 [23:43<1:22:08, 38.50s/it]
 24%|██▍       | 40/167 [24:26<1:23:51, 39.62s/it]
 25%|██▍       | 41/167 [25:08<1:24:47, 40.38s/it]
 25%|██▌       | 42/167 [25:48<1:24:18, 40.47s/it]
 26%|██▌       | 43/167 [26:24<1:20:40, 39.03s/it]
 26%|██▋       | 44/167 [27:00<1:18:09, 38.12s/it]
 27%|██▋       | 45/167 [27:43<1:20:42, 39.69s/it]
 28%|██▊       | 46/167 [28:17<1:16:01, 37.70s/it]
 28%|██▊       | 47/167 [29:10<1:24:41, 42.34s/it]
 29%|██▊       | 48/167 [29:48<1:21:35, 41.14s/it]
 29%|██▉       | 49/167 [30:11<1:10:07, 35.66s/it]
 30%|██▉       | 50/167 [30:47<1:09:31, 35.66s/it]
 31%|███       | 51/167 [31:16<1:05:20, 33.80s/it]
 31%|███       | 52/167 [31:54<1:07:13, 35.07s/it]
 32%|███▏      | 53/167 [32:30<1:07:06, 35.32s/it]
 32%|███▏      | 54/167 [32:47<56:13, 29.85s/it]  
 33%|███▎      | 55/167 [33:23<58:58, 31.59s/it]
 34%|███▎      | 56/167 [33:58<1:00:25, 32.66s/it]
 34%|███▍      | 57/167 [34:50<1:10:50, 38.64s/it]
 35%|███▍      | 58/167 [35:22<1:06:14, 36.46s/it]
 35%|███▌      | 59/167 [36:07<1:10:25, 39.13s/it]
 36%|███▌      | 60/167 [36:43<1:08:09, 38.22s/it]
 37%|███▋      | 61/167 [37:06<59:13, 33.53s/it]  
 37%|███▋      | 62/167 [37:47<1:02:27, 35.69s/it]
 38%|███▊      | 63/167 [38:25<1:03:08, 36.42s/it]
 38%|███▊      | 64/167 [39:01<1:02:25, 36.37s/it]
 39%|███▉      | 65/167 [39:21<53:27, 31.45s/it]  
 40%|███▉      | 66/167 [40:06<59:35, 35.40s/it]
 40%|████      | 67/167 [40:41<59:10, 35.50s/it]
 41%|████      | 68/167 [41:19<59:28, 36.05s/it]
 41%|████▏     | 69/167 [41:55<58:57, 36.10s/it]
 42%|████▏     | 70/167 [42:36<1:00:48, 37.62s/it]
 43%|████▎     | 71/167 [43:12<59:28, 37.18s/it]  
 43%|████▎     | 72/167 [43:40<54:13, 34.25s/it]
 44%|████▎     | 73/167 [44:12<52:46, 33.69s/it]
 44%|████▍     | 74/167 [44:43<51:06, 32.97s/it]
 45%|████▍     | 75/167 [45:10<47:29, 30.97s/it]
 46%|████▌     | 76/167 [45:42<47:33, 31.36s/it]
 46%|████▌     | 77/167 [46:08<44:47, 29.86s/it]
 47%|████▋     | 78/167 [46:26<39:08, 26.39s/it]
 47%|████▋     | 79/167 [47:03<43:08, 29.41s/it]
 48%|████▊     | 80/167 [47:22<38:08, 26.31s/it]
 49%|████▊     | 81/167 [47:54<40:12, 28.05s/it]
 49%|████▉     | 82/167 [48:17<37:28, 26.45s/it]
 50%|████▉     | 83/167 [48:41<36:02, 25.74s/it]
 50%|█████     | 84/167 [49:17<39:53, 28.83s/it]
 51%|█████     | 85/167 [49:57<44:11, 32.34s/it]
 51%|█████▏    | 86/167 [50:23<40:47, 30.22s/it]
 52%|█████▏    | 87/167 [50:55<41:07, 30.84s/it]
 53%|█████▎    | 88/167 [51:31<42:29, 32.27s/it]
 53%|█████▎    | 89/167 [51:53<38:06, 29.32s/it]
 54%|█████▍    | 90/167 [52:29<40:11, 31.31s/it]
 54%|█████▍    | 91/167 [53:09<42:46, 33.78s/it]
 55%|█████▌    | 92/167 [53:42<42:01, 33.62s/it]
 56%|█████▌    | 93/167 [54:02<36:23, 29.51s/it]
 56%|█████▋    | 94/167 [54:27<34:21, 28.24s/it]
 57%|█████▋    | 95/167 [54:50<31:53, 26.58s/it]
 57%|█████▋    | 96/167 [55:18<31:56, 26.99s/it]
 58%|█████▊    | 97/167 [55:54<34:45, 29.79s/it]
 59%|█████▊    | 98/167 [56:21<33:07, 28.80s/it]
 59%|█████▉    | 99/167 [56:50<32:57, 29.08s/it]
 60%|█████▉    | 100/167 [57:22<33:27, 29.96s/it]
 60%|██████    | 101/167 [58:05<37:17, 33.90s/it]
 61%|██████    | 102/167 [58:27<32:35, 30.09s/it]
 62%|██████▏   | 103/167 [59:07<35:25, 33.20s/it]
 62%|██████▏   | 104/167 [59:34<32:56, 31.38s/it]
 63%|██████▎   | 105/167 [1:00:10<33:58, 32.87s/it]
 63%|██████▎   | 106/167 [1:00:43<33:14, 32.70s/it]
 64%|██████▍   | 107/167 [1:01:18<33:29, 33.48s/it]
 65%|██████▍   | 108/167 [1:01:58<34:57, 35.55s/it]
 65%|██████▌   | 109/167 [1:02:19<30:01, 31.06s/it]
 66%|██████▌   | 110/167 [1:02:46<28:27, 29.95s/it]
 66%|██████▋   | 111/167 [1:03:21<29:18, 31.41s/it]
 67%|██████▋   | 112/167 [1:04:02<31:29, 34.36s/it]
 68%|██████▊   | 113/167 [1:04:41<31:57, 35.51s/it]
 68%|██████▊   | 114/167 [1:05:04<28:04, 31.79s/it]
 69%|██████▉   | 115/167 [1:05:22<23:53, 27.57s/it]
 69%|██████▉   | 116/167 [1:05:58<25:44, 30.29s/it]
 70%|███████   | 117/167 [1:06:34<26:45, 32.10s/it]
 71%|███████   | 118/167 [1:06:52<22:39, 27.75s/it]
 71%|███████▏  | 119/167 [1:07:43<27:39, 34.56s/it]
 72%|███████▏  | 120/167 [1:08:19<27:26, 35.03s/it]
 72%|███████▏  | 121/167 [1:08:39<23:31, 30.69s/it]
 73%|███████▎  | 122/167 [1:09:08<22:32, 30.05s/it]
 74%|███████▎  | 123/167 [1:09:46<23:47, 32.43s/it]
 74%|███████▍  | 124/167 [1:10:06<20:43, 28.92s/it]
 75%|███████▍  | 125/167 [1:10:27<18:33, 26.52s/it]
 75%|███████▌  | 126/167 [1:11:00<19:26, 28.44s/it]
 76%|███████▌  | 127/167 [1:11:22<17:33, 26.33s/it]
 77%|███████▋  | 128/167 [1:11:41<15:47, 24.29s/it]
 77%|███████▋  | 129/167 [1:11:57<13:45, 21.71s/it]
 78%|███████▊  | 130/167 [1:12:17<13:09, 21.34s/it]
 78%|███████▊  | 131/167 [1:12:45<13:55, 23.22s/it]
 79%|███████▉  | 132/167 [1:13:13<14:26, 24.74s/it]
 80%|███████▉  | 133/167 [1:13:45<15:12, 26.85s/it]
 80%|████████  | 134/167 [1:14:05<13:33, 24.66s/it]
 81%|████████  | 135/167 [1:14:33<13:42, 25.71s/it]
 81%|████████▏ | 136/167 [1:14:58<13:08, 25.44s/it]
 82%|████████▏ | 137/167 [1:15:19<12:10, 24.37s/it]
 83%|████████▎ | 138/167 [1:15:45<11:52, 24.57s/it]
 83%|████████▎ | 139/167 [1:16:04<10:47, 23.14s/it]
 84%|████████▍ | 140/167 [1:16:24<09:58, 22.15s/it]
 84%|████████▍ | 141/167 [1:16:50<10:00, 23.11s/it]
 85%|████████▌ | 142/167 [1:17:11<09:21, 22.48s/it]
 86%|████████▌ | 143/167 [1:17:46<10:33, 26.38s/it]
 86%|████████▌ | 144/167 [1:18:09<09:42, 25.34s/it]
 87%|████████▋ | 145/167 [1:18:28<08:33, 23.33s/it]
 87%|████████▋ | 146/167 [1:18:44<07:27, 21.29s/it]
 88%|████████▊ | 147/167 [1:19:06<07:08, 21.44s/it]
 89%|████████▊ | 148/167 [1:19:41<08:07, 25.66s/it]
 89%|████████▉ | 149/167 [1:20:17<08:36, 28.69s/it]
 90%|████████▉ | 150/167 [1:20:46<08:08, 28.73s/it]
 90%|█████████ | 151/167 [1:21:07<07:00, 26.30s/it]
 91%|█████████ | 152/167 [1:21:24<05:53, 23.54s/it]
 92%|█████████▏| 153/167 [1:21:48<05:34, 23.89s/it]
 92%|█████████▏| 154/167 [1:22:07<04:48, 22.22s/it]
 93%|█████████▎| 155/167 [1:22:45<05:23, 26.98s/it]
 93%|█████████▎| 156/167 [1:23:21<05:26, 29.69s/it]
 94%|█████████▍| 157/167 [1:23:59<05:22, 32.23s/it]
 95%|█████████▍| 158/167 [1:24:25<04:32, 30.27s/it]
 95%|█████████▌| 159/167 [1:24:59<04:10, 31.37s/it]
 96%|█████████▌| 160/167 [1:25:21<03:20, 28.66s/it]
 96%|█████████▋| 161/167 [1:25:51<02:53, 28.94s/it]
 97%|█████████▋| 162/167 [1:26:19<02:24, 28.88s/it]
 98%|█████████▊| 163/167 [1:26:47<01:53, 28.40s/it]
 98%|█████████▊| 164/167 [1:27:23<01:32, 30.76s/it]
 99%|█████████▉| 165/167 [1:27:52<01:00, 30.36s/it]
 99%|█████████▉| 166/167 [1:28:27<00:31, 31.57s/it]
100%|██████████| 167/167 [1:28:44<00:00, 27.27s/it]
100%|██████████| 167/167 [1:28:44<00:00, 31.88s/it]

  0%|          | 0/231690 [00:00<?, ?it/s]
  5%|▍         | 10825/231690 [00:00<00:02, 108242.88it/s]
  9%|▉         | 21918/231690 [00:00<00:01, 109034.36it/s]
 14%|█▍        | 33498/231690 [00:00<00:01, 110977.08it/s]
 19%|█▉        | 45088/231690 [00:00<00:01, 112406.86it/s]
 25%|██▍       | 57052/231690 [00:00<00:01, 114482.67it/s]
 30%|██▉       | 69068/231690 [00:00<00:01, 116125.87it/s]
 35%|███▍      | 80905/231690 [00:00<00:01, 116788.51it/s]
 40%|███▉      | 92666/231690 [00:00<00:01, 117032.03it/s]
 45%|████▌     | 104869/231690 [00:00<00:01, 118485.71it/s]
 51%|█████     | 117173/231690 [00:01<00:00, 119814.98it/s]
 56%|█████▌    | 129528/231690 [00:01<00:00, 120909.65it/s]
 61%|██████    | 141859/231690 [00:01<00:00, 121618.20it/s]
 67%|██████▋   | 154141/231690 [00:01<00:00, 121974.48it/s]
 72%|███████▏  | 166240/231690 [00:01<00:00, 119591.14it/s]
 77%|███████▋  | 178141/231690 [00:01<00:00, 117287.82it/s]
 82%|████████▏ | 190324/231690 [00:01<00:00, 118611.40it/s]
 87%|████████▋ | 202720/231690 [00:01<00:00, 120165.92it/s]
 93%|█████████▎| 215075/231690 [00:01<00:00, 121159.51it/s]
 98%|█████████▊| 227345/231690 [00:01<00:00, 121615.98it/s]
100%|██████████| 231690/231690 [00:01<00:00, 118966.96it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 10909/150000 [00:00<00:01, 109085.16it/s]
 15%|█▍        | 22209/150000 [00:00<00:01, 110228.09it/s]
 23%|██▎       | 33834/150000 [00:00<00:01, 111966.84it/s]
 31%|███       | 45780/150000 [00:00<00:00, 114112.51it/s]
 39%|███▊      | 57950/150000 [00:00<00:00, 116285.09it/s]
 47%|████▋     | 70369/150000 [00:00<00:00, 118547.42it/s]
 55%|█████▌    | 82674/150000 [00:00<00:00, 119863.08it/s]
 63%|██████▎   | 94954/150000 [00:00<00:00, 120727.63it/s]
 71%|███████▏  | 107223/150000 [00:00<00:00, 121307.44it/s]
 80%|███████▉  | 119529/150000 [00:01<00:00, 121827.37it/s]
 88%|████████▊ | 131970/150000 [00:01<00:00, 122589.05it/s]
 96%|█████████▌| 144373/150000 [00:01<00:00, 123017.32it/s]
100%|██████████| 150000/150000 [00:01<00:00, 120403.22it/s]

0it [00:00, ?it/s]
84it [00:00, 835.24it/s]
166it [00:00, 828.85it/s]
264it [00:00, 866.24it/s]
363it [00:00, 898.22it/s]
444it [00:00, 869.07it/s]
561it [00:00, 940.53it/s]
661it [00:00, 953.14it/s]
778it [00:00, 1007.33it/s]
926it [00:00, 1113.08it/s]
1067it [00:01, 1175.06it/s]
1187it [00:01, 1080.85it/s]
1298it [00:01, 1002.85it/s]
1471it [00:01, 1147.38it/s]
1674it [00:01, 1319.43it/s]
1838it [00:01, 1401.37it/s]
2049it [00:01, 1556.55it/s]
2246it [00:01, 1660.93it/s]
2549it [00:01, 1909.44it/s]
2867it [00:01, 2159.28it/s]
3238it [00:02, 2466.28it/s]
3670it [00:02, 2830.64it/s]
4233it [00:02, 3325.66it/s]
4931it [00:02, 3943.01it/s]
5600it [00:02, 4488.87it/s]
5623it [00:02, 2264.72it/s]

  0%|          | 0/6412 [00:00<?, ?it/s]
 27%|██▋       | 1741/6412 [00:00<00:00, 17166.82it/s]
 63%|██████▎   | 4055/6412 [00:00<00:00, 18606.84it/s]
100%|██████████| 6412/6412 [00:00<00:00, 21438.13it/s]

  0%|          | 0/70275 [00:00<?, ?it/s]
  4%|▍         | 2930/70275 [00:00<00:02, 29293.88it/s]
  8%|▊         | 5524/70275 [00:00<00:02, 28199.56it/s]
 11%|█▏        | 8012/70275 [00:00<00:02, 27106.70it/s]
 15%|█▍        | 10275/70275 [00:00<00:02, 25587.34it/s]
 18%|█▊        | 12716/70275 [00:00<00:02, 25221.23it/s]
 22%|██▏       | 15134/70275 [00:00<00:02, 24898.26it/s]
 25%|██▌       | 17595/70275 [00:00<00:02, 24807.81it/s]
 29%|██▊       | 20128/70275 [00:00<00:02, 24961.28it/s]
 32%|███▏      | 22466/70275 [00:00<00:01, 24464.25it/s]
 35%|███▌      | 24845/70275 [00:01<00:01, 24254.71it/s]
 39%|███▊      | 27211/70275 [00:01<00:01, 24070.44it/s]
 42%|████▏     | 29667/70275 [00:01<00:01, 24212.97it/s]
 46%|████▌     | 32169/70275 [00:01<00:01, 24446.86it/s]
 49%|████▉     | 34608/70275 [00:01<00:01, 24426.53it/s]
 53%|█████▎    | 37033/70275 [00:01<00:01, 24031.81it/s]
 56%|█████▌    | 39457/70275 [00:01<00:01, 24091.87it/s]
 60%|█████▉    | 41859/70275 [00:01<00:01, 23444.23it/s]
 63%|██████▎   | 44376/70275 [00:01<00:01, 23935.13it/s]
 67%|██████▋   | 47055/70275 [00:01<00:00, 24723.30it/s]
 70%|███████   | 49535/70275 [00:02<00:00, 23967.88it/s]
 74%|███████▍  | 51943/70275 [00:02<00:00, 23479.23it/s]
 78%|███████▊  | 54557/70275 [00:02<00:00, 24218.28it/s]
 81%|████████  | 57091/70275 [00:02<00:00, 24540.96it/s]
 85%|████████▍ | 59695/70275 [00:02<00:00, 24972.22it/s]
 89%|████████▊ | 62264/70275 [00:02<00:00, 25182.77it/s]
 93%|█████████▎| 65019/70275 [00:02<00:00, 25846.79it/s]
 97%|█████████▋| 67842/70275 [00:02<00:00, 26516.67it/s]
100%|██████████| 70275/70275 [00:02<00:00, 24937.55it/s]

  0%|          | 0/70275 [00:00<?, ?it/s]
  7%|▋         | 4636/70275 [00:00<00:01, 46357.83it/s]
 12%|█▏        | 8577/70275 [00:00<00:01, 44026.38it/s]
 18%|█▊        | 12492/70275 [00:00<00:01, 42435.59it/s]
 24%|██▎       | 16515/70275 [00:00<00:01, 41341.76it/s]
 29%|██▉       | 20522/70275 [00:00<00:01, 40084.41it/s]
 35%|███▍      | 24396/70275 [00:00<00:01, 39119.00it/s]
 40%|████      | 28265/70275 [00:00<00:01, 38445.73it/s]
 46%|████▌     | 32220/70275 [00:00<00:01, 37840.70it/s]
 52%|█████▏    | 36353/70275 [00:00<00:00, 37758.44it/s]
 58%|█████▊    | 40470/70275 [00:01<00:00, 37894.86it/s]
 64%|██████▎   | 44772/70275 [00:01<00:00, 38329.20it/s]
 70%|███████   | 49497/70275 [00:01<00:00, 38716.94it/s]
 77%|███████▋  | 54425/70275 [00:01<00:00, 41377.64it/s]
 84%|████████▎ | 58735/70275 [00:01<00:00, 41878.86it/s]
 90%|████████▉ | 63204/70275 [00:01<00:00, 42683.20it/s]
 97%|█████████▋| 67848/70275 [00:01<00:00, 43741.51it/s]
100%|██████████| 70275/70275 [00:01<00:00, 40881.28it/s]

  0%|          | 0/5623 [00:00<?, ?it/s]
100%|██████████| 5623/5623 [00:00<00:00, 83320.93it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.38it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.31it/s]
  3%|▎         | 3/100 [00:00<00:13,  7.18it/s]
  4%|▍         | 4/100 [00:00<00:13,  6.95it/s]
  5%|▌         | 5/100 [00:00<00:13,  7.05it/s]
  6%|▌         | 6/100 [00:00<00:13,  7.11it/s]
  7%|▋         | 7/100 [00:00<00:13,  7.04it/s]
  8%|▊         | 8/100 [00:01<00:13,  6.67it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.83it/s]
 10%|█         | 10/100 [00:01<00:13,  6.86it/s]
 11%|█         | 11/100 [00:01<00:12,  7.03it/s]
 12%|█▏        | 12/100 [00:01<00:12,  6.78it/s]
 13%|█▎        | 13/100 [00:01<00:12,  6.94it/s]
 14%|█▍        | 14/100 [00:02<00:12,  6.88it/s]
 15%|█▌        | 15/100 [00:02<00:12,  6.65it/s]
 16%|█▌        | 16/100 [00:02<00:12,  6.59it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.79it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.80it/s]
 19%|█▉        | 19/100 [00:02<00:11,  6.75it/s]
 20%|██        | 20/100 [00:02<00:11,  6.87it/s]
 21%|██        | 21/100 [00:03<00:11,  6.94it/s]
 22%|██▏       | 22/100 [00:03<00:11,  7.05it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.80it/s]
 24%|██▍       | 24/100 [00:03<00:11,  6.54it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.61it/s]
 26%|██▌       | 26/100 [00:03<00:11,  6.29it/s]
 27%|██▋       | 27/100 [00:03<00:11,  6.40it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.06it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.30it/s]
 30%|███       | 30/100 [00:04<00:10,  6.57it/s]
 31%|███       | 31/100 [00:04<00:10,  6.28it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.52it/s]
 33%|███▎      | 33/100 [00:04<00:10,  6.69it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.85it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.84it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.76it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.66it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.80it/s]
 39%|███▉      | 39/100 [00:05<00:09,  6.69it/s]
 40%|████      | 40/100 [00:05<00:08,  6.87it/s]
 41%|████      | 41/100 [00:06<00:08,  6.86it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.92it/s]
 43%|████▎     | 43/100 [00:06<00:08,  7.03it/s]
 44%|████▍     | 44/100 [00:06<00:07,  7.21it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.67it/s]
 46%|████▌     | 46/100 [00:06<00:07,  6.85it/s]
 47%|████▋     | 47/100 [00:06<00:07,  6.97it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.91it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.96it/s]
 50%|█████     | 50/100 [00:07<00:07,  7.01it/s]
 51%|█████     | 51/100 [00:07<00:06,  7.18it/s]
 52%|█████▏    | 52/100 [00:07<00:06,  7.07it/s]
 53%|█████▎    | 53/100 [00:07<00:06,  6.86it/s]
 54%|█████▍    | 54/100 [00:07<00:06,  6.79it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.85it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.93it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.66it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.55it/s]
 59%|█████▉    | 59/100 [00:08<00:06,  6.59it/s]
 60%|██████    | 60/100 [00:08<00:06,  6.46it/s]
 61%|██████    | 61/100 [00:09<00:05,  6.67it/s]
 62%|██████▏   | 62/100 [00:09<00:05,  6.37it/s]
 63%|██████▎   | 63/100 [00:09<00:05,  6.37it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.51it/s]
 65%|██████▌   | 65/100 [00:09<00:05,  6.60it/s]
 66%|██████▌   | 66/100 [00:09<00:04,  6.86it/s]
 67%|██████▋   | 67/100 [00:09<00:04,  6.87it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  7.03it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.90it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.96it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.97it/s]
 72%|███████▏  | 72/100 [00:10<00:04,  6.95it/s]
 73%|███████▎  | 73/100 [00:10<00:04,  6.38it/s]
 74%|███████▍  | 74/100 [00:10<00:03,  6.67it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.91it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.93it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.79it/s]
 78%|███████▊  | 78/100 [00:11<00:03,  6.77it/s]
 79%|███████▉  | 79/100 [00:11<00:03,  6.71it/s]
 80%|████████  | 80/100 [00:11<00:03,  6.46it/s]
 81%|████████  | 81/100 [00:11<00:02,  6.54it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.83it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.98it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.83it/s]
 85%|████████▌ | 85/100 [00:12<00:02,  7.12it/s]
 86%|████████▌ | 86/100 [00:12<00:01,  7.24it/s]
 87%|████████▋ | 87/100 [00:12<00:01,  7.34it/s]
 88%|████████▊ | 88/100 [00:12<00:01,  7.16it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  7.04it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.99it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.97it/s]
 92%|█████████▏| 92/100 [00:13<00:01,  6.85it/s]
 93%|█████████▎| 93/100 [00:13<00:01,  6.85it/s]
 94%|█████████▍| 94/100 [00:13<00:00,  6.62it/s]
 95%|█████████▌| 95/100 [00:13<00:00,  6.69it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  7.01it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.96it/s]
 98%|█████████▊| 98/100 [00:14<00:00,  7.23it/s]
 99%|█████████▉| 99/100 [00:14<00:00,  6.84it/s]
100%|██████████| 100/100 [00:14<00:00,  6.73it/s]
100%|██████████| 100/100 [00:14<00:00,  6.79it/s]

  0%|          | 0/70275 [00:00<?, ?it/s]
  7%|▋         | 4604/70275 [00:00<00:01, 46034.12it/s]
 14%|█▍        | 9960/70275 [00:00<00:01, 48058.29it/s]
 24%|██▍       | 17188/70275 [00:00<00:00, 53429.49it/s]
 35%|███▌      | 24620/70275 [00:00<00:00, 58348.12it/s]
 46%|████▋     | 32596/70275 [00:00<00:00, 63458.36it/s]
 58%|█████▊    | 40804/70275 [00:00<00:00, 68090.50it/s]
 71%|███████   | 49762/70275 [00:00<00:00, 73370.70it/s]
 84%|████████▍ | 59051/70275 [00:00<00:00, 78306.27it/s]
 99%|█████████▊| 69288/70275 [00:00<00:00, 84246.56it/s]
100%|██████████| 70275/70275 [00:00<00:00, 77279.98it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/338002 [00:00<?, ?it/s]
  3%|▎         | 10788/338002 [00:00<00:03, 107878.56it/s]
  6%|▋         | 21829/338002 [00:00<00:02, 108624.37it/s]
 10%|▉         | 32956/338002 [00:00<00:02, 109401.79it/s]
 13%|█▎        | 44624/338002 [00:00<00:02, 111486.45it/s]
 17%|█▋        | 56407/338002 [00:00<00:02, 113314.86it/s]
 20%|██        | 68485/338002 [00:00<00:02, 115455.55it/s]
 24%|██▍       | 80455/338002 [00:00<00:02, 116695.26it/s]
 27%|██▋       | 92664/338002 [00:00<00:02, 118261.51it/s]
 31%|███       | 104603/338002 [00:00<00:01, 118594.81it/s]
 35%|███▍      | 116953/338002 [00:01<00:01, 120022.57it/s]
 38%|███▊      | 129123/338002 [00:01<00:01, 120519.25it/s]
 42%|████▏     | 141462/338002 [00:01<00:01, 121366.22it/s]
 45%|████▌     | 153694/338002 [00:01<00:01, 121649.70it/s]
 49%|████▉     | 166183/338002 [00:01<00:01, 122597.66it/s]
 53%|█████▎    | 178378/338002 [00:01<00:01, 122401.36it/s]
 56%|█████▋    | 190951/338002 [00:01<00:01, 123378.47it/s]
 60%|██████    | 203258/338002 [00:01<00:01, 121957.57it/s]
 64%|██████▍   | 215582/338002 [00:01<00:01, 122338.13it/s]
 68%|██████▊   | 228235/338002 [00:01<00:00, 123562.90it/s]
 71%|███████   | 240592/338002 [00:02<00:00, 123564.89it/s]
 75%|███████▍  | 252945/338002 [00:02<00:00, 122826.89it/s]
 79%|███████▊  | 265679/338002 [00:02<00:00, 124144.14it/s]
 82%|████████▏ | 278096/338002 [00:02<00:00, 123206.32it/s]
 86%|████████▌ | 290839/338002 [00:02<00:00, 124442.29it/s]
 90%|████████▉ | 303515/338002 [00:02<00:00, 125126.75it/s]
 94%|█████████▎| 316033/338002 [00:02<00:00, 124039.23it/s]
 97%|█████████▋| 328703/338002 [00:02<00:00, 124825.07it/s]
100%|██████████| 338002/338002 [00:02<00:00, 121424.22it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6206/150000 [00:00<00:02, 62056.51it/s]
  8%|▊         | 12607/150000 [00:00<00:02, 62627.75it/s]
 13%|█▎        | 19029/150000 [00:00<00:02, 63096.78it/s]
 17%|█▋        | 25549/150000 [00:00<00:01, 63712.13it/s]
 21%|██▏       | 32168/150000 [00:00<00:01, 64433.25it/s]
 26%|██▌       | 38873/150000 [00:00<00:01, 65194.46it/s]
 30%|███       | 45588/150000 [00:00<00:01, 65767.65it/s]
 35%|███▍      | 52416/150000 [00:00<00:01, 66500.35it/s]
 40%|███▉      | 59265/150000 [00:00<00:01, 67084.13it/s]
 44%|████▍     | 66079/150000 [00:01<00:01, 67396.07it/s]
 49%|████▊     | 72947/150000 [00:01<00:01, 67773.33it/s]
 53%|█████▎    | 79780/150000 [00:01<00:01, 67936.79it/s]
 58%|█████▊    | 86662/150000 [00:01<00:00, 68197.83it/s]
 62%|██████▏   | 93480/150000 [00:01<00:00, 68189.77it/s]
 67%|██████▋   | 100276/150000 [00:01<00:00, 68118.45it/s]
 71%|███████▏  | 107110/150000 [00:01<00:00, 68183.37it/s]
 76%|███████▌  | 113913/150000 [00:01<00:00, 68134.38it/s]
 80%|████████  | 120744/150000 [00:01<00:00, 68185.19it/s]
 85%|████████▌ | 127568/150000 [00:01<00:00, 68198.91it/s]
 90%|████████▉ | 134392/150000 [00:02<00:00, 68210.72it/s]
 94%|█████████▍| 141223/150000 [00:02<00:00, 68238.69it/s]
 99%|█████████▊| 148044/150000 [00:02<00:00, 68210.24it/s]
100%|██████████| 150000/150000 [00:02<00:00, 67288.23it/s]

  0%|          | 0/135 [00:00<?, ?it/s]
 5623 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-18 ...
Successfully removed 6412 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-19 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 12065 new domains found

Scrapping urls...
scrapping will be done in 135 phase  where 90 urls attempt in each phase

  1%|          | 1/135 [00:21<48:15, 21.61s/it]
  1%|▏         | 2/135 [00:40<46:03, 20.78s/it]
  2%|▏         | 3/135 [01:01<45:39, 20.75s/it]185.202.2.149 - - [29/May/2020 03:38:41] code 400, message Bad HTTP/0.9 request type ('\x03\x00\x00/*à\x00\x00\x00\x00\x00Cookie:')
185.202.2.149 - - [29/May/2020 03:38:41] "/*àCookie: mstshash=Administr" HTTPStatus.BAD_REQUEST -

  3%|▎         | 4/135 [01:40<57:12, 26.20s/it]
  4%|▎         | 5/135 [02:01<53:32, 24.71s/it]
  4%|▍         | 6/135 [02:29<55:05, 25.62s/it]
  5%|▌         | 7/135 [02:51<52:48, 24.75s/it]
  6%|▌         | 8/135 [03:10<48:26, 22.89s/it]
  7%|▋         | 9/135 [03:31<46:54, 22.34s/it]
  7%|▋         | 10/135 [03:53<46:11, 22.17s/it]
  8%|▊         | 11/135 [04:12<44:06, 21.34s/it]
  9%|▉         | 12/135 [04:32<42:52, 20.91s/it]
 10%|▉         | 13/135 [04:50<40:28, 19.91s/it]
 10%|█         | 14/135 [05:42<59:53, 29.70s/it]
 11%|█         | 15/135 [05:53<48:23, 24.19s/it]
 12%|█▏        | 16/135 [06:30<55:23, 27.93s/it]
 13%|█▎        | 17/135 [06:47<48:20, 24.58s/it]
 13%|█▎        | 18/135 [07:09<46:15, 23.72s/it]
 14%|█▍        | 19/135 [07:25<41:35, 21.51s/it]
 15%|█▍        | 20/135 [07:41<38:09, 19.91s/it]
 16%|█▌        | 21/135 [08:02<38:17, 20.15s/it]
 16%|█▋        | 22/135 [08:17<35:22, 18.78s/it]
 17%|█▋        | 23/135 [08:35<34:35, 18.53s/it]
 18%|█▊        | 24/135 [08:51<32:47, 17.73s/it]
 19%|█▊        | 25/135 [09:27<42:34, 23.22s/it]
 19%|█▉        | 26/135 [09:41<36:50, 20.28s/it]
 20%|██        | 27/135 [10:18<45:52, 25.49s/it]
 21%|██        | 28/135 [10:39<42:41, 23.94s/it]
 21%|██▏       | 29/135 [11:21<52:10, 29.53s/it]
 22%|██▏       | 30/135 [11:42<46:55, 26.81s/it]
 23%|██▎       | 31/135 [12:19<51:49, 29.90s/it]
 24%|██▎       | 32/135 [13:01<57:45, 33.65s/it]
 24%|██▍       | 33/135 [13:35<57:24, 33.77s/it]
 25%|██▌       | 34/135 [13:56<50:20, 29.90s/it]
 26%|██▌       | 35/135 [14:30<51:57, 31.18s/it]
 27%|██▋       | 36/135 [14:52<46:50, 28.39s/it]
 27%|██▋       | 37/135 [15:37<54:13, 33.20s/it]
 28%|██▊       | 38/135 [16:08<52:59, 32.78s/it]
 29%|██▉       | 39/135 [16:44<54:03, 33.79s/it]
 30%|██▉       | 40/135 [17:21<54:40, 34.53s/it]
 30%|███       | 41/135 [18:02<57:27, 36.68s/it]
 31%|███       | 42/135 [18:42<58:07, 37.50s/it]
 32%|███▏      | 43/135 [19:18<56:54, 37.11s/it]
 33%|███▎      | 44/135 [19:50<53:51, 35.51s/it]
 33%|███▎      | 45/135 [20:31<55:39, 37.11s/it]
 34%|███▍      | 46/135 [21:01<51:53, 34.98s/it]
 35%|███▍      | 47/135 [21:33<49:56, 34.05s/it]
 36%|███▌      | 48/135 [22:06<49:01, 33.81s/it]
 36%|███▋      | 49/135 [22:44<50:17, 35.09s/it]
 37%|███▋      | 50/135 [23:16<48:19, 34.11s/it]
 38%|███▊      | 51/135 [23:50<48:01, 34.30s/it]
 39%|███▊      | 52/135 [24:29<49:08, 35.53s/it]
 39%|███▉      | 53/135 [24:53<44:05, 32.26s/it]
 40%|████      | 54/135 [25:32<46:02, 34.11s/it]
 41%|████      | 55/135 [26:08<46:16, 34.71s/it]
 41%|████▏     | 56/135 [26:46<46:51, 35.59s/it]
 42%|████▏     | 57/135 [27:22<46:27, 35.74s/it]
 43%|████▎     | 58/135 [27:49<42:33, 33.17s/it]
 44%|████▎     | 59/135 [28:32<45:36, 36.01s/it]
 44%|████▍     | 60/135 [29:08<45:04, 36.06s/it]
 45%|████▌     | 61/135 [29:41<43:22, 35.17s/it]
 46%|████▌     | 62/135 [30:13<41:52, 34.42s/it]
 47%|████▋     | 63/135 [30:46<40:40, 33.90s/it]
 47%|████▋     | 64/135 [31:19<39:53, 33.70s/it]
 48%|████▊     | 65/135 [31:55<39:57, 34.24s/it]
 49%|████▉     | 66/135 [32:18<35:33, 30.92s/it]
 50%|████▉     | 67/135 [32:46<33:52, 29.88s/it]
 50%|█████     | 68/135 [33:17<33:53, 30.35s/it]
 51%|█████     | 69/135 [33:47<33:20, 30.31s/it]
 52%|█████▏    | 70/135 [34:23<34:33, 31.90s/it]
 53%|█████▎    | 71/135 [34:59<35:22, 33.17s/it]
 53%|█████▎    | 72/135 [35:28<33:24, 31.81s/it]
 54%|█████▍    | 73/135 [35:55<31:34, 30.55s/it]
 55%|█████▍    | 74/135 [36:26<31:09, 30.66s/it]
 56%|█████▌    | 75/135 [36:55<29:59, 30.00s/it]
 56%|█████▋    | 76/135 [37:29<30:41, 31.21s/it]
 57%|█████▋    | 77/135 [37:56<29:04, 30.08s/it]
 58%|█████▊    | 78/135 [38:25<28:10, 29.66s/it]
 59%|█████▊    | 79/135 [38:56<28:07, 30.13s/it]
 59%|█████▉    | 80/135 [39:19<25:42, 28.05s/it]
 60%|██████    | 81/135 [39:53<26:45, 29.72s/it]
 61%|██████    | 82/135 [40:09<22:41, 25.69s/it]
 61%|██████▏   | 83/135 [40:39<23:21, 26.95s/it]
 62%|██████▏   | 84/135 [41:06<22:53, 26.93s/it]
 63%|██████▎   | 85/135 [41:32<22:21, 26.83s/it]
 64%|██████▎   | 86/135 [41:46<18:36, 22.78s/it]
 64%|██████▍   | 87/135 [42:13<19:22, 24.22s/it]
 65%|██████▌   | 88/135 [42:50<21:48, 27.83s/it]
 66%|██████▌   | 89/135 [43:20<21:56, 28.62s/it]
 67%|██████▋   | 90/135 [43:45<20:37, 27.49s/it]
 67%|██████▋   | 91/135 [44:09<19:25, 26.49s/it]
 68%|██████▊   | 92/135 [44:24<16:31, 23.06s/it]
 69%|██████▉   | 93/135 [44:39<14:27, 20.65s/it]
 70%|██████▉   | 94/135 [45:18<17:50, 26.10s/it]
 70%|███████   | 95/135 [45:42<16:56, 25.41s/it]
 71%|███████   | 96/135 [46:06<16:19, 25.12s/it]
 72%|███████▏  | 97/135 [46:45<18:28, 29.18s/it]
 73%|███████▎  | 98/135 [47:04<16:09, 26.20s/it]
 73%|███████▎  | 99/135 [47:34<16:26, 27.40s/it]
 74%|███████▍  | 100/135 [47:49<13:41, 23.48s/it]
 75%|███████▍  | 101/135 [48:24<15:15, 26.92s/it]
 76%|███████▌  | 102/135 [48:50<14:40, 26.68s/it]
 76%|███████▋  | 103/135 [49:21<14:57, 28.03s/it]
 77%|███████▋  | 104/135 [49:44<13:44, 26.59s/it]
 78%|███████▊  | 105/135 [50:11<13:20, 26.68s/it]
 79%|███████▊  | 106/135 [50:28<11:28, 23.74s/it]
 79%|███████▉  | 107/135 [50:50<10:48, 23.17s/it]
 80%|████████  | 108/135 [51:24<11:52, 26.39s/it]
 81%|████████  | 109/135 [51:49<11:20, 26.17s/it]
 81%|████████▏ | 110/135 [52:02<09:13, 22.12s/it]
 82%|████████▏ | 111/135 [52:25<08:59, 22.48s/it]
 83%|████████▎ | 112/135 [52:49<08:45, 22.86s/it]
 84%|████████▎ | 113/135 [53:21<09:21, 25.53s/it]
 84%|████████▍ | 114/135 [53:52<09:29, 27.13s/it]
 85%|████████▌ | 115/135 [54:32<10:24, 31.24s/it]
 86%|████████▌ | 116/135 [55:06<10:07, 31.97s/it]
 87%|████████▋ | 117/135 [55:38<09:37, 32.09s/it]
 87%|████████▋ | 118/135 [55:59<08:05, 28.58s/it]
 88%|████████▊ | 119/135 [56:29<07:44, 29.05s/it]
 89%|████████▉ | 120/135 [57:08<08:01, 32.12s/it]
 90%|████████▉ | 121/135 [57:19<06:01, 25.82s/it]
 90%|█████████ | 122/135 [57:50<05:53, 27.16s/it]
 91%|█████████ | 123/135 [58:04<04:39, 23.28s/it]
 92%|█████████▏| 124/135 [58:25<04:09, 22.64s/it]
 93%|█████████▎| 125/135 [58:41<03:27, 20.75s/it]
 93%|█████████▎| 126/135 [59:07<03:19, 22.17s/it]
 94%|█████████▍| 127/135 [59:20<02:36, 19.54s/it]
 95%|█████████▍| 128/135 [59:34<02:04, 17.73s/it]
 96%|█████████▌| 129/135 [59:50<01:44, 17.34s/it]
 96%|█████████▋| 130/135 [1:00:04<01:21, 16.27s/it]
 97%|█████████▋| 131/135 [1:00:27<01:13, 18.38s/it]
 98%|█████████▊| 132/135 [1:00:50<00:59, 19.68s/it]
 99%|█████████▊| 133/135 [1:01:26<00:49, 24.57s/it]
 99%|█████████▉| 134/135 [1:02:00<00:27, 27.51s/it]
100%|██████████| 135/135 [1:02:01<00:00, 19.58s/it]
100%|██████████| 135/135 [1:02:01<00:00, 27.57s/it]

  0%|          | 0/231266 [00:00<?, ?it/s]
  5%|▍         | 11054/231266 [00:00<00:01, 110538.26it/s]
 10%|▉         | 22079/231266 [00:00<00:01, 110450.79it/s]
 15%|█▍        | 33635/231266 [00:00<00:01, 111932.95it/s]
 20%|█▉        | 45298/231266 [00:00<00:01, 113299.68it/s]
 25%|██▍       | 57248/231266 [00:00<00:01, 115089.20it/s]
 30%|██▉       | 69340/231266 [00:00<00:01, 116776.22it/s]
 35%|███▌      | 81169/231266 [00:00<00:01, 117223.48it/s]
 40%|████      | 93030/231266 [00:00<00:01, 117635.30it/s]
 46%|████▌     | 105251/231266 [00:00<00:01, 118968.81it/s]
 51%|█████     | 117474/231266 [00:01<00:00, 119927.10it/s]
 56%|█████▌    | 129724/231266 [00:01<00:00, 120686.82it/s]
 61%|██████▏   | 141928/231266 [00:01<00:00, 121088.44it/s]
 67%|██████▋   | 154074/231266 [00:01<00:00, 121199.27it/s]
 72%|███████▏  | 166091/231266 [00:01<00:00, 119380.82it/s]
 77%|███████▋  | 177963/231266 [00:01<00:00, 116917.75it/s]
 82%|████████▏ | 190144/231266 [00:01<00:00, 118343.36it/s]
 87%|████████▋ | 202354/231266 [00:01<00:00, 119442.96it/s]
 93%|█████████▎| 214546/231266 [00:01<00:00, 120173.61it/s]
 98%|█████████▊| 226723/231266 [00:01<00:00, 120647.21it/s]
100%|██████████| 231266/231266 [00:01<00:00, 118701.25it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 10729/150000 [00:00<00:01, 107287.54it/s]
 15%|█▍        | 21790/150000 [00:00<00:01, 108260.81it/s]
 22%|██▏       | 33170/150000 [00:00<00:01, 109862.94it/s]
 30%|███       | 45008/150000 [00:00<00:00, 112285.73it/s]
 38%|███▊      | 57054/150000 [00:00<00:00, 114618.20it/s]
 46%|████▌     | 69247/150000 [00:00<00:00, 116715.77it/s]
 54%|█████▍    | 81419/150000 [00:00<00:00, 118172.01it/s]
 62%|██████▏   | 93622/150000 [00:00<00:00, 119300.66it/s]
 71%|███████   | 105902/150000 [00:00<00:00, 120327.52it/s]
 79%|███████▉  | 118199/150000 [00:01<00:00, 121104.80it/s]
 87%|████████▋ | 130621/150000 [00:01<00:00, 122019.61it/s]
 95%|█████████▌| 143098/150000 [00:01<00:00, 122830.73it/s]
100%|██████████| 150000/150000 [00:01<00:00, 119481.57it/s]

0it [00:00, ?it/s]
86it [00:00, 857.92it/s]
227it [00:00, 969.48it/s]
377it [00:00, 1083.47it/s]
531it [00:00, 1188.46it/s]
711it [00:00, 1321.26it/s]
924it [00:00, 1488.18it/s]
1131it [00:00, 1622.22it/s]
1442it [00:00, 1892.26it/s]
1779it [00:00, 2174.00it/s]
1963it [00:00, 2091.75it/s]

  0%|          | 0/2387 [00:00<?, ?it/s]
 28%|██▊       | 675/2387 [00:00<00:00, 6315.75it/s]
 66%|██████▌   | 1575/2387 [00:00<00:00, 6926.08it/s]
100%|██████████| 2387/2387 [00:00<00:00, 9066.37it/s]

  0%|          | 0/67735 [00:00<?, ?it/s]
  0%|          | 98/67735 [00:00<01:12, 939.35it/s]
  1%|          | 361/67735 [00:00<00:58, 1154.01it/s]
  1%|          | 612/67735 [00:00<00:48, 1377.18it/s]
  1%|          | 827/67735 [00:00<00:51, 1290.23it/s]
  1%|▏         | 960/67735 [00:00<00:52, 1275.66it/s]
  2%|▏         | 1090/67735 [00:00<00:52, 1263.79it/s]
  2%|▏         | 1219/67735 [00:00<01:07, 982.15it/s] 
  2%|▏         | 1595/67735 [00:01<00:52, 1261.81it/s]
  6%|▌         | 3903/67735 [00:01<00:36, 1761.29it/s]
  9%|▉         | 6254/67735 [00:01<00:25, 2437.86it/s]
 13%|█▎        | 8727/67735 [00:01<00:17, 3341.45it/s]
 16%|█▋        | 11143/67735 [00:01<00:12, 4506.38it/s]
 20%|██        | 13564/67735 [00:01<00:09, 5961.97it/s]
 24%|██▎       | 16052/67735 [00:01<00:06, 7723.78it/s]
 27%|██▋       | 18507/67735 [00:01<00:05, 9722.67it/s]
 31%|███       | 21000/67735 [00:01<00:03, 11900.09it/s]
 35%|███▍      | 23531/67735 [00:01<00:03, 14148.55it/s]
 38%|███▊      | 26030/67735 [00:02<00:02, 16264.88it/s]
 42%|████▏     | 28580/67735 [00:02<00:02, 18247.21it/s]
 46%|████▌     | 31145/67735 [00:02<00:01, 19976.22it/s]
 50%|████▉     | 33637/67735 [00:02<00:01, 20435.08it/s]
 53%|█████▎    | 36029/67735 [00:02<00:01, 21096.11it/s]
 57%|█████▋    | 38531/67735 [00:02<00:01, 22136.85it/s]
 61%|██████    | 41109/67735 [00:02<00:01, 23115.46it/s]
 64%|██████▍   | 43564/67735 [00:02<00:01, 23191.19it/s]
 68%|██████▊   | 45984/67735 [00:02<00:00, 23436.89it/s]
 72%|███████▏  | 48535/67735 [00:02<00:00, 24020.21it/s]
 75%|███████▌  | 51133/67735 [00:03<00:00, 24574.76it/s]
 79%|███████▉  | 53631/67735 [00:03<00:00, 24254.09it/s]
 83%|████████▎ | 56086/67735 [00:03<00:00, 24294.58it/s]
 87%|████████▋ | 58750/67735 [00:03<00:00, 24952.24it/s]
 91%|█████████ | 61485/67735 [00:03<00:00, 25625.76it/s]
 95%|█████████▍| 64215/67735 [00:03<00:00, 26103.87it/s]
 99%|█████████▊| 66840/67735 [00:03<00:00, 25088.25it/s]
100%|██████████| 67735/67735 [00:03<00:00, 18301.97it/s]

  0%|          | 0/67735 [00:00<?, ?it/s]
  7%|▋         | 4951/67735 [00:00<00:01, 48173.60it/s]
 14%|█▎        | 9251/67735 [00:00<00:01, 45886.67it/s]
 20%|██        | 13830/67735 [00:00<00:01, 44566.18it/s]
 27%|██▋       | 18366/67735 [00:00<00:01, 43231.02it/s]
 34%|███▎      | 22776/67735 [00:00<00:01, 43487.28it/s]
 39%|███▉      | 26422/67735 [00:00<00:01, 41107.84it/s]
 44%|████▍     | 30006/67735 [00:00<00:00, 39302.87it/s]
 50%|████▉     | 33635/67735 [00:00<00:00, 38346.87it/s]
 55%|█████▍    | 37229/67735 [00:00<00:00, 37382.42it/s]
 60%|██████    | 40804/67735 [00:01<00:00, 36645.42it/s]
 65%|██████▌   | 44358/67735 [00:01<00:00, 36282.37it/s]
 71%|███████   | 47910/67735 [00:01<00:00, 35633.65it/s]
 76%|███████▌  | 51628/67735 [00:01<00:00, 35353.16it/s]
 83%|████████▎ | 56005/67735 [00:01<00:00, 36083.02it/s]
 89%|████████▉ | 60594/67735 [00:01<00:00, 36973.46it/s]
 97%|█████████▋| 65465/67735 [00:01<00:00, 39852.79it/s]
100%|██████████| 67735/67735 [00:01<00:00, 38881.27it/s]

  0%|          | 0/1963 [00:00<?, ?it/s]
100%|██████████| 1963/1963 [00:00<00:00, 78174.52it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.27it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.22it/s]
  3%|▎         | 3/100 [00:00<00:13,  7.14it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.85it/s]
  5%|▌         | 5/100 [00:00<00:13,  6.94it/s]
  6%|▌         | 6/100 [00:00<00:13,  6.96it/s]
  7%|▋         | 7/100 [00:01<00:13,  6.91it/s]
  8%|▊         | 8/100 [00:01<00:13,  6.64it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.81it/s]
 10%|█         | 10/100 [00:01<00:13,  6.81it/s]
 11%|█         | 11/100 [00:01<00:12,  6.98it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.73it/s]
 13%|█▎        | 13/100 [00:01<00:12,  6.90it/s]
 14%|█▍        | 14/100 [00:02<00:12,  6.88it/s]
 15%|█▌        | 15/100 [00:02<00:12,  6.59it/s]
 16%|█▌        | 16/100 [00:02<00:12,  6.52it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.74it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.39it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.37it/s]
 20%|██        | 20/100 [00:02<00:12,  6.54it/s]
 21%|██        | 21/100 [00:03<00:11,  6.65it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.83it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.70it/s]
 24%|██▍       | 24/100 [00:03<00:11,  6.38it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.29it/s]
 26%|██▌       | 26/100 [00:03<00:12,  5.90it/s]
 27%|██▋       | 27/100 [00:04<00:12,  5.87it/s]
 28%|██▊       | 28/100 [00:04<00:12,  5.57it/s]
 29%|██▉       | 29/100 [00:04<00:12,  5.75it/s]
 30%|███       | 30/100 [00:04<00:12,  5.63it/s]
 31%|███       | 31/100 [00:04<00:12,  5.55it/s]
 32%|███▏      | 32/100 [00:05<00:11,  5.81it/s]
 33%|███▎      | 33/100 [00:05<00:11,  6.08it/s]
 34%|███▍      | 34/100 [00:05<00:10,  6.31it/s]
 35%|███▌      | 35/100 [00:05<00:10,  6.41it/s]
 36%|███▌      | 36/100 [00:05<00:10,  6.30it/s]
 37%|███▋      | 37/100 [00:05<00:10,  6.28it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.47it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.39it/s]
 40%|████      | 40/100 [00:06<00:09,  6.59it/s]
 41%|████      | 41/100 [00:06<00:08,  6.58it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.70it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.76it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.95it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.42it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.55it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.59it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.56it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.64it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.71it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.86it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.78it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.59it/s]
 54%|█████▍    | 54/100 [00:08<00:06,  6.59it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.69it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.68it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.48it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.34it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.34it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.20it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.40it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.18it/s]
 63%|██████▎   | 63/100 [00:09<00:05,  6.18it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.37it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.42it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.62it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.59it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.81it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.67it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.79it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.87it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.89it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.35it/s]
 74%|███████▍  | 74/100 [00:11<00:03,  6.62it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.83it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.83it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.76it/s]
 78%|███████▊  | 78/100 [00:11<00:03,  6.72it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.59it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.34it/s]
 81%|████████  | 81/100 [00:12<00:02,  6.40it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.69it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.85it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.57it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.79it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.91it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  7.03it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.85it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.77it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.78it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.80it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.63it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.63it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.39it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.48it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.77it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.70it/s]
 98%|█████████▊| 98/100 [00:14<00:00,  6.98it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.68it/s]
100%|██████████| 100/100 [00:15<00:00,  6.58it/s]
100%|██████████| 100/100 [00:15<00:00,  6.55it/s]

  0%|          | 0/67735 [00:00<?, ?it/s]
  6%|▋         | 4402/67735 [00:00<00:01, 44017.73it/s]
 15%|█▍        | 9843/67735 [00:00<00:01, 46691.85it/s]
 25%|██▌       | 17140/67735 [00:00<00:00, 52347.05it/s]
 36%|███▋      | 24667/67735 [00:00<00:00, 57609.58it/s]
 48%|████▊     | 32662/67735 [00:00<00:00, 62879.70it/s]
 60%|██████    | 40950/67735 [00:00<00:00, 67785.69it/s]
 73%|███████▎  | 49504/67735 [00:00<00:00, 72284.84it/s]
 87%|████████▋ | 58798/67735 [00:00<00:00, 77447.35it/s]
100%|██████████| 67735/67735 [00:00<00:00, 76234.96it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/328563 [00:00<?, ?it/s]
  3%|▎         | 11037/328563 [00:00<00:02, 110364.05it/s]
  7%|▋         | 22269/328563 [00:00<00:02, 110941.09it/s]
 10%|█         | 33721/328563 [00:00<00:02, 111990.62it/s]
 14%|█▍        | 45643/328563 [00:00<00:02, 114062.96it/s]
 18%|█▊        | 57790/328563 [00:00<00:02, 116186.26it/s]
 21%|██▏       | 70107/328563 [00:00<00:02, 118195.28it/s]
 25%|██▌       | 82372/328563 [00:00<00:02, 119494.88it/s]
 29%|██▉       | 94836/328563 [00:00<00:01, 120992.62it/s]
 33%|███▎      | 107177/328563 [00:00<00:01, 121705.80it/s]
 36%|███▋      | 119549/328563 [00:01<00:01, 122302.66it/s]
 40%|████      | 131990/328563 [00:01<00:01, 122924.84it/s]
 44%|████▍     | 144580/328563 [00:01<00:01, 123799.94it/s]
 48%|████▊     | 157096/328563 [00:01<00:01, 124203.38it/s]
 52%|█████▏    | 169634/328563 [00:01<00:01, 124552.13it/s]
 55%|█████▌    | 182174/328563 [00:01<00:01, 124802.03it/s]
 59%|█████▉    | 194699/328563 [00:01<00:01, 124935.68it/s]
 63%|██████▎   | 207158/328563 [00:01<00:00, 123654.54it/s]
 67%|██████▋   | 219715/328563 [00:01<00:00, 124221.29it/s]
 71%|███████   | 232463/328563 [00:01<00:00, 125179.99it/s]
 75%|███████▍  | 245160/328563 [00:02<00:00, 125709.22it/s]
 78%|███████▊  | 257726/328563 [00:02<00:00, 125435.66it/s]
 82%|████████▏ | 270649/328563 [00:02<00:00, 126548.98it/s]
 86%|████████▌ | 283304/328563 [00:02<00:00, 125754.82it/s]
 90%|█████████ | 296026/328563 [00:02<00:00, 126189.42it/s]
 94%|█████████▍| 308654/328563 [00:02<00:00, 126214.82it/s]
 98%|█████████▊| 321555/328563 [00:02<00:00, 127039.71it/s]
100%|██████████| 328563/328563 [00:02<00:00, 123217.44it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6420/150000 [00:00<00:02, 64194.40it/s]
  9%|▊         | 13003/150000 [00:00<00:02, 64675.77it/s]
 13%|█▎        | 19599/150000 [00:00<00:02, 65053.90it/s]
 18%|█▊        | 26336/150000 [00:00<00:01, 65730.75it/s]
 22%|██▏       | 33096/150000 [00:00<00:01, 66278.13it/s]
 27%|██▋       | 39960/150000 [00:00<00:01, 66966.29it/s]
 31%|███       | 46838/150000 [00:00<00:01, 67498.69it/s]
 36%|███▌      | 53789/150000 [00:00<00:01, 68087.77it/s]
 40%|████      | 60705/150000 [00:00<00:01, 68405.47it/s]
 45%|████▌     | 67669/150000 [00:01<00:01, 68768.65it/s]
 50%|████▉     | 74663/150000 [00:01<00:01, 69113.93it/s]
 54%|█████▍    | 81612/150000 [00:01<00:00, 69225.06it/s]
 59%|█████▉    | 88596/150000 [00:01<00:00, 69405.76it/s]
 64%|██████▎   | 95561/150000 [00:01<00:00, 69476.94it/s]
 68%|██████▊   | 102513/150000 [00:01<00:00, 69487.19it/s]
 73%|███████▎  | 109517/150000 [00:01<00:00, 69649.75it/s]
 78%|███████▊  | 116462/150000 [00:01<00:00, 69584.04it/s]
 82%|████████▏ | 123408/150000 [00:01<00:00, 69544.41it/s]
 87%|████████▋ | 130353/150000 [00:01<00:00, 69411.92it/s]
 92%|█████████▏| 137288/150000 [00:02<00:00, 69272.13it/s]
 96%|█████████▌| 144211/150000 [00:02<00:00, 69257.77it/s]
100%|██████████| 150000/150000 [00:02<00:00, 68658.10it/s]

  0%|          | 0/125 [00:00<?, ?it/s]
 1963 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-19 ...
Successfully removed 2387 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-20 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 11226 new domains found

Scrapping urls...
scrapping will be done in 125 phase  where 90 urls attempt in each phase

  1%|          | 1/125 [00:12<26:33, 12.85s/it]
  2%|▏         | 2/125 [00:29<28:37, 13.96s/it]
  2%|▏         | 3/125 [01:02<39:51, 19.60s/it]
  3%|▎         | 4/125 [01:18<37:19, 18.51s/it]
  4%|▍         | 5/125 [01:49<44:29, 22.25s/it]
  5%|▍         | 6/125 [02:05<40:42, 20.52s/it]
  6%|▌         | 7/125 [02:31<43:39, 22.20s/it]
  6%|▋         | 8/125 [02:42<36:37, 18.79s/it]
  7%|▋         | 9/125 [02:53<31:41, 16.39s/it]
  8%|▊         | 10/125 [03:16<35:07, 18.32s/it]
  9%|▉         | 11/125 [03:29<32:09, 16.93s/it]
 10%|▉         | 12/125 [03:47<32:11, 17.09s/it]
 10%|█         | 13/125 [04:23<42:51, 22.96s/it]
 11%|█         | 14/125 [04:44<41:11, 22.26s/it]
 12%|█▏        | 15/125 [05:08<41:55, 22.87s/it]
 13%|█▎        | 16/125 [05:29<40:09, 22.11s/it]
 14%|█▎        | 17/125 [05:49<38:53, 21.61s/it]
 14%|█▍        | 18/125 [06:12<39:06, 21.93s/it]
 15%|█▌        | 19/125 [06:25<33:56, 19.21s/it]
 16%|█▌        | 20/125 [06:54<39:02, 22.31s/it]
 17%|█▋        | 21/125 [07:27<44:10, 25.48s/it]
 18%|█▊        | 22/125 [08:03<49:14, 28.68s/it]
 18%|█▊        | 23/125 [08:46<55:52, 32.87s/it]
 19%|█▉        | 24/125 [09:01<46:35, 27.68s/it]
 20%|██        | 25/125 [09:28<45:43, 27.44s/it]
 21%|██        | 26/125 [09:55<44:42, 27.10s/it]
 22%|██▏       | 27/125 [10:33<49:41, 30.42s/it]
 22%|██▏       | 28/125 [11:12<53:36, 33.16s/it]
 23%|██▎       | 29/125 [11:37<48:50, 30.53s/it]
 24%|██▍       | 30/125 [12:19<53:49, 34.00s/it]
 25%|██▍       | 31/125 [12:41<47:54, 30.58s/it]
 26%|██▌       | 32/125 [13:10<46:15, 29.84s/it]
 26%|██▋       | 33/125 [13:35<43:35, 28.43s/it]
 27%|██▋       | 34/125 [14:14<47:55, 31.60s/it]
 28%|██▊       | 35/125 [14:39<44:20, 29.56s/it]
 29%|██▉       | 36/125 [15:11<45:10, 30.45s/it]
 30%|██▉       | 37/125 [15:38<43:15, 29.50s/it]
 30%|███       | 38/125 [16:10<43:49, 30.22s/it]
 31%|███       | 39/125 [16:46<45:29, 31.74s/it]
 32%|███▏      | 40/125 [17:16<44:14, 31.23s/it]
 33%|███▎      | 41/125 [17:38<39:57, 28.54s/it]
 34%|███▎      | 42/125 [18:00<36:49, 26.62s/it]
 34%|███▍      | 43/125 [18:36<40:19, 29.51s/it]
 35%|███▌      | 44/125 [19:06<39:51, 29.53s/it]
 36%|███▌      | 45/125 [19:43<42:32, 31.90s/it]
 37%|███▋      | 46/125 [20:16<42:13, 32.08s/it]
 38%|███▊      | 47/125 [20:47<41:33, 31.97s/it]
 38%|███▊      | 48/125 [21:10<37:25, 29.16s/it]
 39%|███▉      | 49/125 [21:46<39:33, 31.23s/it]
 40%|████      | 50/125 [22:07<35:06, 28.09s/it]
 41%|████      | 51/125 [22:31<33:05, 26.83s/it]
 42%|████▏     | 52/125 [22:55<31:48, 26.14s/it]
 42%|████▏     | 53/125 [23:34<35:56, 29.95s/it]
 43%|████▎     | 54/125 [24:08<36:59, 31.26s/it]
 44%|████▍     | 55/125 [24:47<39:05, 33.51s/it]
 45%|████▍     | 56/125 [25:12<35:28, 30.84s/it]
 46%|████▌     | 57/125 [25:48<36:44, 32.42s/it]
 46%|████▋     | 58/125 [26:19<35:39, 31.94s/it]
 47%|████▋     | 59/125 [26:50<34:48, 31.64s/it]
 48%|████▊     | 60/125 [27:14<31:54, 29.46s/it]
 49%|████▉     | 61/125 [27:40<30:12, 28.31s/it]
 50%|████▉     | 62/125 [28:02<27:54, 26.58s/it]
 50%|█████     | 63/125 [28:29<27:25, 26.55s/it]
 51%|█████     | 64/125 [28:55<26:53, 26.46s/it]
 52%|█████▏    | 65/125 [29:37<31:11, 31.19s/it]
 53%|█████▎    | 66/125 [30:09<30:45, 31.29s/it]
 54%|█████▎    | 67/125 [30:39<29:58, 31.00s/it]
 54%|█████▍    | 68/125 [31:03<27:25, 28.87s/it]
 55%|█████▌    | 69/125 [31:18<23:04, 24.73s/it]
 56%|█████▌    | 70/125 [31:48<24:15, 26.46s/it]
 57%|█████▋    | 71/125 [32:17<24:23, 27.10s/it]
 58%|█████▊    | 72/125 [32:45<24:05, 27.28s/it]
 58%|█████▊    | 73/125 [33:11<23:24, 27.01s/it]
 59%|█████▉    | 74/125 [33:39<23:05, 27.17s/it]
 60%|██████    | 75/125 [34:15<24:52, 29.86s/it]
 61%|██████    | 76/125 [34:31<21:07, 25.87s/it]
 62%|██████▏   | 77/125 [35:05<22:39, 28.32s/it]
 62%|██████▏   | 78/125 [35:26<20:27, 26.12s/it]
 63%|██████▎   | 79/125 [35:49<19:15, 25.12s/it]
 64%|██████▍   | 80/125 [36:06<17:02, 22.72s/it]
 65%|██████▍   | 81/125 [36:34<17:43, 24.17s/it]
 66%|██████▌   | 82/125 [37:02<18:12, 25.41s/it]
 66%|██████▋   | 83/125 [37:41<20:38, 29.48s/it]
 67%|██████▋   | 84/125 [37:56<17:14, 25.23s/it]
 68%|██████▊   | 85/125 [38:09<14:19, 21.48s/it]
 69%|██████▉   | 86/125 [38:46<16:57, 26.10s/it]
 70%|██████▉   | 87/125 [39:17<17:30, 27.63s/it]
 70%|███████   | 88/125 [39:49<17:46, 28.83s/it]
 71%|███████   | 89/125 [40:27<19:02, 31.73s/it]
 72%|███████▏  | 90/125 [40:55<17:46, 30.46s/it]
 73%|███████▎  | 91/125 [41:19<16:08, 28.49s/it]
 74%|███████▎  | 92/125 [41:42<14:49, 26.95s/it]
 74%|███████▍  | 93/125 [41:59<12:40, 23.77s/it]
 75%|███████▌  | 94/125 [42:26<12:51, 24.88s/it]
 76%|███████▌  | 95/125 [42:50<12:22, 24.76s/it]
 77%|███████▋  | 96/125 [43:04<10:16, 21.27s/it]
 78%|███████▊  | 97/125 [43:39<11:55, 25.55s/it]
 78%|███████▊  | 98/125 [43:55<10:13, 22.73s/it]
 79%|███████▉  | 99/125 [44:11<08:58, 20.72s/it]
 80%|████████  | 100/125 [44:39<09:26, 22.67s/it]
 81%|████████  | 101/125 [45:15<10:41, 26.72s/it]
 82%|████████▏ | 102/125 [45:46<10:48, 28.21s/it]
 82%|████████▏ | 103/125 [46:07<09:28, 25.85s/it]
 83%|████████▎ | 104/125 [46:21<07:46, 22.23s/it]
 84%|████████▍ | 105/125 [46:37<06:48, 20.41s/it]
 85%|████████▍ | 106/125 [47:01<06:48, 21.52s/it]
 86%|████████▌ | 107/125 [47:37<07:49, 26.07s/it]
 86%|████████▋ | 108/125 [48:06<07:35, 26.80s/it]
 87%|████████▋ | 109/125 [48:26<06:37, 24.86s/it]
 88%|████████▊ | 110/125 [48:40<05:23, 21.56s/it]
 89%|████████▉ | 111/125 [48:54<04:28, 19.17s/it]
 90%|████████▉ | 112/125 [49:30<05:16, 24.31s/it]
 90%|█████████ | 113/125 [50:02<05:17, 26.50s/it]
 91%|█████████ | 114/125 [50:18<04:17, 23.37s/it]
 92%|█████████▏| 115/125 [50:37<03:40, 22.03s/it]
 93%|█████████▎| 116/125 [50:55<03:08, 20.99s/it]
 94%|█████████▎| 117/125 [51:16<02:47, 20.94s/it]
 94%|█████████▍| 118/125 [51:54<03:02, 26.13s/it]
 95%|█████████▌| 119/125 [52:25<02:44, 27.45s/it]
 96%|█████████▌| 120/125 [52:41<02:00, 24.13s/it]
 97%|█████████▋| 121/125 [53:17<01:50, 27.68s/it]
 98%|█████████▊| 122/125 [53:34<01:12, 24.33s/it]
 98%|█████████▊| 123/125 [54:01<00:50, 25.17s/it]
 99%|█████████▉| 124/125 [54:21<00:23, 23.54s/it]
100%|██████████| 125/125 [54:35<00:00, 20.78s/it]
100%|██████████| 125/125 [54:35<00:00, 26.20s/it]

  0%|          | 0/227222 [00:00<?, ?it/s]
  5%|▍         | 10583/227222 [00:00<00:02, 105827.83it/s]
 10%|▉         | 21664/227222 [00:00<00:01, 107272.38it/s]
 14%|█▍        | 32933/227222 [00:00<00:01, 108839.41it/s]
 20%|█▉        | 44422/227222 [00:00<00:01, 110584.57it/s]
 25%|██▍       | 56190/227222 [00:00<00:01, 112620.66it/s]
 30%|██▉       | 68112/227222 [00:00<00:01, 114520.04it/s]
 35%|███▌      | 79710/227222 [00:00<00:01, 114951.42it/s]
 40%|████      | 91477/227222 [00:00<00:01, 115753.26it/s]
 46%|████▌     | 103464/227222 [00:00<00:01, 116955.78it/s]
 51%|█████     | 115527/227222 [00:01<00:00, 118032.96it/s]
 56%|█████▌    | 127592/227222 [00:01<00:00, 118803.85it/s]
 62%|██████▏   | 139793/227222 [00:01<00:00, 119747.14it/s]
 67%|██████▋   | 151905/227222 [00:01<00:00, 120154.90it/s]
 72%|███████▏  | 163824/227222 [00:01<00:00, 118793.50it/s]
 77%|███████▋  | 175640/227222 [00:01<00:00, 116422.24it/s]
 83%|████████▎ | 187786/227222 [00:01<00:00, 117889.01it/s]
 88%|████████▊ | 199897/227222 [00:01<00:00, 118834.87it/s]
 93%|█████████▎| 212015/227222 [00:01<00:00, 119527.51it/s]
 99%|█████████▊| 224122/227222 [00:01<00:00, 119985.98it/s]
100%|██████████| 227222/227222 [00:01<00:00, 117424.07it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 10724/150000 [00:00<00:01, 107231.41it/s]
 14%|█▍        | 21691/150000 [00:00<00:01, 107949.65it/s]
 22%|██▏       | 33041/150000 [00:00<00:01, 109556.49it/s]
 30%|██▉       | 44766/150000 [00:00<00:00, 111755.67it/s]
 38%|███▊      | 56851/150000 [00:00<00:00, 114334.41it/s]
 46%|████▌     | 68870/150000 [00:00<00:00, 116029.92it/s]
 54%|█████▍    | 81011/150000 [00:00<00:00, 117589.90it/s]
 62%|██████▏   | 93127/150000 [00:00<00:00, 118636.59it/s]
 70%|███████   | 105316/150000 [00:00<00:00, 119592.42it/s]
 78%|███████▊  | 117616/150000 [00:01<00:00, 120591.86it/s]
 87%|████████▋ | 129933/150000 [00:01<00:00, 121350.85it/s]
 95%|█████████▍| 142194/150000 [00:01<00:00, 121722.62it/s]
100%|██████████| 150000/150000 [00:01<00:00, 118789.67it/s]

0it [00:00, ?it/s]
57it [00:00, 554.84it/s]
140it [00:00, 612.17it/s]
208it [00:00, 629.23it/s]
297it [00:00, 689.77it/s]
377it [00:00, 717.73it/s]
463it [00:00, 753.07it/s]
554it [00:00, 789.57it/s]
653it [00:00, 838.76it/s]
758it [00:00, 891.06it/s]
869it [00:01, 943.22it/s]
981it [00:01, 983.51it/s]
1098it [00:01, 994.25it/s]
1248it [00:01, 1101.11it/s]
1393it [00:01, 1183.48it/s]
1557it [00:01, 1289.51it/s]
1718it [00:01, 1366.46it/s]
1914it [00:01, 1502.78it/s]
2088it [00:01, 1566.44it/s]
2302it [00:01, 1695.99it/s]
2366it [00:01, 1202.77it/s]

  0%|          | 0/6410 [00:00<?, ?it/s]
  3%|▎         | 201/6410 [00:00<00:03, 2007.70it/s]
  7%|▋         | 465/6410 [00:00<00:02, 2158.21it/s]
 11%|█         | 713/6410 [00:00<00:02, 2237.69it/s]
 16%|█▌        | 1012/6410 [00:00<00:02, 2416.13it/s]
 21%|██▏       | 1374/6410 [00:00<00:01, 2683.83it/s]
 26%|██▌       | 1678/6410 [00:00<00:01, 2781.48it/s]
 33%|███▎      | 2140/6410 [00:00<00:01, 3158.31it/s]
 41%|████      | 2625/6410 [00:00<00:01, 3527.19it/s]
 47%|████▋     | 2998/6410 [00:00<00:00, 3570.96it/s]
 54%|█████▍    | 3475/6410 [00:01<00:00, 3857.99it/s]
 60%|██████    | 3878/6410 [00:01<00:00, 3624.84it/s]
 68%|██████▊   | 4331/6410 [00:01<00:00, 3852.02it/s]
 74%|███████▍  | 4742/6410 [00:01<00:00, 3918.54it/s]
 83%|████████▎ | 5346/6410 [00:01<00:00, 4378.85it/s]
 92%|█████████▏| 5893/6410 [00:01<00:00, 4649.55it/s]
100%|█████████▉| 6380/6410 [00:01<00:00, 4470.08it/s]
100%|██████████| 6410/6410 [00:01<00:00, 3859.22it/s]

  0%|          | 0/69931 [00:00<?, ?it/s]
  4%|▍         | 2968/69931 [00:00<00:02, 29669.84it/s]
  8%|▊         | 5245/69931 [00:00<00:02, 27194.44it/s]
 11%|█         | 7851/69931 [00:00<00:02, 26843.45it/s]
 15%|█▍        | 10372/69931 [00:00<00:02, 26331.24it/s]
 18%|█▊        | 12839/69931 [00:00<00:02, 25807.23it/s]
 22%|██▏       | 15300/69931 [00:00<00:02, 25433.28it/s]
 26%|██▌       | 17854/69931 [00:00<00:02, 25464.40it/s]
 29%|██▉       | 20166/69931 [00:00<00:02, 24673.68it/s]
 32%|███▏      | 22609/69931 [00:00<00:01, 24598.69it/s]
 36%|███▌      | 25070/69931 [00:01<00:01, 24599.22it/s]
 39%|███▉      | 27492/69931 [00:01<00:01, 24481.69it/s]
 43%|████▎     | 29967/69931 [00:01<00:01, 24561.26it/s]
 46%|████▋     | 32459/69931 [00:01<00:01, 24666.83it/s]
 50%|████▉     | 34900/69931 [00:01<00:01, 24038.73it/s]
 53%|█████▎    | 37326/69931 [00:01<00:01, 24102.11it/s]
 57%|█████▋    | 39762/69931 [00:01<00:01, 24177.74it/s]
 60%|██████    | 42265/69931 [00:01<00:01, 24426.45it/s]
 64%|██████▍   | 44720/69931 [00:01<00:01, 24463.16it/s]
 67%|██████▋   | 47164/69931 [00:01<00:00, 24037.72it/s]
 71%|███████   | 49568/69931 [00:02<00:00, 23469.12it/s]
 74%|███████▍  | 51918/69931 [00:02<00:00, 23083.81it/s]
 78%|███████▊  | 54230/69931 [00:02<00:00, 22650.40it/s]
 81%|████████  | 56500/69931 [00:02<00:00, 22137.41it/s]
 84%|████████▍ | 58757/69931 [00:02<00:00, 22265.32it/s]
 87%|████████▋ | 61093/69931 [00:02<00:00, 22581.38it/s]
 91%|█████████ | 63654/69931 [00:02<00:00, 23411.95it/s]
 95%|█████████▍| 66130/69931 [00:02<00:00, 23798.82it/s]
 98%|█████████▊| 68760/69931 [00:02<00:00, 24495.83it/s]
100%|██████████| 69931/69931 [00:02<00:00, 24224.18it/s]

  0%|          | 0/69931 [00:00<?, ?it/s]
  8%|▊         | 5267/69931 [00:00<00:01, 50008.71it/s]
 14%|█▍        | 9627/69931 [00:00<00:01, 46794.84it/s]
 20%|██        | 14243/69931 [00:00<00:01, 44536.68it/s]
 27%|██▋       | 18789/69931 [00:00<00:01, 44809.12it/s]
 32%|███▏      | 22544/69931 [00:00<00:01, 42349.99it/s]
 37%|███▋      | 26147/69931 [00:00<00:01, 40231.07it/s]
 43%|████▎     | 29763/69931 [00:00<00:01, 38915.43it/s]
 48%|████▊     | 33444/69931 [00:00<00:00, 38257.64it/s]
 53%|█████▎    | 37064/69931 [00:00<00:00, 37613.78it/s]
 58%|█████▊    | 40736/69931 [00:01<00:00, 37338.38it/s]
 64%|██████▎   | 44409/69931 [00:01<00:00, 37152.83it/s]
 69%|██████▊   | 48041/69931 [00:01<00:00, 36361.86it/s]
 74%|███████▍  | 51738/69931 [00:01<00:00, 35589.60it/s]
 80%|████████  | 56034/69931 [00:01<00:00, 36101.07it/s]
 87%|████████▋ | 60586/69931 [00:01<00:00, 38488.74it/s]
 92%|█████████▏| 64600/69931 [00:01<00:00, 38966.55it/s]
 98%|█████████▊| 68644/69931 [00:01<00:00, 39395.96it/s]
100%|██████████| 69931/69931 [00:01<00:00, 39238.71it/s]

  0%|          | 0/2366 [00:00<?, ?it/s]
100%|██████████| 2366/2366 [00:00<00:00, 78291.20it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.25it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.20it/s]
  3%|▎         | 3/100 [00:00<00:13,  7.07it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.82it/s]
  5%|▌         | 5/100 [00:00<00:13,  6.91it/s]
  6%|▌         | 6/100 [00:00<00:13,  6.93it/s]
  7%|▋         | 7/100 [00:01<00:13,  6.88it/s]
  8%|▊         | 8/100 [00:01<00:13,  6.64it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.76it/s]
 10%|█         | 10/100 [00:01<00:13,  6.65it/s]
 11%|█         | 11/100 [00:01<00:12,  6.85it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.66it/s]
 13%|█▎        | 13/100 [00:01<00:12,  6.85it/s]
 14%|█▍        | 14/100 [00:02<00:12,  6.82it/s]
 15%|█▌        | 15/100 [00:02<00:12,  6.57it/s]
 16%|█▌        | 16/100 [00:02<00:12,  6.53it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.68it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.66it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.64it/s]
 20%|██        | 20/100 [00:02<00:11,  6.71it/s]
 21%|██        | 21/100 [00:03<00:11,  6.82it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.91it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.75it/s]
 24%|██▍       | 24/100 [00:03<00:11,  6.45it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.56it/s]
 26%|██▌       | 26/100 [00:03<00:11,  6.26it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.44it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.24it/s]
 29%|██▉       | 29/100 [00:04<00:10,  6.47it/s]
 30%|███       | 30/100 [00:04<00:10,  6.67it/s]
 31%|███       | 31/100 [00:04<00:10,  6.46it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.67it/s]
 33%|███▎      | 33/100 [00:04<00:09,  6.82it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.97it/s]
 35%|███▌      | 35/100 [00:05<00:09,  7.00it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.88it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.73it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.81it/s]
 39%|███▉      | 39/100 [00:05<00:09,  6.70it/s]
 40%|████      | 40/100 [00:05<00:08,  6.84it/s]
 41%|████      | 41/100 [00:06<00:08,  6.82it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.91it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.97it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.86it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.37it/s]
 46%|████▌     | 46/100 [00:06<00:08,  6.57it/s]
 47%|████▋     | 47/100 [00:07<00:07,  6.67it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.72it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.79it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.85it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.94it/s]
 52%|█████▏    | 52/100 [00:07<00:07,  6.66it/s]
 53%|█████▎    | 53/100 [00:07<00:07,  6.32it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.18it/s]
 55%|█████▌    | 55/100 [00:08<00:07,  6.17it/s]
 56%|█████▌    | 56/100 [00:08<00:07,  6.19it/s]
 57%|█████▋    | 57/100 [00:08<00:07,  5.94it/s]
 58%|█████▊    | 58/100 [00:08<00:07,  5.92it/s]
 59%|█████▉    | 59/100 [00:08<00:06,  6.03it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.03it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.29it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.11it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.03it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.20it/s]
 65%|██████▌   | 65/100 [00:09<00:05,  6.34it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.60it/s]
 67%|██████▋   | 67/100 [00:10<00:04,  6.62it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.75it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.58it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.62it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.64it/s]
 72%|███████▏  | 72/100 [00:10<00:04,  6.67it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.19it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.46it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.70it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.67it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.59it/s]
 78%|███████▊  | 78/100 [00:11<00:03,  6.52it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.45it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.28it/s]
 81%|████████  | 81/100 [00:12<00:02,  6.35it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.62it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.72it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.55it/s]
 85%|████████▌ | 85/100 [00:12<00:02,  6.79it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.89it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  7.01it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.79it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.67it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.60it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.56it/s]
 92%|█████████▏| 92/100 [00:13<00:01,  6.42it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.42it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.23it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.41it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.74it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.64it/s]
 98%|█████████▊| 98/100 [00:14<00:00,  6.87it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.60it/s]
100%|██████████| 100/100 [00:15<00:00,  6.53it/s]
100%|██████████| 100/100 [00:15<00:00,  6.58it/s]

  0%|          | 0/69931 [00:00<?, ?it/s]
  6%|▌         | 4349/69931 [00:00<00:01, 43489.42it/s]
 14%|█▍        | 9838/69931 [00:00<00:01, 46377.79it/s]
 24%|██▍       | 17126/69931 [00:00<00:01, 52055.83it/s]
 35%|███▌      | 24578/69931 [00:00<00:00, 57230.54it/s]
 46%|████▋     | 32350/69931 [00:00<00:00, 62144.59it/s]
 58%|█████▊    | 40253/69931 [00:00<00:00, 66399.68it/s]
 69%|██████▉   | 48565/69931 [00:00<00:00, 70662.52it/s]
 82%|████████▏ | 57550/69931 [00:00<00:00, 75498.87it/s]
 96%|█████████▌| 67228/69931 [00:00<00:00, 80828.46it/s]
100%|██████████| 69931/69931 [00:00<00:00, 75477.30it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/317164 [00:00<?, ?it/s]
  3%|▎         | 10859/317164 [00:00<00:02, 108587.00it/s]
  7%|▋         | 21935/317164 [00:00<00:02, 109227.15it/s]
 10%|█         | 33247/317164 [00:00<00:02, 110365.53it/s]
 14%|█▍        | 44873/317164 [00:00<00:02, 112067.70it/s]
 18%|█▊        | 56475/317164 [00:00<00:02, 113223.20it/s]
 22%|██▏       | 68369/317164 [00:00<00:02, 114876.55it/s]
 25%|██▌       | 80327/317164 [00:00<00:02, 116247.49it/s]
 29%|██▉       | 92399/317164 [00:00<00:01, 117551.47it/s]
 33%|███▎      | 104360/317164 [00:00<00:01, 118160.67it/s]
 37%|███▋      | 116510/317164 [00:01<00:01, 119142.73it/s]
 41%|████      | 128694/317164 [00:01<00:01, 119938.73it/s]
 44%|████▍     | 140951/317164 [00:01<00:01, 120715.61it/s]
 48%|████▊     | 153151/317164 [00:01<00:01, 121098.03it/s]
 52%|█████▏    | 165460/317164 [00:01<00:01, 121687.22it/s]
 56%|█████▌    | 177877/317164 [00:01<00:01, 122420.30it/s]
 60%|█████▉    | 190072/317164 [00:01<00:01, 121748.86it/s]
 64%|██████▍   | 202347/317164 [00:01<00:00, 122046.47it/s]
 68%|██████▊   | 214530/317164 [00:01<00:00, 121409.12it/s]
 72%|███████▏  | 226867/317164 [00:01<00:00, 121988.86it/s]
 76%|███████▌  | 239580/317164 [00:02<00:00, 123486.09it/s]
 79%|███████▉  | 251927/317164 [00:02<00:00, 123173.10it/s]
 83%|████████▎ | 264243/317164 [00:02<00:00, 122683.99it/s]
 87%|████████▋ | 276916/317164 [00:02<00:00, 123869.32it/s]
 91%|█████████ | 289306/317164 [00:02<00:00, 122815.21it/s]
 95%|█████████▌| 302021/317164 [00:02<00:00, 124081.99it/s]
 99%|█████████▉| 314435/317164 [00:02<00:00, 122548.72it/s]
100%|██████████| 317164/317164 [00:02<00:00, 120393.29it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6277/150000 [00:00<00:02, 62769.61it/s]
  9%|▊         | 12764/150000 [00:00<00:02, 63385.06it/s]
 13%|█▎        | 19349/150000 [00:00<00:02, 64104.16it/s]
 17%|█▋        | 25934/150000 [00:00<00:01, 64616.18it/s]
 22%|██▏       | 32589/150000 [00:00<00:01, 65181.38it/s]
 26%|██▌       | 39265/150000 [00:00<00:01, 65645.06it/s]
 31%|███       | 45953/150000 [00:00<00:01, 66008.55it/s]
 35%|███▌      | 52716/150000 [00:00<00:01, 66484.21it/s]
 40%|███▉      | 59438/150000 [00:00<00:01, 66701.76it/s]
 44%|████▍     | 66188/150000 [00:01<00:01, 66939.16it/s]
 49%|████▊     | 72905/150000 [00:01<00:01, 67005.30it/s]
 53%|█████▎    | 79672/150000 [00:01<00:01, 67202.02it/s]
 58%|█████▊    | 86457/150000 [00:01<00:00, 67394.84it/s]
 62%|██████▏   | 93238/150000 [00:01<00:00, 67516.90it/s]
 67%|██████▋   | 99995/150000 [00:01<00:00, 67530.67it/s]
 71%|███████   | 106779/150000 [00:01<00:00, 67621.89it/s]
 76%|███████▌  | 113576/150000 [00:01<00:00, 67724.86it/s]
 80%|████████  | 120374/150000 [00:01<00:00, 67798.46it/s]
 85%|████████▍ | 127144/150000 [00:01<00:00, 67652.56it/s]
 89%|████████▉ | 133903/150000 [00:02<00:00, 67580.98it/s]
 94%|█████████▍| 140668/150000 [00:02<00:00, 67600.97it/s]
 98%|█████████▊| 147425/150000 [00:02<00:00, 67586.08it/s]
100%|██████████| 150000/150000 [00:02<00:00, 66992.09it/s]

  0%|          | 0/134 [00:00<?, ?it/s]
 2366 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-20 ...
Successfully removed 6410 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-21 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 11971 new domains found

Scrapping urls...
scrapping will be done in 134 phase  where 90 urls attempt in each phase

  1%|          | 1/134 [00:11<25:40, 11.59s/it]
  1%|▏         | 2/134 [00:31<31:18, 14.23s/it]
  2%|▏         | 3/134 [00:44<29:53, 13.69s/it]
  3%|▎         | 4/134 [01:09<36:58, 17.07s/it]
  4%|▎         | 5/134 [01:32<40:36, 18.89s/it]
  4%|▍         | 6/134 [02:06<50:08, 23.51s/it]
  5%|▌         | 7/134 [02:29<49:17, 23.29s/it]
  6%|▌         | 8/134 [02:57<51:53, 24.71s/it]
  7%|▋         | 9/134 [03:54<1:11:50, 34.49s/it]
  7%|▋         | 10/134 [04:20<1:05:48, 31.84s/it]
  8%|▊         | 11/134 [04:33<53:30, 26.10s/it]  
  9%|▉         | 12/134 [04:44<43:57, 21.62s/it]
 10%|▉         | 13/134 [05:04<42:56, 21.29s/it]
 10%|█         | 14/134 [05:32<46:29, 23.25s/it]
 11%|█         | 15/134 [06:06<52:21, 26.40s/it]
 12%|█▏        | 16/134 [06:32<51:40, 26.27s/it]
 13%|█▎        | 17/134 [06:43<42:24, 21.75s/it]
 13%|█▎        | 18/134 [06:47<31:47, 16.45s/it]
 14%|█▍        | 19/134 [06:51<24:01, 12.54s/it]
 15%|█▍        | 20/134 [07:20<33:20, 17.54s/it]
 16%|█▌        | 21/134 [07:30<28:49, 15.31s/it]
 16%|█▋        | 22/134 [07:35<22:47, 12.21s/it]
 17%|█▋        | 23/134 [07:46<21:54, 11.84s/it]
 18%|█▊        | 24/134 [07:49<16:46,  9.15s/it]
 19%|█▊        | 25/134 [07:52<13:10,  7.26s/it]
 19%|█▉        | 26/134 [07:58<12:29,  6.94s/it]
 20%|██        | 27/134 [08:01<10:34,  5.93s/it]
 21%|██        | 28/134 [08:04<08:46,  4.96s/it]
 22%|██▏       | 29/134 [08:15<11:56,  6.83s/it]
 22%|██▏       | 30/134 [08:41<21:43, 12.53s/it]
 23%|██▎       | 31/134 [08:52<20:27, 11.92s/it]
 24%|██▍       | 32/134 [09:24<30:36, 18.01s/it]
 25%|██▍       | 33/134 [09:35<26:39, 15.84s/it]
 25%|██▌       | 34/134 [09:46<24:03, 14.43s/it]
 26%|██▌       | 35/134 [09:56<21:57, 13.31s/it]
 27%|██▋       | 36/134 [10:19<26:20, 16.13s/it]
 28%|██▊       | 37/134 [10:31<24:06, 14.91s/it]
 28%|██▊       | 38/134 [10:43<22:25, 14.01s/it]
 29%|██▉       | 39/134 [11:10<28:05, 17.74s/it]
 30%|██▉       | 40/134 [11:35<31:23, 20.04s/it]
 31%|███       | 41/134 [12:09<37:45, 24.36s/it]
 31%|███▏      | 42/134 [12:38<39:27, 25.73s/it]
 32%|███▏      | 43/134 [12:52<33:22, 22.01s/it]
 33%|███▎      | 44/134 [13:06<29:41, 19.79s/it]
 34%|███▎      | 45/134 [13:41<35:55, 24.21s/it]
 34%|███▍      | 46/134 [14:03<34:45, 23.70s/it]
 35%|███▌      | 47/134 [14:40<39:59, 27.58s/it]
 36%|███▌      | 48/134 [14:50<32:05, 22.39s/it]
 37%|███▋      | 49/134 [15:01<26:53, 18.99s/it]
 37%|███▋      | 50/134 [15:12<23:16, 16.62s/it]
 38%|███▊      | 51/134 [15:20<19:14, 13.91s/it]
 39%|███▉      | 52/134 [15:51<26:03, 19.07s/it]
 40%|███▉      | 53/134 [16:19<29:30, 21.86s/it]
 40%|████      | 54/134 [16:53<33:42, 25.28s/it]
 41%|████      | 55/134 [17:07<28:55, 21.97s/it]
 42%|████▏     | 56/134 [17:16<23:23, 18.00s/it]
 43%|████▎     | 57/134 [17:36<23:59, 18.69s/it]
 43%|████▎     | 58/134 [17:48<21:17, 16.81s/it]
 44%|████▍     | 59/134 [17:59<18:33, 14.85s/it]
 45%|████▍     | 60/134 [18:10<16:50, 13.66s/it]
 46%|████▌     | 61/134 [18:18<14:49, 12.18s/it]
 46%|████▋     | 62/134 [18:30<14:28, 12.07s/it]
 47%|████▋     | 63/134 [18:59<20:14, 17.10s/it]
 48%|████▊     | 64/134 [19:34<26:15, 22.51s/it]
 49%|████▊     | 65/134 [19:43<21:21, 18.57s/it]
 49%|████▉     | 66/134 [20:12<24:30, 21.63s/it]
 50%|█████     | 67/134 [20:39<25:42, 23.03s/it]
 51%|█████     | 68/134 [21:08<27:31, 25.02s/it]
 51%|█████▏    | 69/134 [21:32<26:41, 24.64s/it]
 52%|█████▏    | 70/134 [21:46<22:48, 21.38s/it]
 53%|█████▎    | 71/134 [22:12<23:55, 22.79s/it]
 54%|█████▎    | 72/134 [22:47<27:19, 26.44s/it]
 54%|█████▍    | 73/134 [23:29<31:43, 31.21s/it]
 55%|█████▌    | 74/134 [23:46<27:01, 27.03s/it]
 56%|█████▌    | 75/134 [24:06<24:14, 24.65s/it]
 57%|█████▋    | 76/134 [24:29<23:28, 24.29s/it]
 57%|█████▋    | 77/134 [25:01<25:24, 26.75s/it]
 58%|█████▊    | 78/134 [25:27<24:39, 26.42s/it]
 59%|█████▉    | 79/134 [25:53<24:06, 26.30s/it]
 60%|█████▉    | 80/134 [26:19<23:37, 26.25s/it]
 60%|██████    | 81/134 [26:53<25:03, 28.36s/it]
 61%|██████    | 82/134 [27:11<22:04, 25.46s/it]
 62%|██████▏   | 83/134 [27:28<19:29, 22.94s/it]
 63%|██████▎   | 84/134 [27:45<17:32, 21.05s/it]
 63%|██████▎   | 85/134 [28:05<16:55, 20.73s/it]
 64%|██████▍   | 86/134 [28:23<15:52, 19.84s/it]
 65%|██████▍   | 87/134 [28:37<14:20, 18.30s/it]
 66%|██████▌   | 88/134 [29:06<16:20, 21.32s/it]
 66%|██████▋   | 89/134 [29:28<16:17, 21.73s/it]
 67%|██████▋   | 90/134 [29:44<14:30, 19.77s/it]
 68%|██████▊   | 91/134 [30:23<18:26, 25.73s/it]
 69%|██████▊   | 92/134 [30:44<16:53, 24.12s/it]
 69%|██████▉   | 93/134 [30:58<14:31, 21.24s/it]
 70%|███████   | 94/134 [31:13<12:52, 19.30s/it]
 71%|███████   | 95/134 [31:46<15:11, 23.38s/it]
 72%|███████▏  | 96/134 [32:02<13:26, 21.21s/it]
 72%|███████▏  | 97/134 [32:26<13:40, 22.17s/it]
 73%|███████▎  | 98/134 [33:08<16:42, 27.85s/it]
 74%|███████▍  | 99/134 [33:36<16:26, 28.17s/it]
 75%|███████▍  | 100/134 [33:50<13:29, 23.81s/it]
 75%|███████▌  | 101/134 [34:05<11:37, 21.15s/it]
 76%|███████▌  | 102/134 [34:24<10:53, 20.41s/it]
 77%|███████▋  | 103/134 [34:38<09:39, 18.69s/it]
 78%|███████▊  | 104/134 [34:51<08:25, 16.85s/it]
 78%|███████▊  | 105/134 [35:03<07:24, 15.34s/it]
 79%|███████▉  | 106/134 [35:14<06:34, 14.10s/it]
 80%|███████▉  | 107/134 [35:42<08:13, 18.28s/it]
 81%|████████  | 108/134 [36:12<09:25, 21.76s/it]
 81%|████████▏ | 109/134 [36:33<08:55, 21.43s/it]
 82%|████████▏ | 110/134 [37:02<09:34, 23.92s/it]
 83%|████████▎ | 111/134 [37:23<08:47, 22.94s/it]
 84%|████████▎ | 112/134 [37:37<07:23, 20.16s/it]
 84%|████████▍ | 113/134 [37:50<06:22, 18.22s/it]
 85%|████████▌ | 114/134 [38:07<05:54, 17.73s/it]
 86%|████████▌ | 115/134 [38:25<05:37, 17.77s/it]
 87%|████████▋ | 116/134 [38:47<05:46, 19.24s/it]
 87%|████████▋ | 117/134 [39:21<06:40, 23.54s/it]
 88%|████████▊ | 118/134 [39:52<06:52, 25.80s/it]
 89%|████████▉ | 119/134 [40:05<05:30, 22.07s/it]
 90%|████████▉ | 120/134 [40:17<04:24, 18.87s/it]
 90%|█████████ | 121/134 [40:53<05:12, 24.06s/it]
 91%|█████████ | 122/134 [41:22<05:05, 25.42s/it]
 92%|█████████▏| 123/134 [42:00<05:21, 29.27s/it]
 93%|█████████▎| 124/134 [42:21<04:29, 26.93s/it]
 93%|█████████▎| 125/134 [42:55<04:19, 28.85s/it]
 94%|█████████▍| 126/134 [43:15<03:30, 26.30s/it]
 95%|█████████▍| 127/134 [43:41<03:04, 26.36s/it]
 96%|█████████▌| 128/134 [44:13<02:47, 27.93s/it]
 96%|█████████▋| 129/134 [44:33<02:07, 25.47s/it]
 97%|█████████▋| 130/134 [44:47<01:28, 22.07s/it]
 98%|█████████▊| 131/134 [45:12<01:08, 22.91s/it]
 99%|█████████▊| 132/134 [45:26<00:40, 20.19s/it]
 99%|█████████▉| 133/134 [45:37<00:17, 17.40s/it]
100%|██████████| 134/134 [45:38<00:00, 12.61s/it]
100%|██████████| 134/134 [45:38<00:00, 20.44s/it]

  0%|          | 0/227859 [00:00<?, ?it/s]
  5%|▍         | 11037/227859 [00:00<00:01, 110359.58it/s]
 10%|▉         | 22324/227859 [00:00<00:01, 111099.76it/s]
 15%|█▍        | 33840/227859 [00:00<00:01, 112286.07it/s]
 20%|█▉        | 45493/227859 [00:00<00:01, 113523.62it/s]
 25%|██▌       | 57439/227859 [00:00<00:01, 115241.60it/s]
 31%|███       | 69518/227859 [00:00<00:01, 116850.51it/s]
 36%|███▌      | 81346/227859 [00:00<00:01, 117275.16it/s]
 41%|████      | 93208/227859 [00:00<00:01, 117675.19it/s]
 46%|████▋     | 105439/227859 [00:00<00:01, 119026.04it/s]
 52%|█████▏    | 117665/227859 [00:01<00:00, 119975.79it/s]
 57%|█████▋    | 129950/227859 [00:01<00:00, 120822.93it/s]
 62%|██████▏   | 142224/227859 [00:01<00:00, 121391.00it/s]
 68%|██████▊   | 154422/227859 [00:01<00:00, 121566.27it/s]
 73%|███████▎  | 166476/227859 [00:01<00:00, 120494.53it/s]
 78%|███████▊  | 178456/227859 [00:01<00:00, 118079.81it/s]
 84%|████████▎ | 190818/227859 [00:01<00:00, 119686.98it/s]
 89%|████████▉ | 203181/227859 [00:01<00:00, 120840.72it/s]
 95%|█████████▍| 215547/227859 [00:01<00:00, 121672.78it/s]
100%|██████████| 227859/227859 [00:01<00:00, 119421.94it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11106/150000 [00:00<00:01, 111056.93it/s]
 15%|█▍        | 22380/150000 [00:00<00:01, 111555.00it/s]
 23%|██▎       | 33983/150000 [00:00<00:01, 112858.35it/s]
 30%|███       | 45702/150000 [00:00<00:00, 114122.69it/s]
 38%|███▊      | 57716/150000 [00:00<00:00, 115863.10it/s]
 47%|████▋     | 69847/150000 [00:00<00:00, 117443.27it/s]
 55%|█████▍    | 81968/150000 [00:00<00:00, 118548.18it/s]
 63%|██████▎   | 94184/150000 [00:00<00:00, 119608.96it/s]
 71%|███████   | 106516/150000 [00:00<00:00, 120695.77it/s]
 79%|███████▉  | 118871/150000 [00:01<00:00, 121526.50it/s]
 88%|████████▊ | 131261/150000 [00:01<00:00, 122225.36it/s]
 96%|█████████▌| 143670/150000 [00:01<00:00, 122778.01it/s]
100%|██████████| 150000/150000 [00:01<00:00, 119845.22it/s]

0it [00:00, ?it/s]
138it [00:00, 1379.04it/s]
344it [00:00, 1530.11it/s]
532it [00:00, 1615.90it/s]
734it [00:00, 1696.02it/s]
1022it [00:00, 1934.47it/s]
1270it [00:00, 2068.51it/s]
1568it [00:00, 2273.93it/s]
2022it [00:00, 2671.49it/s]
2040it [00:00, 2518.04it/s]

  0%|          | 0/1403 [00:00<?, ?it/s]
 71%|███████▏  | 1001/1403 [00:00<00:00, 10009.77it/s]
100%|██████████| 1403/1403 [00:00<00:00, 10774.49it/s]

  0%|          | 0/66552 [00:00<?, ?it/s]
  3%|▎         | 2124/66552 [00:00<00:03, 21156.69it/s]
  6%|▋         | 4310/66552 [00:00<00:02, 21360.80it/s]
 10%|█         | 6788/66552 [00:00<00:02, 22281.98it/s]
 14%|█▍        | 9282/66552 [00:00<00:02, 23016.15it/s]
 17%|█▋        | 11618/66552 [00:00<00:02, 23116.62it/s]
 20%|██        | 13584/66552 [00:00<00:03, 17408.10it/s]
 23%|██▎       | 15292/66552 [00:00<00:04, 11150.99it/s]
 27%|██▋       | 17725/66552 [00:01<00:03, 13313.99it/s]
 30%|███       | 20152/66552 [00:01<00:03, 15398.38it/s]
 33%|███▎      | 22066/66552 [00:01<00:03, 13958.09it/s]
 37%|███▋      | 24359/66552 [00:01<00:02, 15813.36it/s]
 40%|███▉      | 26546/66552 [00:01<00:02, 17243.70it/s]
 43%|████▎     | 28722/66552 [00:01<00:02, 18387.56it/s]
 46%|████▋     | 30880/66552 [00:01<00:01, 19240.16it/s]
 50%|████▉     | 33035/66552 [00:01<00:01, 19877.37it/s]
 53%|█████▎    | 35321/66552 [00:01<00:01, 20686.33it/s]
 57%|█████▋    | 37743/66552 [00:02<00:01, 21630.65it/s]
 60%|██████    | 40137/66552 [00:02<00:01, 22274.69it/s]
 64%|██████▍   | 42567/66552 [00:02<00:01, 22844.53it/s]
 68%|██████▊   | 44950/66552 [00:02<00:00, 23130.52it/s]
 71%|███████   | 47383/66552 [00:02<00:00, 23477.33it/s]
 75%|███████▍  | 49870/66552 [00:02<00:00, 23876.77it/s]
 79%|███████▊  | 52349/66552 [00:02<00:00, 24142.21it/s]
 82%|████████▏ | 54788/66552 [00:02<00:00, 24212.64it/s]
 86%|████████▌ | 57269/66552 [00:02<00:00, 24386.82it/s]
 90%|████████▉ | 59715/66552 [00:02<00:00, 24197.30it/s]
 93%|█████████▎| 62200/66552 [00:03<00:00, 24389.34it/s]
 97%|█████████▋| 64763/66552 [00:03<00:00, 24747.33it/s]
100%|██████████| 66552/66552 [00:03<00:00, 20746.81it/s]

  0%|          | 0/66552 [00:00<?, ?it/s]
  9%|▊         | 5794/66552 [00:00<00:01, 52805.54it/s]
 16%|█▌        | 10515/66552 [00:00<00:01, 48653.22it/s]
 23%|██▎       | 15241/66552 [00:00<00:01, 46679.59it/s]
 30%|██▉       | 19948/66552 [00:00<00:00, 46795.48it/s]
 36%|███▌      | 23858/66552 [00:00<00:00, 44183.77it/s]
 42%|████▏     | 27667/66552 [00:00<00:00, 42158.06it/s]
 47%|████▋     | 31410/66552 [00:00<00:00, 40618.28it/s]
 53%|█████▎    | 35128/66552 [00:00<00:00, 39519.04it/s]
 58%|█████▊    | 38821/66552 [00:00<00:00, 38541.25it/s]
 64%|██████▍   | 42557/66552 [00:01<00:00, 38175.99it/s]
 69%|██████▉   | 46253/66552 [00:01<00:00, 37776.77it/s]
 75%|███████▌  | 49946/66552 [00:01<00:00, 37414.44it/s]
 81%|████████  | 53727/66552 [00:01<00:00, 37531.15it/s]
 87%|████████▋ | 57621/66552 [00:01<00:00, 37941.01it/s]
 93%|█████████▎| 61729/66552 [00:01<00:00, 38829.57it/s]
100%|██████████| 66552/66552 [00:01<00:00, 40779.09it/s]

  0%|          | 0/2040 [00:00<?, ?it/s]
100%|██████████| 2040/2040 [00:00<00:00, 79473.73it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.27it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.25it/s]
  3%|▎         | 3/100 [00:00<00:13,  7.12it/s]
  4%|▍         | 4/100 [00:00<00:13,  6.91it/s]
  5%|▌         | 5/100 [00:00<00:13,  7.00it/s]
  6%|▌         | 6/100 [00:00<00:13,  7.05it/s]
  7%|▋         | 7/100 [00:01<00:13,  6.98it/s]
  8%|▊         | 8/100 [00:01<00:13,  6.60it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.81it/s]
 10%|█         | 10/100 [00:01<00:13,  6.79it/s]
 11%|█         | 11/100 [00:01<00:12,  6.94it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.71it/s]
 13%|█▎        | 13/100 [00:01<00:12,  6.86it/s]
 14%|█▍        | 14/100 [00:02<00:12,  6.85it/s]
 15%|█▌        | 15/100 [00:02<00:12,  6.57it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.45it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.63it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.65it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.65it/s]
 20%|██        | 20/100 [00:02<00:12,  6.66it/s]
 21%|██        | 21/100 [00:03<00:11,  6.68it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.78it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.60it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.32it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.40it/s]
 26%|██▌       | 26/100 [00:03<00:12,  6.12it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.32it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.13it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.36it/s]
 30%|███       | 30/100 [00:04<00:10,  6.63it/s]
 31%|███       | 31/100 [00:04<00:10,  6.38it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.57it/s]
 33%|███▎      | 33/100 [00:04<00:09,  6.72it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.83it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.80it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.65it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.48it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.59it/s]
 39%|███▉      | 39/100 [00:05<00:09,  6.55it/s]
 40%|████      | 40/100 [00:06<00:08,  6.68it/s]
 41%|████      | 41/100 [00:06<00:08,  6.62it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.72it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.76it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.89it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.37it/s]
 46%|████▌     | 46/100 [00:06<00:08,  6.49it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.59it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.59it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.63it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.67it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.82it/s]
 52%|█████▏    | 52/100 [00:07<00:07,  6.76it/s]
 53%|█████▎    | 53/100 [00:07<00:07,  6.60it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.55it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.59it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.66it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.40it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.36it/s]
 59%|█████▉    | 59/100 [00:08<00:06,  6.37it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.30it/s]
 61%|██████    | 61/100 [00:09<00:05,  6.53it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.29it/s]
 63%|██████▎   | 63/100 [00:09<00:05,  6.29it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.49it/s]
 65%|██████▌   | 65/100 [00:09<00:05,  6.56it/s]
 66%|██████▌   | 66/100 [00:09<00:04,  6.80it/s]
 67%|██████▋   | 67/100 [00:10<00:04,  6.75it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.94it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.81it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.87it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.84it/s]
 72%|███████▏  | 72/100 [00:10<00:04,  6.86it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.35it/s]
 74%|███████▍  | 74/100 [00:11<00:03,  6.57it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.81it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.85it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.76it/s]
 78%|███████▊  | 78/100 [00:11<00:03,  6.72it/s]
 79%|███████▉  | 79/100 [00:11<00:03,  6.65it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.43it/s]
 81%|████████  | 81/100 [00:12<00:02,  6.51it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.78it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.91it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.72it/s]
 85%|████████▌ | 85/100 [00:12<00:02,  6.94it/s]
 86%|████████▌ | 86/100 [00:12<00:01,  7.02it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.80it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.61it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.65it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.67it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.71it/s]
 92%|█████████▏| 92/100 [00:13<00:01,  6.57it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.62it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.42it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.41it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.38it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.35it/s]
 98%|█████████▊| 98/100 [00:14<00:00,  6.63it/s]
 99%|█████████▉| 99/100 [00:14<00:00,  6.32it/s]
100%|██████████| 100/100 [00:15<00:00,  6.16it/s]
100%|██████████| 100/100 [00:15<00:00,  6.61it/s]

  0%|          | 0/66552 [00:00<?, ?it/s]
  6%|▋         | 4175/66552 [00:00<00:01, 41746.56it/s]
 14%|█▍        | 9355/66552 [00:00<00:01, 44326.35it/s]
 25%|██▍       | 16399/66552 [00:00<00:01, 49871.93it/s]
 36%|███▌      | 23780/66552 [00:00<00:00, 55245.63it/s]
 47%|████▋     | 31372/66552 [00:00<00:00, 60160.10it/s]
 59%|█████▉    | 39214/66552 [00:00<00:00, 64677.51it/s]
 71%|███████   | 47378/66552 [00:00<00:00, 68975.46it/s]
 84%|████████▍ | 56157/66552 [00:00<00:00, 73714.68it/s]
 99%|█████████▉| 65894/66552 [00:00<00:00, 79509.11it/s]
100%|██████████| 66552/66552 [00:00<00:00, 73404.27it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/299778 [00:00<?, ?it/s]
  4%|▎         | 11043/299778 [00:00<00:02, 110429.84it/s]
  7%|▋         | 22348/299778 [00:00<00:02, 111200.42it/s]
 11%|█▏        | 33777/299778 [00:00<00:02, 112107.60it/s]
 15%|█▌        | 45589/299778 [00:00<00:02, 113843.34it/s]
 19%|█▉        | 57691/299778 [00:00<00:02, 115903.31it/s]
 23%|██▎       | 69955/299778 [00:00<00:01, 117843.80it/s]
 27%|██▋       | 82209/299778 [00:00<00:01, 119210.64it/s]
 31%|███▏      | 94420/299778 [00:00<00:01, 120064.91it/s]
 36%|███▌      | 106619/299778 [00:00<00:01, 120633.16it/s]
 40%|███▉      | 118961/299778 [00:01<00:01, 121453.89it/s]
 44%|████▍     | 131329/299778 [00:01<00:01, 122110.88it/s]
 48%|████▊     | 143697/299778 [00:01<00:01, 122576.59it/s]
 52%|█████▏    | 156212/299778 [00:01<00:01, 123337.03it/s]
 56%|█████▌    | 168444/299778 [00:01<00:01, 122790.26it/s]
 60%|██████    | 180901/299778 [00:01<00:00, 123318.42it/s]
 65%|██████▍   | 193382/299778 [00:01<00:00, 123762.08it/s]
 69%|██████▊   | 205941/299778 [00:01<00:00, 124302.33it/s]
 73%|███████▎  | 218348/299778 [00:01<00:00, 122844.07it/s]
 77%|███████▋  | 230620/299778 [00:01<00:00, 122586.54it/s]
 81%|████████  | 242927/299778 [00:02<00:00, 122730.78it/s]
 85%|████████▌ | 255839/299778 [00:02<00:00, 124579.58it/s]
 89%|████████▉ | 268300/299778 [00:02<00:00, 123679.85it/s]
 94%|█████████▍| 281194/299778 [00:02<00:00, 125210.22it/s]
 98%|█████████▊| 293722/299778 [00:02<00:00, 123328.59it/s]
100%|██████████| 299778/299778 [00:02<00:00, 121880.25it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6293/150000 [00:00<00:02, 62920.61it/s]
  9%|▊         | 12758/150000 [00:00<00:02, 63427.34it/s]
 13%|█▎        | 19312/150000 [00:00<00:02, 64044.34it/s]
 17%|█▋        | 25905/150000 [00:00<00:01, 64597.32it/s]
 22%|██▏       | 32610/150000 [00:00<00:01, 65313.64it/s]
 26%|██▌       | 39330/150000 [00:00<00:01, 65865.67it/s]
 31%|███       | 46156/150000 [00:00<00:01, 66564.08it/s]
 35%|███▌      | 53056/150000 [00:00<00:01, 67273.52it/s]
 40%|███▉      | 59970/150000 [00:00<00:01, 67820.63it/s]
 45%|████▍     | 66842/150000 [00:01<00:01, 68085.47it/s]
 49%|████▉     | 73732/150000 [00:01<00:01, 68325.88it/s]
 54%|█████▎    | 80623/150000 [00:01<00:01, 68497.37it/s]
 58%|█████▊    | 87535/150000 [00:01<00:00, 68681.09it/s]
 63%|██████▎   | 94411/150000 [00:01<00:00, 68702.80it/s]
 68%|██████▊   | 101313/150000 [00:01<00:00, 68795.56it/s]
 72%|███████▏  | 108243/150000 [00:01<00:00, 68944.82it/s]
 77%|███████▋  | 115166/150000 [00:01<00:00, 69028.37it/s]
 81%|████████▏ | 122137/150000 [00:01<00:00, 69229.95it/s]
 86%|████████▌ | 129051/150000 [00:01<00:00, 68991.96it/s]
 91%|█████████ | 135944/150000 [00:02<00:00, 68773.83it/s]
 95%|█████████▌| 142817/150000 [00:02<00:00, 68649.14it/s]
100%|█████████▉| 149679/150000 [00:02<00:00, 68563.74it/s]
100%|██████████| 150000/150000 [00:02<00:00, 67930.21it/s]

  0%|          | 0/125 [00:00<?, ?it/s]
 2040 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-21 ...
Successfully removed 1403 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-22 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 11164 new domains found

Scrapping urls...
scrapping will be done in 125 phase  where 90 urls attempt in each phase

  1%|          | 1/125 [00:33<1:09:57, 33.85s/it]
  2%|▏         | 2/125 [00:54<1:01:04, 29.80s/it]
  2%|▏         | 3/125 [01:05<49:04, 24.14s/it]  
  3%|▎         | 4/125 [01:15<40:25, 20.04s/it]
  4%|▍         | 5/125 [01:44<45:28, 22.74s/it]27.56.245.116 - - [29/May/2020 06:25:29] "GET / HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:25:32] "GET /favicon.ico HTTP/1.1" 200 -

  5%|▍         | 6/125 [02:06<44:44, 22.56s/it]
  6%|▌         | 7/125 [02:27<43:20, 22.04s/it]
  6%|▋         | 8/125 [03:08<54:09, 27.77s/it]
  7%|▋         | 9/125 [03:35<52:52, 27.35s/it]
  8%|▊         | 10/125 [04:13<58:50, 30.70s/it]
  9%|▉         | 11/125 [04:41<56:36, 29.79s/it]
 10%|▉         | 12/125 [05:22<1:02:40, 33.28s/it]
 10%|█         | 13/125 [05:53<1:00:47, 32.57s/it]
 11%|█         | 14/125 [06:30<1:02:38, 33.86s/it]
 12%|█▏        | 15/125 [06:54<56:46, 30.97s/it]  
 13%|█▎        | 16/125 [07:11<48:30, 26.70s/it]
 14%|█▎        | 17/125 [07:46<52:31, 29.18s/it]
 14%|█▍        | 18/125 [08:22<55:52, 31.33s/it]
 15%|█▌        | 19/125 [08:57<56:56, 32.23s/it]
 16%|█▌        | 20/125 [09:26<54:57, 31.41s/it]27.56.245.116 - - [29/May/2020 06:33:07] "GET / HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:33:08] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:33:16] "GET /msg91.com HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:33:17] "GET /favicon.ico HTTP/1.1" 200 -

 17%|█▋        | 21/125 [10:01<56:11, 32.42s/it]
 18%|█▊        | 22/125 [10:35<56:19, 32.81s/it]
 18%|█▊        | 23/125 [11:08<56:12, 33.06s/it]
 19%|█▉        | 24/125 [11:42<55:53, 33.20s/it]
 20%|██        | 25/125 [12:24<59:37, 35.77s/it]
 21%|██        | 26/125 [12:47<53:02, 32.14s/it]
 22%|██▏       | 27/125 [13:12<48:46, 29.86s/it]
 22%|██▏       | 28/125 [13:28<41:33, 25.71s/it]phase 27 failed

 23%|██▎       | 29/125 [14:04<46:11, 28.87s/it]
 24%|██▍       | 30/125 [14:32<45:08, 28.51s/it]
 25%|██▍       | 31/125 [15:01<45:09, 28.83s/it]
 26%|██▌       | 32/125 [15:18<38:51, 25.07s/it]
 26%|██▋       | 33/125 [15:43<38:37, 25.19s/it]
 27%|██▋       | 34/125 [16:01<34:54, 23.01s/it]
 28%|██▊       | 35/125 [16:18<32:01, 21.35s/it]27.56.245.116 - - [29/May/2020 06:40:27] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:40:27] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 29%|██▉       | 36/125 [16:50<36:13, 24.42s/it]
 30%|██▉       | 37/125 [17:24<40:00, 27.27s/it]27.56.245.116 - - [29/May/2020 06:41:04] "GET /getClusterData/2020-04-05/2020-05-07/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:41:07] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:41:09] "GET /getClusterData/2020-05-05/2020-05-07/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:41:10] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:41:16] "GET /getClusterData/2020-05-05/2020-05-27/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 06:41:18] "GET /favicon.ico HTTP/1.1" 200 -

 30%|███       | 38/125 [17:44<36:18, 25.04s/it]
 31%|███       | 39/125 [17:59<31:40, 22.10s/it]
 32%|███▏      | 40/125 [18:39<38:52, 27.44s/it]
 33%|███▎      | 41/125 [18:54<33:24, 23.86s/it]
 34%|███▎      | 42/125 [19:13<30:36, 22.13s/it]
 34%|███▍      | 43/125 [19:44<34:03, 24.93s/it]
 35%|███▌      | 44/125 [20:01<30:20, 22.48s/it]
 36%|███▌      | 45/125 [20:23<29:57, 22.47s/it]
 37%|███▋      | 46/125 [20:53<32:25, 24.62s/it]
 38%|███▊      | 47/125 [21:08<28:12, 21.70s/it]
 38%|███▊      | 48/125 [21:33<29:10, 22.73s/it]
 39%|███▉      | 49/125 [21:53<27:44, 21.90s/it]
 40%|████      | 50/125 [22:32<33:45, 27.00s/it]
 41%|████      | 51/125 [22:44<28:01, 22.72s/it]
 42%|████▏     | 52/125 [23:00<24:56, 20.50s/it]
 42%|████▏     | 53/125 [23:27<27:00, 22.50s/it]
 43%|████▎     | 54/125 [23:43<24:29, 20.69s/it]
 44%|████▍     | 55/125 [24:17<28:38, 24.55s/it]
 45%|████▍     | 56/125 [24:56<33:17, 28.94s/it]
fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)

favicon.ico

fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)

favicon.ico
msg91.com
favicon.ico
favicon.ico
favicon.ico
favicon.ico

fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
162.243.137.132 - - [29/May/2020 06:48:35] "GET / HTTP/1.1" 200 -

 46%|████▌     | 57/125 [25:17<29:52, 26.37s/it]
 46%|████▋     | 58/125 [25:44<29:54, 26.78s/it]
 47%|████▋     | 59/125 [26:17<31:23, 28.54s/it]
 48%|████▊     | 60/125 [26:33<26:56, 24.87s/it]
 49%|████▉     | 61/125 [27:03<28:02, 26.30s/it]
 50%|████▉     | 62/125 [27:19<24:28, 23.31s/it]
 50%|█████     | 63/125 [27:56<28:10, 27.26s/it]
 51%|█████     | 64/125 [28:25<28:26, 27.97s/it]
 52%|█████▏    | 65/125 [28:49<26:36, 26.61s/it]
 53%|█████▎    | 66/125 [29:20<27:39, 28.13s/it]
 54%|█████▎    | 67/125 [29:33<22:35, 23.37s/it]
 54%|█████▍    | 68/125 [30:00<23:19, 24.55s/it]
 55%|█████▌    | 69/125 [30:30<24:25, 26.16s/it]
 56%|█████▌    | 70/125 [30:49<22:08, 24.15s/it]
 57%|█████▋    | 71/125 [31:10<20:42, 23.01s/it]
 58%|█████▊    | 72/125 [31:39<22:03, 24.97s/it]
 58%|█████▊    | 73/125 [31:53<18:45, 21.65s/it]
 59%|█████▉    | 74/125 [32:31<22:26, 26.40s/it]
 60%|██████    | 75/125 [32:47<19:26, 23.34s/it]
 61%|██████    | 76/125 [33:01<16:48, 20.58s/it]
 62%|██████▏   | 77/125 [33:28<18:01, 22.54s/it]
 62%|██████▏   | 78/125 [33:48<16:59, 21.70s/it]
 63%|██████▎   | 79/125 [34:19<18:53, 24.65s/it]
 64%|██████▍   | 80/125 [34:46<18:50, 25.13s/it]
 65%|██████▍   | 81/125 [35:22<20:52, 28.47s/it]
 66%|██████▌   | 82/125 [35:35<17:11, 23.98s/it]
 66%|██████▋   | 83/125 [35:56<16:01, 22.90s/it]
 67%|██████▋   | 84/125 [36:09<13:37, 19.94s/it]phase 83 failed

 68%|██████▊   | 85/125 [36:21<11:39, 17.48s/it]phase 84 failed

 69%|██████▉   | 86/125 [36:38<11:17, 17.37s/it]
 70%|██████▉   | 87/125 [36:54<10:48, 17.06s/it]
 70%|███████   | 88/125 [37:26<13:22, 21.69s/it]
 71%|███████   | 89/125 [37:47<12:49, 21.37s/it]
 72%|███████▏  | 90/125 [38:04<11:42, 20.06s/it]
 73%|███████▎  | 91/125 [38:19<10:32, 18.60s/it]
 74%|███████▎  | 92/125 [38:50<12:09, 22.12s/it]
 74%|███████▍  | 93/125 [39:03<10:26, 19.59s/it]
 75%|███████▌  | 94/125 [39:33<11:39, 22.55s/it]
 76%|███████▌  | 95/125 [39:46<09:56, 19.88s/it]phase 94 failed

 77%|███████▋  | 96/125 [43:08<35:59, 74.46s/it]phase 95 failed

 78%|███████▊  | 97/125 [43:20<25:57, 55.64s/it]phase 96 failed

 78%|███████▊  | 98/125 [43:55<22:12, 49.35s/it]
 79%|███████▉  | 99/125 [44:06<16:29, 38.05s/it]phase 98 failed

 80%|████████  | 100/125 [44:17<12:23, 29.72s/it]phase 99 failed

 81%|████████  | 101/125 [44:29<09:47, 24.50s/it]phase 100 failed

 82%|████████▏ | 102/125 [44:38<07:39, 19.98s/it]phase 101 failed

 82%|████████▏ | 103/125 [44:49<06:18, 17.19s/it]phase 102 failed

 83%|████████▎ | 104/125 [45:16<07:01, 20.09s/it]
 84%|████████▍ | 105/125 [45:31<06:10, 18.51s/it]
 85%|████████▍ | 106/125 [45:42<05:10, 16.36s/it]phase 105 failed

 86%|████████▌ | 107/125 [45:52<04:21, 14.55s/it]phase 106 failed

 86%|████████▋ | 108/125 [46:04<03:54, 13.78s/it]
 87%|████████▋ | 109/125 [46:13<03:17, 12.32s/it]phase 108 failed

 88%|████████▊ | 110/125 [46:21<02:45, 11.05s/it]phase 109 failed

 89%|████████▉ | 111/125 [46:26<02:07,  9.12s/it]phase 110 failed

 90%|████████▉ | 112/125 [47:01<03:40, 16.95s/it]
 90%|█████████ | 113/125 [47:19<03:28, 17.34s/it]
 91%|█████████ | 114/125 [47:37<03:11, 17.40s/it]
 92%|█████████▏| 115/125 [47:57<03:02, 18.23s/it]
 93%|█████████▎| 116/125 [48:10<02:29, 16.63s/it]
 94%|█████████▎| 117/125 [48:40<02:45, 20.64s/it]
 94%|█████████▍| 118/125 [49:06<02:34, 22.14s/it]
 95%|█████████▌| 119/125 [49:38<02:31, 25.19s/it]
 96%|█████████▌| 120/125 [50:06<02:10, 26.18s/it]
 97%|█████████▋| 121/125 [50:34<01:46, 26.72s/it]
 98%|█████████▊| 122/125 [51:05<01:23, 27.86s/it]
 98%|█████████▊| 123/125 [51:28<00:53, 26.52s/it]
 99%|█████████▉| 124/125 [51:44<00:23, 23.22s/it]
100%|██████████| 125/125 [51:46<00:00, 17.03s/it]
100%|██████████| 125/125 [51:46<00:00, 24.86s/it]

  0%|          | 0/231051 [00:00<?, ?it/s]
  5%|▍         | 10580/231051 [00:00<00:02, 105796.57it/s]
  9%|▉         | 21514/231051 [00:00<00:01, 106832.29it/s]
 14%|█▍        | 32662/231051 [00:00<00:01, 108184.30it/s]
 19%|█▉        | 43985/231051 [00:00<00:01, 109648.28it/s]
 24%|██▍       | 55742/231051 [00:00<00:01, 111909.97it/s]
 29%|██▉       | 67582/231051 [00:00<00:01, 113780.12it/s]
 34%|███▍      | 79319/231051 [00:00<00:01, 114833.60it/s]
 39%|███▉      | 91007/231051 [00:00<00:01, 115437.95it/s]
 45%|████▍     | 103141/231051 [00:00<00:01, 117145.96it/s]
 50%|████▉     | 115253/231051 [00:01<00:00, 118306.92it/s]
 55%|█████▌    | 127341/231051 [00:01<00:00, 119066.38it/s]
 60%|██████    | 139320/231051 [00:01<00:00, 119279.35it/s]
 65%|██████▌   | 151310/231051 [00:01<00:00, 119463.97it/s]
 71%|███████   | 163160/231051 [00:01<00:00, 118628.58it/s]
 76%|███████▌  | 174957/231051 [00:01<00:00, 116972.43it/s]
 81%|████████  | 187184/231051 [00:01<00:00, 118511.99it/s]
 86%|████████▋ | 199485/231051 [00:01<00:00, 119823.33it/s]
 92%|█████████▏| 211865/231051 [00:01<00:00, 120987.50it/s]
 97%|█████████▋| 224180/231051 [00:01<00:00, 121628.19it/s]
100%|██████████| 231051/231051 [00:01<00:00, 117789.54it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 10849/150000 [00:00<00:01, 108480.79it/s]
 15%|█▍        | 21988/150000 [00:00<00:01, 109335.66it/s]
 22%|██▏       | 33479/150000 [00:00<00:01, 110950.14it/s]
 30%|███       | 45271/150000 [00:00<00:00, 112951.81it/s]
 38%|███▊      | 57324/150000 [00:00<00:00, 115122.53it/s]
 46%|████▋     | 69429/150000 [00:00<00:00, 116836.44it/s]
 54%|█████▍    | 81578/150000 [00:00<00:00, 118194.43it/s]
 63%|██████▎   | 93850/150000 [00:00<00:00, 119513.45it/s]
 71%|███████   | 106146/150000 [00:00<00:00, 120525.72it/s]
 79%|███████▉  | 118546/150000 [00:01<00:00, 121545.21it/s]
 87%|████████▋ | 130968/150000 [00:01<00:00, 122333.09it/s]
 96%|█████████▌| 143324/150000 [00:01<00:00, 122696.54it/s]
100%|██████████| 150000/150000 [00:01<00:00, 119635.45it/s]

0it [00:00, ?it/s]
30it [00:00, 298.74it/s]
78it [00:00, 336.86it/s]
132it [00:00, 379.12it/s]
183it [00:00, 406.83it/s]
240it [00:00, 444.18it/s]
290it [00:00, 453.95it/s]
339it [00:00, 462.92it/s]
390it [00:00, 473.87it/s]
448it [00:00, 490.19it/s]
514it [00:01, 529.60it/s]
590it [00:01, 581.91it/s]
661it [00:01, 613.77it/s]
741it [00:01, 659.80it/s]
817it [00:01, 684.82it/s]
894it [00:01, 706.92it/s]
972it [00:01, 725.81it/s]
1058it [00:01, 759.45it/s]
1150it [00:01, 801.33it/s]
1232it [00:01, 782.16it/s]
1346it [00:02, 860.10it/s]
1446it [00:02, 895.24it/s]
1544it [00:02, 914.36it/s]
1646it [00:02, 943.64it/s]
1742it [00:02, 899.85it/s]
1844it [00:02, 928.55it/s]
1939it [00:02, 912.49it/s]
2032it [00:02, 897.74it/s]
2134it [00:02, 928.98it/s]
2289it [00:02, 1052.74it/s]
2445it [00:03, 1152.82it/s]
2577it [00:03, 1198.33it/s]
2726it [00:03, 1272.55it/s]
2939it [00:03, 1445.16it/s]
3129it [00:03, 1554.31it/s]
3395it [00:03, 1764.37it/s]
3696it [00:03, 2009.17it/s]
3787it [00:03, 1012.39it/s]

  0%|          | 0/595 [00:00<?, ?it/s]
 28%|██▊       | 164/595 [00:00<00:00, 1633.63it/s]
 56%|█████▋    | 336/595 [00:00<00:00, 1658.07it/s]
 79%|███████▉  | 472/595 [00:00<00:00, 1555.29it/s]
100%|██████████| 595/595 [00:00<00:00, 1526.50it/s]

  0%|          | 0/68297 [00:00<?, ?it/s]
  1%|          | 785/68297 [00:00<00:12, 5326.24it/s]
  5%|▍         | 3198/68297 [00:00<00:09, 6951.26it/s]
  8%|▊         | 5552/68297 [00:00<00:07, 8814.77it/s]
 12%|█▏        | 7865/68297 [00:00<00:05, 10823.82it/s]
 14%|█▍        | 9463/68297 [00:00<00:06, 8511.80it/s] 
 17%|█▋        | 11624/68297 [00:00<00:05, 10403.50it/s]
 20%|██        | 13851/68297 [00:00<00:04, 12382.42it/s]
 24%|██▎       | 16084/68297 [00:01<00:03, 14291.95it/s]
 27%|██▋       | 18319/68297 [00:01<00:03, 16024.20it/s]
 30%|███       | 20572/68297 [00:01<00:02, 17543.15it/s]
 33%|███▎      | 22647/68297 [00:01<00:02, 18255.16it/s]
 37%|███▋      | 25036/68297 [00:01<00:02, 19644.32it/s]
 40%|████      | 27457/68297 [00:01<00:01, 20820.90it/s]
 44%|████▍     | 29883/68297 [00:01<00:01, 21745.11it/s]
 47%|████▋     | 32309/68297 [00:01<00:01, 22441.34it/s]
 51%|█████     | 34728/68297 [00:01<00:01, 22936.20it/s]
 54%|█████▍    | 37144/68297 [00:01<00:01, 23289.92it/s]
 58%|█████▊    | 39613/68297 [00:02<00:01, 23692.48it/s]
 62%|██████▏   | 42213/68297 [00:02<00:01, 24338.26it/s]
 66%|██████▌   | 44804/68297 [00:02<00:00, 24787.74it/s]
 69%|██████▉   | 47407/68297 [00:02<00:00, 25147.63it/s]
 73%|███████▎  | 49939/68297 [00:02<00:00, 24191.33it/s]
 77%|███████▋  | 52664/68297 [00:02<00:00, 25031.58it/s]
 81%|████████  | 55491/68297 [00:02<00:00, 25921.52it/s]
 85%|████████▌ | 58319/68297 [00:02<00:00, 26583.51it/s]
 89%|████████▉ | 60998/68297 [00:02<00:00, 26520.01it/s]
 93%|█████████▎| 63665/68297 [00:02<00:00, 26238.73it/s]
 97%|█████████▋| 66580/68297 [00:03<00:00, 27048.10it/s]
100%|██████████| 68297/68297 [00:03<00:00, 21966.63it/s]

  0%|          | 0/68297 [00:00<?, ?it/s]
  9%|▉         | 6340/68297 [00:00<00:00, 63398.55it/s]
 15%|█▌        | 10497/68297 [00:00<00:01, 54769.59it/s]
 21%|██▏       | 14628/68297 [00:00<00:01, 49890.99it/s]
 27%|██▋       | 18438/68297 [00:00<00:01, 45651.62it/s]
 32%|███▏      | 22137/68297 [00:00<00:01, 42652.68it/s]
 37%|███▋      | 25539/68297 [00:00<00:01, 39274.10it/s]
 43%|████▎     | 29226/68297 [00:00<00:01, 38519.32it/s]
 48%|████▊     | 33071/68297 [00:00<00:00, 37132.19it/s]
 54%|█████▍    | 36792/68297 [00:00<00:00, 36587.73it/s]
 59%|█████▉    | 40323/68297 [00:01<00:00, 35222.32it/s]
 64%|██████▍   | 43909/68297 [00:01<00:00, 35365.23it/s]
 70%|██████▉   | 47573/68297 [00:01<00:00, 35320.78it/s]
 75%|███████▌  | 51425/68297 [00:01<00:00, 35571.08it/s]
 82%|████████▏ | 55710/68297 [00:01<00:00, 36515.06it/s]
 89%|████████▊ | 60552/68297 [00:01<00:00, 39422.81it/s]
 95%|█████████▍| 64871/68297 [00:01<00:00, 40481.11it/s]
100%|██████████| 68297/68297 [00:01<00:00, 39393.89it/s]

  0%|          | 0/3787 [00:00<?, ?it/s]
100%|██████████| 3787/3787 [00:00<00:00, 83941.05it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:14,  6.60it/s]
  2%|▏         | 2/100 [00:00<00:14,  6.64it/s]
  3%|▎         | 3/100 [00:00<00:14,  6.53it/s]
  4%|▍         | 4/100 [00:00<00:15,  6.36it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.47it/s]
  6%|▌         | 6/100 [00:00<00:14,  6.56it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.49it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.30it/s]
  9%|▉         | 9/100 [00:01<00:14,  6.46it/s]
 10%|█         | 10/100 [00:01<00:13,  6.46it/s]
 11%|█         | 11/100 [00:01<00:13,  6.64it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.32it/s]
 13%|█▎        | 13/100 [00:02<00:13,  6.52it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.48it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.23it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.14it/s]
 17%|█▋        | 17/100 [00:02<00:13,  6.36it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.41it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.40it/s]
 20%|██        | 20/100 [00:03<00:12,  6.42it/s]
 21%|██        | 21/100 [00:03<00:12,  6.52it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.62it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.46it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.22it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.29it/s]
 26%|██▌       | 26/100 [00:04<00:12,  6.05it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.21it/s]
 28%|██▊       | 28/100 [00:04<00:12,  5.93it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.19it/s]
 30%|███       | 30/100 [00:04<00:10,  6.45it/s]
 31%|███       | 31/100 [00:04<00:11,  6.26it/s]
 32%|███▏      | 32/100 [00:05<00:10,  6.46it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.63it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.73it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.75it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.65it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.51it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.59it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.52it/s]
 40%|████      | 40/100 [00:06<00:09,  6.65it/s]
 41%|████      | 41/100 [00:06<00:09,  6.33it/s]
 42%|████▏     | 42/100 [00:06<00:09,  6.39it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.52it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.72it/s]
 45%|████▌     | 45/100 [00:07<00:08,  6.20it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.29it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.39it/s]
 48%|████▊     | 48/100 [00:07<00:08,  6.37it/s]
 49%|████▉     | 49/100 [00:07<00:08,  6.33it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.33it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.40it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.27it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.04it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  5.96it/s]
 55%|█████▌    | 55/100 [00:08<00:07,  6.05it/s]
 56%|█████▌    | 56/100 [00:08<00:07,  6.18it/s]
 57%|█████▋    | 57/100 [00:08<00:07,  6.02it/s]
 58%|█████▊    | 58/100 [00:09<00:06,  6.00it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.05it/s]
 60%|██████    | 60/100 [00:09<00:06,  5.91it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.17it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.04it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.07it/s]
 64%|██████▍   | 64/100 [00:10<00:05,  6.21it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.23it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.44it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.46it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.66it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.58it/s]
 70%|███████   | 70/100 [00:11<00:04,  6.67it/s]
 71%|███████   | 71/100 [00:11<00:04,  6.72it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.73it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.14it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.38it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.64it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.73it/s]
 77%|███████▋  | 77/100 [00:12<00:03,  6.61it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.54it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.43it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.18it/s]
 81%|████████  | 81/100 [00:12<00:03,  6.09it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.38it/s]
 83%|████████▎ | 83/100 [00:13<00:02,  6.55it/s]
 84%|████████▍ | 84/100 [00:13<00:02,  6.47it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.73it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.80it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.90it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.74it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.66it/s]
 90%|█████████ | 90/100 [00:14<00:01,  6.50it/s]
 91%|█████████ | 91/100 [00:14<00:01,  6.51it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.39it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.44it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.21it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.31it/s]
 96%|█████████▌| 96/100 [00:15<00:00,  6.52it/s]
 97%|█████████▋| 97/100 [00:15<00:00,  6.47it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.75it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.41it/s]
100%|██████████| 100/100 [00:15<00:00,  6.25it/s]
100%|██████████| 100/100 [00:15<00:00,  6.39it/s]

  0%|          | 0/68297 [00:00<?, ?it/s]
  6%|▌         | 3947/68297 [00:00<00:01, 39468.25it/s]
 13%|█▎        | 8645/68297 [00:00<00:01, 41456.09it/s]
 23%|██▎       | 15550/68297 [00:00<00:01, 47102.10it/s]
 33%|███▎      | 22851/68297 [00:00<00:00, 52712.98it/s]
 44%|████▍     | 30318/68297 [00:00<00:00, 57812.34it/s]
 56%|█████▌    | 37935/68297 [00:00<00:00, 62316.85it/s]
 67%|██████▋   | 45911/68297 [00:00<00:00, 66689.91it/s]
 80%|███████▉  | 54544/68297 [00:00<00:00, 71574.71it/s]
 94%|█████████▍| 64099/68297 [00:00<00:00, 77400.29it/s]
100%|██████████| 68297/68297 [00:00<00:00, 72745.22it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/302071 [00:00<?, ?it/s]
  4%|▎         | 11086/302071 [00:00<00:02, 110859.31it/s]
  7%|▋         | 22384/302071 [00:00<00:02, 111486.20it/s]
 11%|█         | 33857/302071 [00:00<00:02, 112439.32it/s]
 15%|█▌        | 45709/302071 [00:00<00:02, 114196.78it/s]
 19%|█▉        | 57821/302071 [00:00<00:02, 116186.91it/s]
 23%|██▎       | 70045/302071 [00:00<00:01, 117935.88it/s]
 27%|██▋       | 82368/302071 [00:00<00:01, 119473.03it/s]
 31%|███▏      | 94712/302071 [00:00<00:01, 120634.02it/s]
 35%|███▌      | 106875/302071 [00:00<00:01, 120929.32it/s]
 40%|███▉      | 119439/302071 [00:01<00:01, 122302.49it/s]
 44%|████▎     | 131899/302071 [00:01<00:01, 122981.18it/s]
 48%|████▊     | 144481/302071 [00:01<00:01, 123816.79it/s]
 52%|█████▏    | 157154/302071 [00:01<00:01, 124675.40it/s]
 56%|█████▌    | 169721/302071 [00:01<00:01, 124968.43it/s]
 60%|██████    | 182430/302071 [00:01<00:00, 125595.95it/s]
 65%|██████▍   | 195144/302071 [00:01<00:00, 126053.19it/s]
 69%|██████▉   | 207716/302071 [00:01<00:00, 125097.09it/s]
 73%|███████▎  | 220340/302071 [00:01<00:00, 125436.13it/s]
 77%|███████▋  | 232906/302071 [00:01<00:00, 125501.27it/s]
 81%|████████▏ | 245527/302071 [00:02<00:00, 125709.95it/s]
 86%|████████▌ | 258674/302071 [00:02<00:00, 127382.27it/s]
 90%|████████▉ | 271413/302071 [00:02<00:00, 126491.06it/s]
 94%|█████████▍| 284277/302071 [00:02<00:00, 127127.21it/s]
 98%|█████████▊| 296992/302071 [00:02<00:00, 126606.93it/s]
100%|██████████| 302071/302071 [00:02<00:00, 123548.10it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6446/150000 [00:00<00:02, 64452.69it/s]
  9%|▊         | 13031/150000 [00:00<00:02, 64865.27it/s]
 13%|█▎        | 19681/150000 [00:00<00:01, 65346.47it/s]
 18%|█▊        | 26388/150000 [00:00<00:01, 65851.58it/s]
 22%|██▏       | 33213/150000 [00:00<00:01, 66550.87it/s]
 27%|██▋       | 40116/150000 [00:00<00:01, 67274.37it/s]
 31%|███▏      | 47110/150000 [00:00<00:01, 68050.84it/s]
 36%|███▌      | 54044/150000 [00:00<00:01, 68431.72it/s]
 41%|████      | 61068/150000 [00:00<00:01, 68961.47it/s]
 45%|████▌     | 68043/150000 [00:01<00:01, 69193.76it/s]
 50%|█████     | 75018/150000 [00:01<00:01, 69357.60it/s]
 55%|█████▍    | 81965/150000 [00:01<00:00, 69388.77it/s]
 59%|█████▉    | 88915/150000 [00:01<00:00, 69419.35it/s]
 64%|██████▍   | 95846/150000 [00:01<00:00, 69384.29it/s]
 68%|██████▊   | 102744/150000 [00:01<00:00, 69261.83it/s]
 73%|███████▎  | 109732/150000 [00:01<00:00, 69445.85it/s]
 78%|███████▊  | 116686/150000 [00:01<00:00, 69471.76it/s]
 82%|████████▏ | 123675/150000 [00:01<00:00, 69593.50it/s]
 87%|████████▋ | 130720/150000 [00:01<00:00, 69845.95it/s]
 92%|█████████▏| 137722/150000 [00:02<00:00, 69895.50it/s]
 96%|█████████▋| 144707/150000 [00:02<00:00, 69829.94it/s]
100%|██████████| 150000/150000 [00:02<00:00, 68898.82it/s]

  0%|          | 0/94 [00:00<?, ?it/s]
 3787 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-22 ...
Successfully removed 595 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-23 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 8408 new domains found

Scrapping urls...
scrapping will be done in 94 phase  where 90 urls attempt in each phase

  1%|          | 1/94 [00:29<46:03, 29.71s/it]
  2%|▏         | 2/94 [00:46<39:34, 25.81s/it]
  3%|▎         | 3/94 [00:57<32:15, 21.27s/it]
  4%|▍         | 4/94 [01:04<25:47, 17.20s/it]
  5%|▌         | 5/94 [01:41<34:08, 23.01s/it]
  6%|▋         | 6/94 [02:14<38:11, 26.04s/it]
  7%|▋         | 7/94 [02:50<42:10, 29.08s/it]
  9%|▊         | 8/94 [03:25<43:58, 30.68s/it]
 10%|▉         | 9/94 [03:42<37:46, 26.67s/it]
 11%|█         | 10/94 [04:21<42:32, 30.38s/it]
 12%|█▏        | 11/94 [04:57<44:24, 32.11s/it]
 13%|█▎        | 12/94 [05:37<47:09, 34.50s/it]
 14%|█▍        | 13/94 [06:15<48:04, 35.61s/it]
 15%|█▍        | 14/94 [06:48<46:09, 34.62s/it]
 16%|█▌        | 15/94 [07:32<49:21, 37.49s/it]
 17%|█▋        | 16/94 [08:12<49:57, 38.43s/it]
 18%|█▊        | 17/94 [08:46<47:21, 36.90s/it]
 19%|█▉        | 18/94 [09:29<49:06, 38.77s/it]
 20%|██        | 19/94 [10:07<48:16, 38.62s/it]
 21%|██▏       | 20/94 [10:48<48:36, 39.41s/it]
 22%|██▏       | 21/94 [11:24<46:26, 38.17s/it]
 23%|██▎       | 22/94 [12:01<45:25, 37.86s/it]
 24%|██▍       | 23/94 [12:39<44:56, 37.98s/it]
 26%|██▌       | 24/94 [13:22<46:03, 39.48s/it]
 27%|██▋       | 25/94 [14:02<45:36, 39.66s/it]
 28%|██▊       | 26/94 [14:47<46:39, 41.17s/it]
 29%|██▊       | 27/94 [15:24<44:34, 39.91s/it]
 30%|██▉       | 28/94 [16:00<42:42, 38.83s/it]
 31%|███       | 29/94 [16:36<41:02, 37.88s/it]
 32%|███▏      | 30/94 [17:14<40:26, 37.92s/it]
 33%|███▎      | 31/94 [17:46<37:58, 36.16s/it]
 34%|███▍      | 32/94 [18:23<37:47, 36.57s/it]
 35%|███▌      | 33/94 [19:01<37:37, 37.01s/it]
 36%|███▌      | 34/94 [19:18<30:53, 30.90s/it]
 37%|███▋      | 35/94 [19:42<28:12, 28.68s/it]
 38%|███▊      | 36/94 [20:20<30:29, 31.54s/it]
 39%|███▉      | 37/94 [21:03<33:18, 35.07s/it]
 40%|████      | 38/94 [21:42<33:44, 36.15s/it]
 41%|████▏     | 39/94 [22:07<30:14, 32.99s/it]
 43%|████▎     | 40/94 [22:24<25:23, 28.22s/it]
 44%|████▎     | 41/94 [22:46<23:09, 26.23s/it]
 45%|████▍     | 42/94 [23:22<25:19, 29.23s/it]
 46%|████▌     | 43/94 [24:02<27:29, 32.34s/it]
 47%|████▋     | 44/94 [24:21<23:45, 28.52s/it]
 48%|████▊     | 45/94 [24:40<20:49, 25.50s/it]
 49%|████▉     | 46/94 [25:01<19:16, 24.09s/it]
 50%|█████     | 47/94 [25:38<21:51, 27.91s/it]
 51%|█████     | 48/94 [26:17<24:02, 31.36s/it]
 52%|█████▏    | 49/94 [26:53<24:36, 32.80s/it]
 53%|█████▎    | 50/94 [27:23<23:24, 31.91s/it]
 54%|█████▍    | 51/94 [27:35<18:42, 26.10s/it]
 55%|█████▌    | 52/94 [28:13<20:45, 29.65s/it]
 56%|█████▋    | 53/94 [28:50<21:44, 31.83s/it]
 57%|█████▋    | 54/94 [29:11<18:58, 28.45s/it]
 59%|█████▊    | 55/94 [29:28<16:11, 24.92s/it]
 60%|█████▉    | 56/94 [30:05<18:07, 28.63s/it]
 61%|██████    | 57/94 [30:17<14:41, 23.82s/it]
 62%|██████▏   | 58/94 [30:54<16:39, 27.78s/it]
 63%|██████▎   | 59/94 [31:26<16:52, 28.92s/it]
 64%|██████▍   | 60/94 [32:03<17:48, 31.42s/it]
 65%|██████▍   | 61/94 [32:15<14:02, 25.53s/it]
 66%|██████▌   | 62/94 [32:28<11:38, 21.81s/it]
 67%|██████▋   | 63/94 [33:06<13:47, 26.71s/it]
 68%|██████▊   | 64/94 [33:35<13:38, 27.28s/it]
 69%|██████▉   | 65/94 [34:07<13:55, 28.82s/it]
 70%|███████   | 66/94 [34:43<14:27, 30.97s/it]
 71%|███████▏  | 67/94 [35:07<12:53, 28.63s/it]
 72%|███████▏  | 68/94 [35:47<13:53, 32.07s/it]
 73%|███████▎  | 69/94 [36:01<11:05, 26.61s/it]
 74%|███████▍  | 70/94 [36:36<11:41, 29.22s/it]
 76%|███████▌  | 71/94 [36:59<10:27, 27.29s/it]
 77%|███████▋  | 72/94 [37:35<10:57, 29.89s/it]
 78%|███████▊  | 73/94 [38:11<11:08, 31.85s/it]
 79%|███████▊  | 74/94 [38:40<10:17, 30.88s/it]
 80%|███████▉  | 75/94 [39:11<09:47, 30.94s/it]
 81%|████████  | 76/94 [39:51<10:07, 33.75s/it]
 82%|████████▏ | 77/94 [40:06<08:00, 28.26s/it]
 83%|████████▎ | 78/94 [40:48<08:37, 32.34s/it]
 84%|████████▍ | 79/94 [41:27<08:31, 34.11s/it]
 85%|████████▌ | 80/94 [41:46<06:53, 29.57s/it]
 86%|████████▌ | 81/94 [42:12<06:11, 28.61s/it]
 87%|████████▋ | 82/94 [42:53<06:27, 32.27s/it]
 88%|████████▊ | 83/94 [43:28<06:03, 33.03s/it]
 89%|████████▉ | 84/94 [44:07<05:49, 34.97s/it]
 90%|█████████ | 85/94 [44:44<05:19, 35.47s/it]
 91%|█████████▏| 86/94 [44:59<03:55, 29.48s/it]
 93%|█████████▎| 87/94 [45:41<03:51, 33.07s/it]
 94%|█████████▎| 88/94 [46:21<03:31, 35.30s/it]
 95%|█████████▍| 89/94 [46:39<02:30, 30.19s/it]
 96%|█████████▌| 90/94 [46:56<01:44, 26.20s/it]
 97%|█████████▋| 91/94 [47:22<01:18, 26.06s/it]
 98%|█████████▊| 92/94 [47:53<00:55, 27.69s/it]
 99%|█████████▉| 93/94 [48:10<00:24, 24.48s/it]
100%|██████████| 94/94 [48:22<00:00, 20.48s/it]
100%|██████████| 94/94 [48:22<00:00, 30.87s/it]

  0%|          | 0/232157 [00:00<?, ?it/s]
  5%|▍         | 10923/232157 [00:00<00:02, 109087.58it/s]
 10%|▉         | 22224/232157 [00:00<00:01, 110232.75it/s]
 15%|█▍        | 33886/232157 [00:00<00:01, 112072.49it/s]
 20%|█▉        | 45484/232157 [00:00<00:01, 113215.19it/s]
 25%|██▍       | 57475/232157 [00:00<00:01, 115143.36it/s]
 30%|██▉       | 69524/232157 [00:00<00:01, 116696.24it/s]
 35%|███▌      | 81570/232157 [00:00<00:01, 117799.57it/s]
 40%|████      | 93520/232157 [00:00<00:01, 118304.29it/s]
 46%|████▌     | 105805/232157 [00:00<00:01, 119629.65it/s]
 51%|█████     | 118060/232157 [00:01<00:00, 120488.95it/s]
 56%|█████▌    | 130398/232157 [00:01<00:00, 121340.86it/s]
 61%|██████▏   | 142763/232157 [00:01<00:00, 122024.45it/s]
 67%|██████▋   | 155174/232157 [00:01<00:00, 122639.72it/s]
 72%|███████▏  | 167338/232157 [00:01<00:00, 121807.90it/s]
 77%|███████▋  | 179450/232157 [00:01<00:00, 119089.69it/s]
 83%|████████▎ | 191762/232157 [00:01<00:00, 120267.81it/s]
 88%|████████▊ | 204213/232157 [00:01<00:00, 121508.56it/s]
 93%|█████████▎| 216626/232157 [00:01<00:00, 122280.66it/s]
 99%|█████████▊| 229100/232157 [00:01<00:00, 123006.00it/s]
100%|██████████| 232157/232157 [00:01<00:00, 120071.83it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11062/150000 [00:00<00:01, 110609.82it/s]
 15%|█▍        | 22414/150000 [00:00<00:01, 111464.77it/s]
 23%|██▎       | 34111/150000 [00:00<00:01, 113060.87it/s]
 31%|███       | 46207/150000 [00:00<00:00, 115318.97it/s]
 39%|███▉      | 58438/150000 [00:00<00:00, 117328.11it/s]
 47%|████▋     | 70713/150000 [00:00<00:00, 118900.78it/s]
 55%|█████▌    | 82949/150000 [00:00<00:00, 119916.69it/s]
 64%|██████▎   | 95298/150000 [00:00<00:00, 120964.02it/s]
 72%|███████▏  | 107627/150000 [00:00<00:00, 121649.66it/s]
 80%|████████  | 120057/150000 [00:01<00:00, 122430.26it/s]
 88%|████████▊ | 132593/150000 [00:01<00:00, 123294.05it/s]
 97%|█████████▋| 145124/150000 [00:01<00:00, 123889.31it/s]
100%|██████████| 150000/150000 [00:01<00:00, 121005.48it/s]

0it [00:00, ?it/s]
28it [00:00, 277.62it/s]
85it [00:00, 328.11it/s]
171it [00:00, 402.32it/s]
281it [00:00, 496.27it/s]
369it [00:00, 569.83it/s]
452it [00:00, 628.49it/s]
526it [00:00, 643.59it/s]
610it [00:00, 687.29it/s]
720it [00:00, 772.37it/s]
864it [00:01, 897.04it/s]
986it [00:01, 972.28it/s]
1122it [00:01, 1063.18it/s]
1245it [00:01, 1107.87it/s]
1434it [00:01, 1257.13it/s]
1588it [00:01, 1328.00it/s]
1731it [00:01, 1348.77it/s]
1992it [00:01, 1577.11it/s]
2216it [00:01, 1222.23it/s]

  0%|          | 0/1110 [00:00<?, ?it/s]
 63%|██████▎   | 698/1110 [00:00<00:00, 6963.11it/s]
100%|██████████| 1110/1110 [00:00<00:00, 8114.13it/s]

  0%|          | 0/68810 [00:00<?, ?it/s]
  0%|          | 17/68810 [00:00<06:44, 169.95it/s]
  0%|          | 35/68810 [00:00<06:58, 164.44it/s]
  0%|          | 55/68810 [00:00<06:48, 168.13it/s]
  0%|          | 91/68810 [00:00<05:45, 199.06it/s]
  0%|          | 300/68810 [00:00<04:12, 271.74it/s]
  1%|          | 429/68810 [00:00<03:12, 354.74it/s]
  1%|          | 514/68810 [00:00<02:41, 423.76it/s]
  1%|          | 793/68810 [00:00<02:00, 562.79it/s]
  1%|▏         | 923/68810 [00:01<01:48, 628.09it/s]
  2%|▏         | 1040/68810 [00:01<01:49, 616.21it/s]
  2%|▏         | 1244/68810 [00:01<01:36, 699.94it/s]
  2%|▏         | 1374/68810 [00:01<01:38, 683.71it/s]
  2%|▏         | 1464/68810 [00:01<01:51, 604.56it/s]
  2%|▏         | 1542/68810 [00:01<01:48, 622.53it/s]
  6%|▌         | 3889/68810 [00:02<01:13, 879.27it/s]
  9%|▉         | 6251/68810 [00:02<00:50, 1236.36it/s]
 13%|█▎        | 8845/68810 [00:02<00:34, 1730.87it/s]
 16%|█▋        | 11289/68810 [00:02<00:23, 2399.81it/s]
 20%|██        | 13765/68810 [00:02<00:16, 3291.56it/s]
 24%|██▎       | 16316/68810 [00:02<00:11, 4455.73it/s]
 27%|██▋       | 18774/68810 [00:02<00:08, 5906.31it/s]
 31%|███       | 21230/68810 [00:02<00:06, 7649.12it/s]
 34%|███▍      | 23691/68810 [00:02<00:04, 9642.72it/s]
 38%|███▊      | 26201/68810 [00:02<00:03, 11827.55it/s]
 42%|████▏     | 28739/68810 [00:03<00:02, 14083.22it/s]
 45%|████▌     | 31261/68810 [00:03<00:02, 16233.16it/s]
 49%|████▉     | 33784/68810 [00:03<00:01, 18176.26it/s]
 53%|█████▎    | 36295/68810 [00:03<00:01, 19817.69it/s]
 56%|█████▋    | 38808/68810 [00:03<00:01, 21158.76it/s]
 60%|██████    | 41371/68810 [00:03<00:01, 22326.03it/s]
 64%|██████▍   | 43907/68810 [00:03<00:01, 23156.42it/s]
 68%|██████▊   | 46493/68810 [00:03<00:00, 23904.98it/s]
 71%|███████▏  | 49036/68810 [00:03<00:00, 23543.66it/s]
 75%|███████▍  | 51498/68810 [00:03<00:00, 23655.78it/s]
 79%|███████▊  | 54083/68810 [00:04<00:00, 24271.80it/s]
 82%|████████▏ | 56676/68810 [00:04<00:00, 24745.23it/s]
 86%|████████▌ | 59193/68810 [00:04<00:00, 24483.37it/s]
 90%|████████▉ | 61672/68810 [00:04<00:00, 23726.71it/s]
 93%|█████████▎| 64071/68810 [00:04<00:00, 23446.30it/s]
 97%|█████████▋| 66790/68810 [00:04<00:00, 24455.50it/s]
100%|██████████| 68810/68810 [00:04<00:00, 14785.03it/s]

  0%|          | 0/68810 [00:00<?, ?it/s]
  9%|▊         | 5944/68810 [00:00<00:01, 52795.63it/s]
 16%|█▌        | 10759/68810 [00:00<00:01, 49412.96it/s]
 22%|██▏       | 15468/68810 [00:00<00:01, 48691.72it/s]
 28%|██▊       | 19271/68810 [00:00<00:01, 44914.19it/s]
 33%|███▎      | 22905/68810 [00:00<00:01, 41943.11it/s]
 38%|███▊      | 26360/68810 [00:00<00:01, 39412.85it/s]
 43%|████▎     | 29805/68810 [00:00<00:01, 37778.87it/s]
 48%|████▊     | 33232/68810 [00:00<00:00, 36549.13it/s]
 53%|█████▎    | 36650/68810 [00:00<00:00, 35707.29it/s]
 58%|█████▊    | 40229/68810 [00:01<00:00, 35532.66it/s]
 64%|██████▍   | 44026/68810 [00:01<00:00, 35509.63it/s]
 69%|██████▉   | 47792/68810 [00:01<00:00, 35339.31it/s]
 75%|███████▌  | 51684/68810 [00:01<00:00, 34863.86it/s]
 81%|████████▏ | 55911/68810 [00:01<00:00, 35589.48it/s]
 88%|████████▊ | 60523/68810 [00:01<00:00, 36647.91it/s]
 95%|█████████▍| 65323/68810 [00:01<00:00, 39444.92it/s]
100%|██████████| 68810/68810 [00:01<00:00, 38777.93it/s]

  0%|          | 0/2216 [00:00<?, ?it/s]
100%|██████████| 2216/2216 [00:00<00:00, 75695.52it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:14,  6.89it/s]
  2%|▏         | 2/100 [00:00<00:14,  6.85it/s]
  3%|▎         | 3/100 [00:00<00:14,  6.78it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.53it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.57it/s]
  6%|▌         | 6/100 [00:00<00:14,  6.60it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.50it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.27it/s]
  9%|▉         | 9/100 [00:01<00:14,  6.42it/s]
 10%|█         | 10/100 [00:01<00:14,  6.41it/s]
 11%|█         | 11/100 [00:01<00:13,  6.55it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.33it/s]
 13%|█▎        | 13/100 [00:02<00:13,  6.54it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.52it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.30it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.26it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.45it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.48it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.47it/s]
 20%|██        | 20/100 [00:03<00:12,  6.56it/s]
 21%|██        | 21/100 [00:03<00:11,  6.64it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.76it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.50it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.25it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.31it/s]
 26%|██▌       | 26/100 [00:04<00:12,  6.07it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.22it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.01it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.26it/s]
 30%|███       | 30/100 [00:04<00:10,  6.46it/s]
 31%|███       | 31/100 [00:04<00:11,  6.21it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.38it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.57it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.71it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.68it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.57it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.47it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.60it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.53it/s]
 40%|████      | 40/100 [00:06<00:09,  6.63it/s]
 41%|████      | 41/100 [00:06<00:08,  6.56it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.65it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.67it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.81it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.33it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.50it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.59it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.56it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.60it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.62it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.75it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.65it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.48it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.45it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.51it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.56it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.30it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.25it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.29it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.19it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.40it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.16it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.16it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.33it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.36it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.60it/s]
 67%|██████▋   | 67/100 [00:10<00:04,  6.62it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.82it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.67it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.73it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.76it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.71it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.21it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.43it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.63it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.61it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.55it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.52it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.47it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.23it/s]
 81%|████████  | 81/100 [00:12<00:03,  6.32it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.54it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.69it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.52it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.75it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.88it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.97it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.71it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.62it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.58it/s]
 91%|█████████ | 91/100 [00:14<00:01,  6.54it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.42it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.42it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.23it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.24it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.61it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.58it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.84it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.54it/s]
100%|██████████| 100/100 [00:15<00:00,  6.43it/s]
100%|██████████| 100/100 [00:15<00:00,  6.49it/s]

  0%|          | 0/68810 [00:00<?, ?it/s]
  6%|▋         | 4363/68810 [00:00<00:01, 43629.63it/s]
 14%|█▍        | 9620/68810 [00:00<00:01, 45973.45it/s]
 24%|██▍       | 16628/68810 [00:00<00:01, 51262.34it/s]
 35%|███▍      | 23967/68810 [00:00<00:00, 56359.71it/s]
 46%|████▌     | 31574/68810 [00:00<00:00, 61108.00it/s]
 57%|█████▋    | 39312/68810 [00:00<00:00, 65221.12it/s]
 68%|██████▊   | 46764/68810 [00:00<00:00, 67755.98it/s]
 80%|████████  | 55086/68810 [00:00<00:00, 71754.06it/s]
 93%|█████████▎| 64145/68810 [00:00<00:00, 76525.79it/s]
100%|██████████| 68810/68810 [00:00<00:00, 72225.62it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/328014 [00:00<?, ?it/s]
  3%|▎         | 11326/328014 [00:00<00:02, 113253.36it/s]
  7%|▋         | 22663/328014 [00:00<00:02, 113287.14it/s]
 10%|█         | 34237/328014 [00:00<00:02, 114009.23it/s]
 14%|█▍        | 46217/328014 [00:00<00:02, 115684.93it/s]
 18%|█▊        | 58467/328014 [00:00<00:02, 117647.06it/s]
 22%|██▏       | 70860/328014 [00:00<00:02, 119463.23it/s]
 25%|██▌       | 83272/328014 [00:00<00:02, 120821.92it/s]
 29%|██▉       | 95749/328014 [00:00<00:01, 121977.16it/s]
 33%|███▎      | 108112/328014 [00:00<00:01, 122467.61it/s]
 37%|███▋      | 120735/328014 [00:01<00:01, 123572.17it/s]
 41%|████      | 133325/328014 [00:01<00:01, 124260.32it/s]
 44%|████▍     | 145954/328014 [00:01<00:01, 124858.93it/s]
 48%|████▊     | 158574/328014 [00:01<00:01, 125256.33it/s]
 52%|█████▏    | 171147/328014 [00:01<00:01, 125395.95it/s]
 56%|█████▌    | 183750/328014 [00:01<00:01, 125584.09it/s]
 60%|█████▉    | 196257/328014 [00:01<00:01, 124500.32it/s]
 64%|██████▎   | 208905/328014 [00:01<00:00, 125087.19it/s]
 68%|██████▊   | 221563/328014 [00:01<00:00, 125529.85it/s]
 72%|███████▏  | 234664/328014 [00:01<00:00, 127124.38it/s]
 75%|███████▌  | 247371/328014 [00:02<00:00, 126989.71it/s]
 79%|███████▉  | 260486/328014 [00:02<00:00, 128207.34it/s]
 83%|████████▎ | 273307/328014 [00:02<00:00, 127907.65it/s]
 87%|████████▋ | 286435/328014 [00:02<00:00, 128898.79it/s]
 91%|█████████▏| 299327/328014 [00:02<00:00, 128266.00it/s]
 95%|█████████▌| 312371/328014 [00:02<00:00, 128910.22it/s]
 99%|█████████▉| 325310/328014 [00:02<00:00, 129053.08it/s]
100%|██████████| 328014/328014 [00:02<00:00, 124861.87it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6394/150000 [00:00<00:02, 63933.66it/s]
  9%|▊         | 12994/150000 [00:00<00:02, 64538.24it/s]
 13%|█▎        | 19682/150000 [00:00<00:01, 65189.14it/s]
 18%|█▊        | 26452/150000 [00:00<00:01, 65921.63it/s]
 22%|██▏       | 33258/150000 [00:00<00:01, 66548.03it/s]
 27%|██▋       | 40109/150000 [00:00<00:01, 67122.34it/s]
 31%|███▏      | 47055/150000 [00:00<00:01, 67805.17it/s]
 36%|███▌      | 53986/150000 [00:00<00:01, 68246.94it/s]
 41%|████      | 60967/150000 [00:00<00:01, 68706.59it/s]
 45%|████▌     | 67962/150000 [00:01<00:01, 69074.58it/s]
 50%|████▉     | 74933/150000 [00:01<00:01, 69263.37it/s]
 55%|█████▍    | 81898/150000 [00:01<00:00, 69377.74it/s]
 59%|█████▉    | 88854/150000 [00:01<00:00, 69431.39it/s]
 64%|██████▍   | 95855/150000 [00:01<00:00, 69600.67it/s]
 69%|██████▊   | 102822/150000 [00:01<00:00, 69619.98it/s]
 73%|███████▎  | 109785/150000 [00:01<00:00, 69622.11it/s]
 78%|███████▊  | 116757/150000 [00:01<00:00, 69650.37it/s]
 82%|████████▏ | 123709/150000 [00:01<00:00, 69608.94it/s]
 87%|████████▋ | 130737/150000 [00:01<00:00, 69806.30it/s]
 92%|█████████▏| 137722/150000 [00:02<00:00, 69816.23it/s]
 96%|█████████▋| 144727/150000 [00:02<00:00, 69883.42it/s]
100%|██████████| 150000/150000 [00:02<00:00, 68919.95it/s]

  0%|          | 0/135 [00:00<?, ?it/s]
 2216 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-23 ...
Successfully removed 1110 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-24 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 12062 new domains found

Scrapping urls...
scrapping will be done in 135 phase  where 90 urls attempt in each phase

  1%|          | 1/135 [00:18<40:55, 18.33s/it]
  1%|▏         | 2/135 [00:52<51:25, 23.20s/it]
  2%|▏         | 3/135 [01:08<46:06, 20.96s/it]
  3%|▎         | 4/135 [01:44<55:40, 25.50s/it]
  4%|▎         | 5/135 [02:15<58:39, 27.07s/it]
  4%|▍         | 6/135 [02:26<47:46, 22.22s/it]
  5%|▌         | 7/135 [02:39<41:44, 19.57s/it]
  6%|▌         | 8/135 [03:02<43:25, 20.51s/it]
  7%|▋         | 9/135 [03:13<37:18, 17.76s/it]
  7%|▋         | 10/135 [03:50<48:36, 23.33s/it]
  8%|▊         | 11/135 [04:08<45:24, 21.98s/it]
  9%|▉         | 12/135 [04:47<55:00, 26.83s/it]
 10%|▉         | 13/135 [05:20<58:28, 28.76s/it]
 10%|█         | 14/135 [05:56<1:02:44, 31.11s/it]
 11%|█         | 15/135 [06:30<1:03:51, 31.93s/it]
 12%|█▏        | 16/135 [07:12<1:09:24, 34.99s/it]
 13%|█▎        | 17/135 [07:53<1:11:52, 36.55s/it]
 13%|█▎        | 18/135 [08:09<59:31, 30.52s/it]  
 14%|█▍        | 19/135 [08:47<1:03:27, 32.82s/it]
 15%|█▍        | 20/135 [09:25<1:05:32, 34.20s/it]
 16%|█▌        | 21/135 [10:01<1:06:06, 34.80s/it]
 16%|█▋        | 22/135 [10:42<1:08:50, 36.55s/it]
 17%|█▋        | 23/135 [11:22<1:10:22, 37.70s/it]
 18%|█▊        | 24/135 [12:05<1:12:30, 39.20s/it]209.17.96.50 - - [29/May/2020 08:18:30] "GET / HTTP/1.1" 200 -

 19%|█▊        | 25/135 [12:23<1:00:11, 32.83s/it]
 19%|█▉        | 26/135 [13:02<1:03:04, 34.72s/it]
 20%|██        | 27/135 [13:35<1:01:52, 34.37s/it]
 21%|██        | 28/135 [14:11<1:02:01, 34.78s/it]
 21%|██▏       | 29/135 [14:48<1:02:22, 35.31s/it]
 22%|██▏       | 30/135 [15:27<1:03:54, 36.52s/it]
 23%|██▎       | 31/135 [16:07<1:05:07, 37.58s/it]
 24%|██▎       | 32/135 [16:47<1:05:52, 38.37s/it]
 24%|██▍       | 33/135 [17:23<1:04:09, 37.74s/it]
 25%|██▌       | 34/135 [18:08<1:07:12, 39.92s/it]
 26%|██▌       | 35/135 [18:39<1:01:59, 37.19s/it]
 27%|██▋       | 36/135 [19:11<58:28, 35.44s/it]  
 27%|██▋       | 37/135 [19:56<1:02:55, 38.53s/it]
 28%|██▊       | 38/135 [20:33<1:01:25, 37.99s/it]27.56.245.116 - - [29/May/2020 08:27:16] "GET /getClusterData/2020-05-05/2020-05-27/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:27:17] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:27:22] "GET /getClusterData/2020-05-05/2020-05-27/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:27:23] "GET /favicon.ico HTTP/1.1" 200 -

 29%|██▉       | 39/135 [21:12<1:01:15, 38.28s/it]
 30%|██▉       | 40/135 [21:52<1:01:17, 38.71s/it]
 30%|███       | 41/135 [22:36<1:03:24, 40.47s/it]
 31%|███       | 42/135 [23:16<1:02:13, 40.15s/it]
 32%|███▏      | 43/135 [23:49<58:13, 37.97s/it]  27.56.245.116 - - [29/May/2020 08:30:42] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:30:42] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 33%|███▎      | 44/135 [24:22<55:24, 36.53s/it]27.56.245.116 - - [29/May/2020 08:30:46] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:30:54] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:30:54] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:05] "GET /getClusterData/2020-05-01/2020-05-29/12 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:21] "GET /getClusterData/2020-05-01/2020-05-29/13 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:21] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:24] "GET /getClusterData/2020-05-01/2020-05-29/40 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:25] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:27] "GET /getClusterData/2020-05-01/2020-05-29/66 HTTP/1.1" 200 -

 33%|███▎      | 45/135 [25:04<57:12, 38.14s/it]27.56.245.116 - - [29/May/2020 08:31:28] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:36] "GET /getClusterData/2020-05-01/2020-05-29/22 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:36] "GET /getclusterurl/22/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:39] "GET /getClusterData/2020-05-01/2020-05-29/21 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:39] "GET /getclusterurl/21/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:44] "GET /getClusterData/2020-05-01/2020-05-29/42 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 08:31:44] "GET /getclusterurl/42/page/0 HTTP/1.1" 200 -

 34%|███▍      | 46/135 [25:43<56:59, 38.42s/it]
 35%|███▍      | 47/135 [26:26<58:38, 39.98s/it]
 36%|███▌      | 48/135 [26:44<48:05, 33.17s/it]
 36%|███▋      | 49/135 [27:18<48:06, 33.57s/it]
 37%|███▋      | 50/135 [27:37<41:13, 29.10s/it]
 38%|███▊      | 51/135 [28:15<44:42, 31.94s/it]
 39%|███▊      | 52/135 [28:44<42:37, 30.81s/it]
 39%|███▉      | 53/135 [29:20<44:31, 32.58s/it]
 40%|████      | 54/135 [30:04<48:38, 36.03s/it]
 41%|████      | 55/135 [30:40<48:02, 36.04s/it]
 41%|████▏     | 56/135 [31:11<45:06, 34.26s/it]
 42%|████▏     | 57/135 [31:42<43:29, 33.46s/it]
 43%|████▎     | 58/135 [32:22<45:32, 35.49s/it]
 44%|████▎     | 59/135 [33:02<46:24, 36.64s/it]
 44%|████▍     | 60/135 [33:36<44:54, 35.93s/it]
 45%|████▌     | 61/135 [34:13<44:40, 36.23s/it]
 46%|████▌     | 62/135 [34:43<41:53, 34.43s/it]
 47%|████▋     | 63/135 [35:17<41:11, 34.33s/it]
 47%|████▋     | 64/135 [35:56<42:15, 35.70s/it]
 48%|████▊     | 65/135 [36:18<36:52, 31.61s/it]
 49%|████▉     | 66/135 [37:02<40:40, 35.37s/it]
 50%|████▉     | 67/135 [37:33<38:27, 33.93s/it]
 50%|█████     | 68/135 [37:52<32:49, 29.40s/it]
 51%|█████     | 69/135 [38:23<32:52, 29.89s/it]
 52%|█████▏    | 70/135 [38:57<33:43, 31.14s/it]
 53%|█████▎    | 71/135 [39:33<34:43, 32.56s/it]
 53%|█████▎    | 72/135 [40:11<35:59, 34.28s/it]
 54%|█████▍    | 73/135 [40:47<36:07, 34.95s/it]
 55%|█████▍    | 74/135 [41:06<30:33, 30.06s/it]
 56%|█████▌    | 75/135 [41:28<27:34, 27.58s/it]
 56%|█████▋    | 76/135 [42:07<30:28, 31.00s/it]
 57%|█████▋    | 77/135 [42:45<32:05, 33.20s/it]
 58%|█████▊    | 78/135 [43:21<32:18, 34.01s/it]
 59%|█████▊    | 79/135 [43:43<28:18, 30.34s/it]
 59%|█████▉    | 80/135 [44:25<30:59, 33.81s/it]
 60%|██████    | 81/135 [45:10<33:26, 37.16s/it]
 61%|██████    | 82/135 [45:35<29:45, 33.69s/it]
 61%|██████▏   | 83/135 [46:06<28:22, 32.74s/it]
 62%|██████▏   | 84/135 [46:32<26:04, 30.67s/it]
 63%|██████▎   | 85/135 [47:25<31:06, 37.33s/it]
 64%|██████▎   | 86/135 [47:58<29:32, 36.16s/it]
 64%|██████▍   | 87/135 [48:33<28:45, 35.94s/it]
 65%|██████▌   | 88/135 [49:15<29:23, 37.52s/it]
 66%|██████▌   | 89/135 [49:39<25:42, 33.53s/it]
 67%|██████▋   | 90/135 [49:58<21:58, 29.30s/it]
 67%|██████▋   | 91/135 [50:16<19:00, 25.91s/it]
 68%|██████▊   | 92/135 [50:51<20:33, 28.68s/it]
 69%|██████▉   | 93/135 [51:28<21:39, 30.94s/it]
 70%|██████▉   | 94/135 [52:08<23:09, 33.89s/it]
 70%|███████   | 95/135 [52:28<19:38, 29.46s/it]
 71%|███████   | 96/135 [53:02<20:05, 30.91s/it]
 72%|███████▏  | 97/135 [53:45<21:53, 34.57s/it]
 73%|███████▎  | 98/135 [54:22<21:45, 35.28s/it]
 73%|███████▎  | 99/135 [54:56<21:01, 35.04s/it]
 74%|███████▍  | 100/135 [55:41<22:07, 37.92s/it]
 75%|███████▍  | 101/135 [55:59<18:04, 31.90s/it]
 76%|███████▌  | 102/135 [56:30<17:22, 31.58s/it]
 76%|███████▋  | 103/135 [57:11<18:19, 34.35s/it]
 77%|███████▋  | 104/135 [57:50<18:32, 35.87s/it]
 78%|███████▊  | 105/135 [58:35<19:21, 38.71s/it]
 79%|███████▊  | 106/135 [59:16<18:59, 39.28s/it]
 79%|███████▉  | 107/135 [59:35<15:31, 33.26s/it]
 80%|████████  | 108/135 [1:00:16<16:00, 35.57s/it]
 81%|████████  | 109/135 [1:00:54<15:45, 36.35s/it]
 81%|████████▏ | 110/135 [1:01:17<13:30, 32.41s/it]
 82%|████████▏ | 111/135 [1:01:57<13:47, 34.48s/it]
 83%|████████▎ | 112/135 [1:02:28<12:47, 33.39s/it]
 84%|████████▎ | 113/135 [1:03:07<12:55, 35.27s/it]
 84%|████████▍ | 114/135 [1:03:41<12:14, 34.96s/it]
 85%|████████▌ | 115/135 [1:04:25<12:29, 37.46s/it]
 86%|████████▌ | 116/135 [1:04:42<09:54, 31.28s/it]
 87%|████████▋ | 117/135 [1:05:17<09:46, 32.59s/it]
 87%|████████▋ | 118/135 [1:05:58<09:55, 35.01s/it]
 88%|████████▊ | 119/135 [1:06:30<09:04, 34.05s/it]
 89%|████████▉ | 120/135 [1:07:07<08:46, 35.08s/it]
 90%|████████▉ | 121/135 [1:07:44<08:17, 35.53s/it]
 90%|█████████ | 122/135 [1:08:02<06:35, 30.42s/it]
 91%|█████████ | 123/135 [1:08:44<06:45, 33.83s/it]
 92%|█████████▏| 124/135 [1:09:01<05:15, 28.65s/it]
 93%|█████████▎| 125/135 [1:09:35<05:03, 30.31s/it]
 93%|█████████▎| 126/135 [1:10:13<04:54, 32.67s/it]
 94%|█████████▍| 127/135 [1:10:41<04:10, 31.36s/it]
 95%|█████████▍| 128/135 [1:10:58<03:09, 27.10s/it]
 96%|█████████▌| 129/135 [1:11:33<02:55, 29.21s/it]
 96%|█████████▋| 130/135 [1:12:09<02:37, 31.41s/it]
 97%|█████████▋| 131/135 [1:12:28<01:50, 27.72s/it]
 98%|█████████▊| 132/135 [1:13:01<01:28, 29.35s/it]
 99%|█████████▊| 133/135 [1:13:40<01:04, 32.08s/it]
 99%|█████████▉| 134/135 [1:14:11<00:31, 31.72s/it]
100%|██████████| 135/135 [1:14:13<00:00, 22.89s/it]
100%|██████████| 135/135 [1:14:13<00:00, 32.99s/it]

  0%|          | 0/233711 [00:00<?, ?it/s]
  5%|▍         | 10965/233711 [00:00<00:02, 109645.92it/s]
  9%|▉         | 22101/233711 [00:00<00:01, 110152.04it/s]
 14%|█▍        | 33559/233711 [00:00<00:01, 111441.78it/s]
 19%|█▉        | 45167/233711 [00:00<00:01, 112792.63it/s]
 24%|██▍       | 56991/233711 [00:00<00:01, 114372.96it/s]
 29%|██▉       | 68910/233711 [00:00<00:01, 115775.35it/s]
 34%|███▍      | 80621/233711 [00:00<00:01, 116169.69it/s]
 40%|███▉      | 92324/233711 [00:00<00:01, 116424.25it/s]
 45%|████▍     | 104616/233711 [00:00<00:01, 118298.94it/s]
 50%|█████     | 116884/233711 [00:01<00:00, 119578.00it/s]
 55%|█████▌    | 129194/233711 [00:01<00:00, 120610.41it/s]
 61%|██████    | 141544/233711 [00:01<00:00, 121462.77it/s]
 66%|██████▌   | 153820/233711 [00:01<00:00, 121846.32it/s]
 71%|███████   | 165905/233711 [00:01<00:00, 120071.04it/s]
 76%|███████▌  | 177848/233711 [00:01<00:00, 119271.02it/s]
 81%|████████▏ | 190175/233711 [00:01<00:00, 120442.48it/s]
 87%|████████▋ | 202433/233711 [00:01<00:00, 121073.93it/s]
 92%|█████████▏| 214732/233711 [00:01<00:00, 121640.85it/s]
 97%|█████████▋| 227084/233711 [00:01<00:00, 122196.87it/s]
100%|██████████| 233711/233711 [00:01<00:00, 119215.36it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11039/150000 [00:00<00:01, 110385.10it/s]
 15%|█▍        | 22297/150000 [00:00<00:01, 111032.69it/s]
 23%|██▎       | 34022/150000 [00:00<00:01, 112826.86it/s]
 31%|███       | 46007/150000 [00:00<00:00, 114842.87it/s]
 39%|███▊      | 58045/150000 [00:00<00:00, 116448.28it/s]
 47%|████▋     | 70204/150000 [00:00<00:00, 117942.04it/s]
 55%|█████▍    | 82394/150000 [00:00<00:00, 119101.38it/s]
 63%|██████▎   | 94760/150000 [00:00<00:00, 120431.55it/s]
 71%|███████▏  | 107184/150000 [00:00<00:00, 121548.61it/s]
 80%|███████▉  | 119637/150000 [00:01<00:00, 122427.10it/s]
 88%|████████▊ | 132247/150000 [00:01<00:00, 123504.44it/s]
 97%|█████████▋| 144796/150000 [00:01<00:00, 124091.23it/s]
100%|██████████| 150000/150000 [00:01<00:00, 120745.35it/s]

0it [00:00, ?it/s]
61it [00:00, 607.90it/s]
151it [00:00, 671.07it/s]
253it [00:00, 747.39it/s]
330it [00:00, 753.93it/s]
416it [00:00, 775.38it/s]
502it [00:00, 795.14it/s]
587it [00:00, 793.82it/s]
684it [00:00, 837.61it/s]
773it [00:00, 852.42it/s]
889it [00:01, 926.07it/s]
1007it [00:01, 988.91it/s]
1130it [00:01, 1045.28it/s]
1261it [00:01, 1108.96it/s]
1410it [00:01, 1194.92it/s]
1582it [00:01, 1315.29it/s]
1820it [00:01, 1517.47it/s]
2047it [00:01, 1684.91it/s]
2235it [00:01, 1734.16it/s]
2700it [00:01, 2135.01it/s]
3057it [00:02, 1510.20it/s]

  0%|          | 0/1503 [00:00<?, ?it/s]
 76%|███████▌  | 1137/1503 [00:00<00:00, 11367.19it/s]
100%|██████████| 1503/1503 [00:00<00:00, 12481.12it/s]

  0%|          | 0/68597 [00:00<?, ?it/s]
  2%|▏         | 1080/68597 [00:00<00:06, 10752.30it/s]
  4%|▍         | 2695/68597 [00:00<00:05, 11947.65it/s]
  7%|▋         | 4460/68597 [00:00<00:04, 13229.12it/s]
 10%|▉         | 6747/68597 [00:00<00:04, 15144.35it/s]
 13%|█▎        | 8621/68597 [00:00<00:03, 16067.40it/s]
 16%|█▌        | 10692/68597 [00:00<00:03, 17225.66it/s]
 19%|█▉        | 13207/68597 [00:00<00:02, 19022.90it/s]
 23%|██▎       | 15757/68597 [00:00<00:02, 20591.19it/s]
 27%|██▋       | 18329/68597 [00:00<00:02, 21900.05it/s]
 30%|███       | 20732/68597 [00:01<00:02, 22497.23it/s]
 34%|███▎      | 23149/68597 [00:01<00:01, 22969.88it/s]
 37%|███▋      | 25505/68597 [00:01<00:01, 23142.79it/s]
 41%|████      | 28039/68597 [00:01<00:01, 23759.51it/s]
 45%|████▍     | 30651/68597 [00:01<00:01, 24419.98it/s]
 48%|████▊     | 33253/68597 [00:01<00:01, 24877.57it/s]
 52%|█████▏    | 35758/68597 [00:01<00:01, 24252.40it/s]
 56%|█████▌    | 38414/68597 [00:01<00:01, 24900.86it/s]
 60%|█████▉    | 41073/68597 [00:01<00:01, 25382.61it/s]
 64%|██████▎   | 43630/68597 [00:01<00:00, 25437.09it/s]
 67%|██████▋   | 46183/68597 [00:02<00:00, 24558.58it/s]
 71%|███████   | 48652/68597 [00:02<00:00, 24065.52it/s]
 74%|███████▍  | 51070/68597 [00:02<00:00, 23584.34it/s]
 78%|███████▊  | 53439/68597 [00:02<00:00, 23511.71it/s]
 81%|████████▏ | 55883/68597 [00:02<00:00, 23781.70it/s]
 85%|████████▌ | 58494/68597 [00:02<00:00, 24434.09it/s]
 89%|████████▉ | 61155/68597 [00:02<00:00, 25048.23it/s]
 93%|█████████▎| 63771/68597 [00:02<00:00, 25371.56it/s]
 97%|█████████▋| 66481/68597 [00:02<00:00, 25864.50it/s]
100%|██████████| 68597/68597 [00:02<00:00, 23506.61it/s]

  0%|          | 0/68597 [00:00<?, ?it/s]
  9%|▉         | 6163/68597 [00:00<00:01, 52758.70it/s]
 16%|█▌        | 11049/68597 [00:00<00:01, 49604.62it/s]
 23%|██▎       | 15777/68597 [00:00<00:01, 46657.54it/s]
 30%|██▉       | 20313/68597 [00:00<00:01, 46258.23it/s]
 35%|███▌      | 24056/68597 [00:00<00:01, 43200.19it/s]
 40%|████      | 27562/68597 [00:00<00:01, 40337.81it/s]
 45%|████▌     | 31198/68597 [00:00<00:00, 39055.49it/s]
 51%|█████     | 34809/68597 [00:00<00:00, 38121.82it/s]
 56%|█████▌    | 38372/68597 [00:00<00:00, 37335.34it/s]
 61%|██████    | 41962/68597 [00:01<00:00, 36890.53it/s]
 67%|██████▋   | 45672/68597 [00:01<00:00, 36951.19it/s]
 72%|███████▏  | 49370/68597 [00:01<00:00, 36957.01it/s]
 77%|███████▋  | 53139/68597 [00:01<00:00, 37172.54it/s]
 83%|████████▎ | 57023/68597 [00:01<00:00, 37657.46it/s]
 89%|████████▊ | 60830/68597 [00:01<00:00, 37577.81it/s]
 96%|█████████▌| 65577/68597 [00:01<00:00, 40083.52it/s]
100%|██████████| 68597/68597 [00:01<00:00, 39610.72it/s]

  0%|          | 0/3057 [00:00<?, ?it/s]
100%|██████████| 3057/3057 [00:00<00:00, 79810.70it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.14it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.00it/s]
  3%|▎         | 3/100 [00:00<00:14,  6.81it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.58it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.63it/s]
  6%|▌         | 6/100 [00:00<00:14,  6.66it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.54it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.30it/s]
  9%|▉         | 9/100 [00:01<00:14,  6.38it/s]
 10%|█         | 10/100 [00:01<00:14,  6.37it/s]
 11%|█         | 11/100 [00:01<00:13,  6.55it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.36it/s]
 13%|█▎        | 13/100 [00:02<00:13,  6.55it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.52it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.23it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.20it/s]
 17%|█▋        | 17/100 [00:02<00:13,  6.37it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.39it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.38it/s]
 20%|██        | 20/100 [00:03<00:12,  6.45it/s]
 21%|██        | 21/100 [00:03<00:12,  6.47it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.62it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.45it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.18it/s]
 25%|██▌       | 25/100 [00:03<00:12,  6.22it/s]
 26%|██▌       | 26/100 [00:04<00:12,  5.96it/s]
 27%|██▋       | 27/100 [00:04<00:12,  6.07it/s]
 28%|██▊       | 28/100 [00:04<00:12,  5.85it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.07it/s]
 30%|███       | 30/100 [00:04<00:11,  6.32it/s]
 31%|███       | 31/100 [00:04<00:11,  6.14it/s]
 32%|███▏      | 32/100 [00:05<00:10,  6.28it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.41it/s]
 34%|███▍      | 34/100 [00:05<00:10,  6.54it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.54it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.44it/s]
 37%|███▋      | 37/100 [00:05<00:10,  6.26it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.37it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.29it/s]
 40%|████      | 40/100 [00:06<00:09,  6.38it/s]
 41%|████      | 41/100 [00:06<00:09,  6.36it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.49it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.60it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.78it/s]
 45%|████▌     | 45/100 [00:07<00:08,  6.27it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.42it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.44it/s]
 48%|████▊     | 48/100 [00:07<00:08,  6.41it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.47it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.50it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.64it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.57it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.36it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.36it/s]
 55%|█████▌    | 55/100 [00:08<00:07,  6.41it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.49it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.27it/s]
 58%|█████▊    | 58/100 [00:09<00:06,  6.22it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.16it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.02it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.23it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.02it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.03it/s]
 64%|██████▍   | 64/100 [00:10<00:05,  6.20it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.22it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.46it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.52it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.75it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.65it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.70it/s]
 71%|███████   | 71/100 [00:11<00:04,  6.74it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.68it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.18it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.40it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.59it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.68it/s]
 77%|███████▋  | 77/100 [00:12<00:03,  6.54it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.50it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.40it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.20it/s]
 81%|████████  | 81/100 [00:12<00:03,  6.25it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.57it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.73it/s]
 84%|████████▍ | 84/100 [00:13<00:02,  6.55it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.67it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.79it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.87it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.71it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.61it/s]
 90%|█████████ | 90/100 [00:14<00:01,  6.56it/s]
 91%|█████████ | 91/100 [00:14<00:01,  6.50it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.40it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.41it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.22it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.31it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.60it/s]
 97%|█████████▋| 97/100 [00:15<00:00,  6.53it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.70it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.41it/s]
100%|██████████| 100/100 [00:15<00:00,  6.36it/s]
100%|██████████| 100/100 [00:15<00:00,  6.42it/s]

  0%|          | 0/68597 [00:00<?, ?it/s]
  6%|▌         | 4247/68597 [00:00<00:01, 42465.69it/s]
 14%|█▎        | 9355/68597 [00:00<00:01, 44728.57it/s]
 24%|██▎       | 16178/68597 [00:00<00:01, 49882.26it/s]
 34%|███▍      | 23492/68597 [00:00<00:00, 55141.64it/s]
 45%|████▌     | 30901/68597 [00:00<00:00, 59723.79it/s]
 57%|█████▋    | 38804/68597 [00:00<00:00, 64445.12it/s]
 69%|██████▊   | 47073/68597 [00:00<00:00, 69012.19it/s]
 81%|████████▏ | 55802/68597 [00:00<00:00, 73636.67it/s]
 95%|█████████▌| 65373/68597 [00:00<00:00, 79108.77it/s]
100%|██████████| 68597/68597 [00:00<00:00, 73631.57it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/361300 [00:00<?, ?it/s]
  3%|▎         | 11248/361300 [00:00<00:03, 112475.82it/s]
  6%|▌         | 21881/361300 [00:00<00:03, 110555.98it/s]
  9%|▉         | 33515/361300 [00:00<00:02, 112226.97it/s]
 13%|█▎        | 45532/361300 [00:00<00:02, 114497.27it/s]
 16%|█▌        | 57822/361300 [00:00<00:02, 116893.84it/s]
 19%|█▉        | 70141/361300 [00:00<00:02, 118713.03it/s]
 23%|██▎       | 82288/361300 [00:00<00:02, 119526.50it/s]
 26%|██▌       | 94763/361300 [00:00<00:02, 121044.63it/s]
 30%|██▉       | 107502/361300 [00:00<00:02, 122878.27it/s]
 33%|███▎      | 120160/361300 [00:01<00:01, 123965.63it/s]
 37%|███▋      | 132807/361300 [00:01<00:01, 124703.42it/s]
 40%|████      | 145671/361300 [00:01<00:01, 125858.02it/s]
 44%|████▍     | 158455/361300 [00:01<00:01, 126444.02it/s]
 47%|████▋     | 171293/361300 [00:01<00:01, 127017.52it/s]
 51%|█████     | 184133/361300 [00:01<00:01, 127426.99it/s]
 54%|█████▍    | 196828/361300 [00:01<00:01, 126185.35it/s]
 58%|█████▊    | 209531/361300 [00:01<00:01, 126434.36it/s]
 62%|██████▏   | 222576/361300 [00:01<00:01, 127611.37it/s]
 65%|██████▌   | 235325/361300 [00:01<00:00, 127254.64it/s]
 69%|██████▊   | 248364/361300 [00:02<00:00, 128176.90it/s]
 72%|███████▏  | 261178/361300 [00:02<00:00, 127398.46it/s]
 76%|███████▌  | 274236/361300 [00:02<00:00, 128334.76it/s]
 80%|███████▉  | 287313/361300 [00:02<00:00, 129052.51it/s]
 83%|████████▎ | 300220/361300 [00:02<00:00, 128026.34it/s]
 87%|████████▋ | 313266/361300 [00:02<00:00, 128745.25it/s]
 90%|█████████ | 326389/361300 [00:02<00:00, 129478.86it/s]
 94%|█████████▍| 339442/361300 [00:02<00:00, 129790.79it/s]
 98%|█████████▊| 352424/361300 [00:02<00:00, 128210.26it/s]
100%|██████████| 361300/361300 [00:02<00:00, 125295.24it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6480/150000 [00:00<00:02, 64790.95it/s]
  9%|▊         | 13098/150000 [00:00<00:02, 65200.00it/s]
 13%|█▎        | 19806/150000 [00:00<00:01, 65752.03it/s]
 18%|█▊        | 26550/150000 [00:00<00:01, 66248.71it/s]
 22%|██▏       | 33356/150000 [00:00<00:01, 66781.68it/s]
 27%|██▋       | 40211/150000 [00:00<00:01, 67301.36it/s]
 31%|███▏      | 47127/150000 [00:00<00:01, 67848.06it/s]
 36%|███▌      | 53997/150000 [00:00<00:01, 68099.99it/s]
 41%|████      | 60852/150000 [00:00<00:01, 68232.59it/s]
 45%|████▌     | 67758/150000 [00:01<00:01, 68477.34it/s]
 50%|████▉     | 74730/150000 [00:01<00:01, 68844.02it/s]
 54%|█████▍    | 81715/150000 [00:01<00:00, 69141.02it/s]
 59%|█████▉    | 88696/150000 [00:01<00:00, 69337.03it/s]
 64%|██████▎   | 95568/150000 [00:01<00:00, 69131.97it/s]
 68%|██████▊   | 102472/150000 [00:01<00:00, 69100.86it/s]
 73%|███████▎  | 109380/150000 [00:01<00:00, 69092.20it/s]
 78%|███████▊  | 116328/150000 [00:01<00:00, 69205.89it/s]
 82%|████████▏ | 123308/150000 [00:01<00:00, 69381.89it/s]
 87%|████████▋ | 130308/150000 [00:01<00:00, 69565.90it/s]
 92%|█████████▏| 137280/150000 [00:02<00:00, 69609.49it/s]
 96%|█████████▌| 144288/150000 [00:02<00:00, 69749.85it/s]
100%|██████████| 150000/150000 [00:02<00:00, 68756.97it/s]

  0%|          | 0/174 [00:00<?, ?it/s]
 3057 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-24 ...
Successfully removed 1503 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-25 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 15578 new domains found

Scrapping urls...
scrapping will be done in 174 phase  where 90 urls attempt in each phase

  1%|          | 1/174 [00:13<37:50, 13.12s/it]
  1%|          | 2/174 [00:42<51:33, 17.99s/it]
  2%|▏         | 3/174 [00:53<45:14, 15.87s/it]
  2%|▏         | 4/174 [01:08<43:55, 15.50s/it]
  3%|▎         | 5/174 [01:38<56:00, 19.88s/it]
  3%|▎         | 6/174 [01:57<54:53, 19.61s/it]
  4%|▍         | 7/174 [02:15<53:48, 19.33s/it]
  5%|▍         | 8/174 [02:23<43:34, 15.75s/it]
  5%|▌         | 9/174 [02:34<39:45, 14.46s/it]
  6%|▌         | 10/174 [03:00<49:11, 17.99s/it]
  6%|▋         | 11/174 [03:33<1:00:57, 22.44s/it]
  7%|▋         | 12/174 [04:11<1:12:41, 26.92s/it]
  7%|▋         | 13/174 [04:36<1:10:53, 26.42s/it]
  8%|▊         | 14/174 [05:11<1:17:37, 29.11s/it]
  9%|▊         | 15/174 [05:49<1:24:19, 31.82s/it]
  9%|▉         | 16/174 [06:01<1:07:58, 25.82s/it]
 10%|▉         | 17/174 [06:35<1:13:50, 28.22s/it]
 10%|█         | 18/174 [06:56<1:07:37, 26.01s/it]
 11%|█         | 19/174 [07:03<52:53, 20.47s/it]  
 11%|█▏        | 20/174 [07:10<41:30, 16.17s/it]
 12%|█▏        | 21/174 [07:21<37:21, 14.65s/it]
 13%|█▎        | 22/174 [07:32<34:55, 13.79s/it]
 13%|█▎        | 23/174 [08:08<51:32, 20.48s/it]
 14%|█▍        | 24/174 [08:22<45:44, 18.30s/it]
 14%|█▍        | 25/174 [08:33<40:09, 16.17s/it]
 15%|█▍        | 26/174 [08:42<34:35, 14.03s/it]
 16%|█▌        | 27/174 [09:15<48:35, 19.83s/it]
 16%|█▌        | 28/174 [09:26<41:47, 17.17s/it]
 17%|█▋        | 29/174 [09:37<36:50, 15.25s/it]
 17%|█▋        | 30/174 [09:38<26:32, 11.06s/it]
 18%|█▊        | 31/174 [09:49<26:05, 10.95s/it]
 18%|█▊        | 32/174 [10:26<44:07, 18.64s/it]
 19%|█▉        | 33/174 [10:34<36:41, 15.61s/it]
 20%|█▉        | 34/174 [11:09<49:42, 21.30s/it]
 20%|██        | 35/174 [11:20<42:03, 18.15s/it]
 21%|██        | 36/174 [11:44<46:09, 20.07s/it]
 21%|██▏       | 37/174 [12:15<53:00, 23.22s/it]
 22%|██▏       | 38/174 [12:39<53:12, 23.48s/it]
 22%|██▏       | 39/174 [13:15<1:01:26, 27.31s/it]
 23%|██▎       | 40/174 [13:47<1:04:06, 28.70s/it]
 24%|██▎       | 41/174 [14:23<1:08:38, 30.97s/it]
 24%|██▍       | 42/174 [15:02<1:13:36, 33.46s/it]
 25%|██▍       | 43/174 [15:42<1:16:57, 35.24s/it]
 25%|██▌       | 44/174 [16:15<1:14:44, 34.49s/it]
 26%|██▌       | 45/174 [16:44<1:10:46, 32.92s/it]
 26%|██▋       | 46/174 [17:22<1:13:39, 34.53s/it]
 27%|██▋       | 47/174 [18:02<1:16:13, 36.01s/it]
 28%|██▊       | 48/174 [18:41<1:17:39, 36.98s/it]
 28%|██▊       | 49/174 [19:18<1:17:21, 37.13s/it]
 29%|██▊       | 50/174 [19:47<1:11:33, 34.62s/it]
 29%|██▉       | 51/174 [20:26<1:13:23, 35.80s/it]
 30%|██▉       | 52/174 [20:50<1:05:41, 32.31s/it]
 30%|███       | 53/174 [21:26<1:07:39, 33.55s/it]
 31%|███       | 54/174 [21:44<57:45, 28.88s/it]  
 32%|███▏      | 55/174 [22:12<56:33, 28.52s/it]
 32%|███▏      | 56/174 [22:36<53:46, 27.34s/it]
 33%|███▎      | 57/174 [23:06<54:29, 27.94s/it]
 33%|███▎      | 58/174 [23:43<59:14, 30.65s/it]
 34%|███▍      | 59/174 [24:17<1:00:50, 31.74s/it]
 34%|███▍      | 60/174 [24:51<1:01:28, 32.35s/it]
 35%|███▌      | 61/174 [25:15<56:24, 29.95s/it]  
 36%|███▌      | 62/174 [25:52<59:36, 31.93s/it]
 36%|███▌      | 63/174 [26:28<1:01:36, 33.30s/it]
 37%|███▋      | 64/174 [27:05<1:02:42, 34.20s/it]
 37%|███▋      | 65/174 [27:38<1:01:34, 33.90s/it]
 38%|███▊      | 66/174 [28:14<1:02:16, 34.60s/it]
 39%|███▊      | 67/174 [28:28<50:25, 28.27s/it]  
 39%|███▉      | 68/174 [28:56<49:51, 28.22s/it]
 40%|███▉      | 69/174 [29:33<54:14, 31.00s/it]
 40%|████      | 70/174 [29:46<44:11, 25.50s/it]
 41%|████      | 71/174 [30:11<43:45, 25.49s/it]
 41%|████▏     | 72/174 [30:44<47:04, 27.69s/it]
 42%|████▏     | 73/174 [31:22<51:55, 30.84s/it]
 43%|████▎     | 74/174 [32:01<55:18, 33.18s/it]
 43%|████▎     | 75/174 [32:37<56:21, 34.16s/it]
 44%|████▎     | 76/174 [32:54<47:09, 28.87s/it]
 44%|████▍     | 77/174 [33:27<48:47, 30.18s/it]
 45%|████▍     | 78/174 [33:49<44:32, 27.83s/it]
 45%|████▌     | 79/174 [34:25<47:42, 30.13s/it]
 46%|████▌     | 80/174 [34:58<48:47, 31.15s/it]
 47%|████▋     | 81/174 [35:35<50:38, 32.67s/it]27.56.245.116 - - [29/May/2020 09:57:47] "GET /msg91.com HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 09:57:48] "GET /favicon.ico HTTP/1.1" 200 -

 47%|████▋     | 82/174 [35:55<44:28, 29.01s/it]
 48%|████▊     | 83/174 [36:31<47:16, 31.17s/it]
 48%|████▊     | 84/174 [37:10<50:18, 33.54s/it]
 49%|████▉     | 85/174 [37:48<51:38, 34.82s/it]
 49%|████▉     | 86/174 [38:25<51:50, 35.35s/it]
 50%|█████     | 87/174 [38:40<42:36, 29.39s/it]
 51%|█████     | 88/174 [38:57<36:43, 25.62s/it]
 51%|█████     | 89/174 [39:24<36:52, 26.04s/it]
 52%|█████▏    | 90/174 [39:44<33:47, 24.14s/it]
 52%|█████▏    | 91/174 [40:24<40:03, 28.96s/it]
 53%|█████▎    | 92/174 [41:05<44:24, 32.49s/it]
 53%|█████▎    | 93/174 [41:41<45:22, 33.61s/it]
 54%|█████▍    | 94/174 [42:02<39:52, 29.91s/it]
 55%|█████▍    | 95/174 [42:47<45:16, 34.39s/it]
 55%|█████▌    | 96/174 [43:14<41:37, 32.02s/it]
 56%|█████▌    | 97/174 [43:44<40:24, 31.48s/it]
 56%|█████▋    | 98/174 [44:06<36:29, 28.80s/it]
 57%|█████▋    | 99/174 [44:47<40:16, 32.22s/it]
 57%|█████▋    | 100/174 [45:27<42:54, 34.79s/it]
 58%|█████▊    | 101/174 [46:04<42:53, 35.25s/it]
 59%|█████▊    | 102/174 [46:42<43:30, 36.25s/it]
 59%|█████▉    | 103/174 [47:16<41:57, 35.46s/it]
 60%|█████▉    | 104/174 [47:44<38:48, 33.27s/it]
 60%|██████    | 105/174 [48:21<39:22, 34.24s/it]
 61%|██████    | 106/174 [48:57<39:28, 34.84s/it]
 61%|██████▏   | 107/174 [49:30<38:30, 34.49s/it]
 62%|██████▏   | 108/174 [49:53<33:50, 30.77s/it]
 63%|██████▎   | 109/174 [50:29<35:07, 32.43s/it]
 63%|██████▎   | 110/174 [50:56<32:47, 30.74s/it]
 64%|██████▍   | 111/174 [51:29<33:06, 31.53s/it]
 64%|██████▍   | 112/174 [51:46<28:05, 27.19s/it]
 65%|██████▍   | 113/174 [52:21<30:08, 29.65s/it]
 66%|██████▌   | 114/174 [52:54<30:39, 30.66s/it]
 66%|██████▌   | 115/174 [53:13<26:32, 26.99s/it]
 67%|██████▋   | 116/174 [53:31<23:36, 24.42s/it]
 67%|██████▋   | 117/174 [54:08<26:38, 28.05s/it]
 68%|██████▊   | 118/174 [54:26<23:16, 24.95s/it]
 68%|██████▊   | 119/174 [55:00<25:31, 27.84s/it]
 69%|██████▉   | 120/174 [55:36<27:16, 30.31s/it]
 70%|██████▉   | 121/174 [56:00<25:07, 28.44s/it]
 70%|███████   | 122/174 [56:28<24:30, 28.27s/it]
 71%|███████   | 123/174 [56:44<20:55, 24.62s/it]
 71%|███████▏  | 124/174 [57:04<19:22, 23.25s/it]
 72%|███████▏  | 125/174 [57:21<17:23, 21.30s/it]
 72%|███████▏  | 126/174 [57:59<21:05, 26.36s/it]
 73%|███████▎  | 127/174 [58:19<19:06, 24.39s/it]
 74%|███████▎  | 128/174 [58:50<20:09, 26.30s/it]
 74%|███████▍  | 129/174 [59:08<17:58, 23.96s/it]
 75%|███████▍  | 130/174 [59:35<18:12, 24.83s/it]
 75%|███████▌  | 131/174 [1:00:21<22:19, 31.15s/it]
 76%|███████▌  | 132/174 [1:00:52<21:50, 31.21s/it]
 76%|███████▋  | 133/174 [1:01:20<20:40, 30.25s/it]
 77%|███████▋  | 134/174 [1:01:54<20:45, 31.13s/it]
 78%|███████▊  | 135/174 [1:02:33<21:55, 33.72s/it]
 78%|███████▊  | 136/174 [1:03:12<22:15, 35.14s/it]
 79%|███████▊  | 137/174 [1:03:44<21:09, 34.31s/it]
 79%|███████▉  | 138/174 [1:04:21<20:57, 34.94s/it]
 80%|███████▉  | 139/174 [1:04:39<17:25, 29.87s/it]
 80%|████████  | 140/174 [1:05:02<15:45, 27.80s/it]
 81%|████████  | 141/174 [1:05:29<15:16, 27.76s/it]
 82%|████████▏ | 142/174 [1:05:53<14:05, 26.42s/it]
 82%|████████▏ | 143/174 [1:06:22<14:04, 27.23s/it]
 83%|████████▎ | 144/174 [1:07:01<15:30, 31.00s/it]
 83%|████████▎ | 145/174 [1:07:17<12:40, 26.24s/it]
 84%|████████▍ | 146/174 [1:07:46<12:45, 27.33s/it]
 84%|████████▍ | 147/174 [1:08:04<10:59, 24.43s/it]
 85%|████████▌ | 148/174 [1:08:45<12:40, 29.23s/it]
 86%|████████▌ | 149/174 [1:09:20<12:56, 31.06s/it]
 86%|████████▌ | 150/174 [1:09:58<13:14, 33.09s/it]
 87%|████████▋ | 151/174 [1:10:31<12:44, 33.25s/it]
 87%|████████▋ | 152/174 [1:11:01<11:46, 32.12s/it]
 88%|████████▊ | 153/174 [1:11:13<09:10, 26.21s/it]
 89%|████████▊ | 154/174 [1:11:55<10:16, 30.80s/it]
 89%|████████▉ | 155/174 [1:12:26<09:45, 30.84s/it]
 90%|████████▉ | 156/174 [1:12:55<09:06, 30.34s/it]
 90%|█████████ | 157/174 [1:13:26<08:38, 30.52s/it]
 91%|█████████ | 158/174 [1:14:04<08:44, 32.79s/it]
 91%|█████████▏| 159/174 [1:14:40<08:27, 33.83s/it]
 92%|█████████▏| 160/174 [1:15:04<07:11, 30.81s/it]
 93%|█████████▎| 161/174 [1:15:35<06:41, 30.92s/it]
 93%|█████████▎| 162/174 [1:16:14<06:41, 33.43s/it]
 94%|█████████▎| 163/174 [1:16:44<05:55, 32.30s/it]
 94%|█████████▍| 164/174 [1:17:20<05:33, 33.33s/it]
 95%|█████████▍| 165/174 [1:17:48<04:45, 31.68s/it]
 95%|█████████▌| 166/174 [1:18:16<04:04, 30.55s/it]
 96%|█████████▌| 167/174 [1:18:30<03:00, 25.86s/it]
 97%|█████████▋| 168/174 [1:18:49<02:22, 23.76s/it]
 97%|█████████▋| 169/174 [1:19:26<02:18, 27.65s/it]
 98%|█████████▊| 170/174 [1:19:48<01:43, 25.80s/it]
 98%|█████████▊| 171/174 [1:20:09<01:13, 24.50s/it]
 99%|█████████▉| 172/174 [1:20:36<00:50, 25.20s/it]
 99%|█████████▉| 173/174 [1:21:05<00:26, 26.39s/it]
100%|██████████| 174/174 [1:21:14<00:00, 21.24s/it]
100%|██████████| 174/174 [1:21:14<00:00, 28.02s/it]

  0%|          | 0/235133 [00:00<?, ?it/s]
  5%|▍         | 11082/235133 [00:00<00:02, 110811.92it/s]
  9%|▉         | 22252/235133 [00:00<00:01, 111076.25it/s]
 14%|█▍        | 33868/235133 [00:00<00:01, 112553.83it/s]
 19%|█▉        | 45308/235133 [00:00<00:01, 113100.71it/s]
 24%|██▍       | 57203/235133 [00:00<00:01, 114793.66it/s]
 29%|██▉       | 69229/235133 [00:00<00:01, 116379.01it/s]
 35%|███▍      | 81159/235133 [00:00<00:01, 117239.71it/s]
 40%|███▉      | 93073/235133 [00:00<00:01, 117802.10it/s]
 45%|████▍     | 105349/235133 [00:00<00:01, 119246.86it/s]
 50%|█████     | 117627/235133 [00:01<00:00, 120285.21it/s]
 55%|█████▌    | 129954/235133 [00:01<00:00, 121164.29it/s]
 61%|██████    | 142350/235133 [00:01<00:00, 121988.76it/s]
 66%|██████▌   | 154825/235133 [00:01<00:00, 122803.99it/s]
 71%|███████   | 167004/235133 [00:01<00:00, 121018.17it/s]
 76%|███████▌  | 179041/235133 [00:01<00:00, 119815.80it/s]
 81%|████████▏ | 191367/235133 [00:01<00:00, 120822.36it/s]
 87%|████████▋ | 203674/235133 [00:01<00:00, 121484.50it/s]
 92%|█████████▏| 215973/235133 [00:01<00:00, 121929.88it/s]
 97%|█████████▋| 228309/235133 [00:01<00:00, 122354.07it/s]
100%|██████████| 235133/235133 [00:01<00:00, 119816.38it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11134/150000 [00:00<00:01, 111339.58it/s]
 15%|█▍        | 22390/150000 [00:00<00:01, 111700.01it/s]
 23%|██▎       | 34008/150000 [00:00<00:01, 113004.59it/s]
 31%|███       | 45960/150000 [00:00<00:00, 114882.78it/s]
 39%|███▊      | 58046/150000 [00:00<00:00, 116610.63it/s]
 47%|████▋     | 70331/150000 [00:00<00:00, 118412.70it/s]
 55%|█████▌    | 82795/150000 [00:00<00:00, 120212.85it/s]
 63%|██████▎   | 95177/150000 [00:00<00:00, 121270.47it/s]
 72%|███████▏  | 107541/150000 [00:00<00:00, 121968.59it/s]
 80%|███████▉  | 119986/150000 [00:01<00:00, 122700.60it/s]
 88%|████████▊ | 132501/150000 [00:01<00:00, 123425.24it/s]
 97%|█████████▋| 144868/150000 [00:01<00:00, 123496.05it/s]
100%|██████████| 150000/150000 [00:01<00:00, 120835.64it/s]

0it [00:00, ?it/s]
43it [00:00, 428.36it/s]
104it [00:00, 467.82it/s]
165it [00:00, 502.18it/s]
246it [00:00, 565.48it/s]
314it [00:00, 588.22it/s]
387it [00:00, 620.38it/s]
458it [00:00, 644.17it/s]
525it [00:00, 650.92it/s]
604it [00:00, 687.04it/s]
712it [00:01, 770.22it/s]
827it [00:01, 852.96it/s]
919it [00:01, 871.22it/s]
1114it [00:01, 1044.39it/s]
1236it [00:01, 1063.04it/s]
1358it [00:01, 1098.99it/s]
1492it [00:01, 1161.66it/s]
1693it [00:01, 1325.44it/s]
1923it [00:01, 1507.75it/s]
2094it [00:01, 1560.43it/s]
2263it [00:02, 1574.10it/s]
2778it [00:02, 1986.37it/s]
2900it [00:02, 1335.91it/s]

  0%|          | 0/1478 [00:00<?, ?it/s]
 46%|████▌     | 678/1478 [00:00<00:00, 6371.21it/s]
100%|██████████| 1478/1478 [00:00<00:00, 8049.49it/s]

  0%|          | 0/66320 [00:00<?, ?it/s]
  0%|          | 101/66320 [00:00<01:10, 934.43it/s]
  0%|          | 130/66320 [00:00<02:01, 545.47it/s]
  0%|          | 203/66320 [00:00<01:55, 570.29it/s]
  1%|          | 332/66320 [00:00<01:41, 648.68it/s]
  1%|          | 388/66320 [00:00<02:33, 430.59it/s]
  1%|          | 435/66320 [00:00<03:11, 344.87it/s]
  1%|          | 475/66320 [00:01<05:36, 195.60it/s]
  1%|          | 506/66320 [00:01<05:23, 203.54it/s]
  1%|          | 535/66320 [00:01<04:56, 221.65it/s]
  3%|▎         | 2077/66320 [00:01<03:24, 314.64it/s]
  6%|▌         | 3716/66320 [00:01<02:20, 445.81it/s]
  9%|▉         | 6158/66320 [00:01<01:35, 631.93it/s]
 13%|█▎        | 8473/66320 [00:01<01:04, 892.32it/s]
 16%|█▋        | 10924/66320 [00:02<00:44, 1255.15it/s]
 20%|██        | 13433/66320 [00:02<00:30, 1755.42it/s]
 24%|██▍       | 15919/66320 [00:02<00:20, 2434.08it/s]
 28%|██▊       | 18388/66320 [00:02<00:14, 3336.27it/s]
 31%|███▏      | 20830/66320 [00:02<00:10, 4502.38it/s]
 35%|███▌      | 23275/66320 [00:02<00:07, 5961.47it/s]
 39%|███▉      | 25755/66320 [00:02<00:05, 7720.73it/s]
 43%|████▎     | 28261/66320 [00:02<00:03, 9742.79it/s]
 46%|████▋     | 30697/66320 [00:02<00:02, 11881.64it/s]
 50%|████▉     | 33137/66320 [00:02<00:02, 14042.19it/s]
 54%|█████▎    | 35593/66320 [00:03<00:01, 16112.19it/s]
 57%|█████▋    | 38073/66320 [00:03<00:01, 18004.04it/s]
 61%|██████    | 40559/66320 [00:03<00:01, 19627.40it/s]
 65%|██████▍   | 43068/66320 [00:03<00:01, 20997.23it/s]
 69%|██████▊   | 45581/66320 [00:03<00:00, 22086.28it/s]
 72%|███████▏  | 48068/66320 [00:03<00:00, 22020.74it/s]
 76%|███████▌  | 50469/66320 [00:03<00:00, 22580.53it/s]
 80%|████████  | 53065/66320 [00:03<00:00, 23495.59it/s]
 84%|████████▍ | 55639/66320 [00:03<00:00, 24125.23it/s]
 88%|████████▊ | 58131/66320 [00:03<00:00, 23483.66it/s]
 91%|█████████▏| 60539/66320 [00:04<00:00, 23264.07it/s]
 95%|█████████▍| 62917/66320 [00:04<00:00, 23413.31it/s]
 99%|█████████▉| 65510/66320 [00:04<00:00, 24113.69it/s]
100%|██████████| 66320/66320 [00:04<00:00, 15356.32it/s]

  0%|          | 0/66320 [00:00<?, ?it/s]
  8%|▊         | 5619/66320 [00:00<00:01, 50853.82it/s]
 15%|█▌        | 10251/66320 [00:00<00:01, 48148.11it/s]
 22%|██▏       | 14637/66320 [00:00<00:01, 45293.97it/s]
 29%|██▊       | 19048/66320 [00:00<00:01, 43064.95it/s]
 35%|███▍      | 23063/66320 [00:00<00:01, 41257.95it/s]
 41%|████      | 27108/66320 [00:00<00:00, 39895.15it/s]
 47%|████▋     | 30839/66320 [00:00<00:00, 38435.94it/s]
 52%|█████▏    | 34523/66320 [00:00<00:00, 37288.84it/s]
 58%|█████▊    | 38219/66320 [00:00<00:00, 36410.40it/s]
 63%|██████▎   | 42026/66320 [00:01<00:00, 36548.84it/s]
 69%|██████▉   | 45857/66320 [00:01<00:00, 36732.60it/s]
 75%|███████▌  | 49820/66320 [00:01<00:00, 37007.37it/s]
 81%|████████▏ | 54010/66320 [00:01<00:00, 37704.88it/s]
 88%|████████▊ | 58681/66320 [00:01<00:00, 38659.48it/s]
 96%|█████████▌| 63534/66320 [00:01<00:00, 39560.99it/s]
100%|██████████| 66320/66320 [00:01<00:00, 39494.32it/s]

  0%|          | 0/2900 [00:00<?, ?it/s]
100%|██████████| 2900/2900 [00:00<00:00, 79280.18it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:14,  6.97it/s]
  2%|▏         | 2/100 [00:00<00:14,  6.99it/s]
  3%|▎         | 3/100 [00:00<00:14,  6.84it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.58it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.68it/s]
  6%|▌         | 6/100 [00:00<00:14,  6.68it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.61it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.33it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.52it/s]
 10%|█         | 10/100 [00:01<00:13,  6.47it/s]
 11%|█         | 11/100 [00:01<00:13,  6.56it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.32it/s]
 13%|█▎        | 13/100 [00:01<00:13,  6.47it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.45it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.22it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.16it/s]
 17%|█▋        | 17/100 [00:02<00:13,  6.29it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.32it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.35it/s]
 20%|██        | 20/100 [00:03<00:12,  6.38it/s]
 21%|██        | 21/100 [00:03<00:12,  6.44it/s]
 22%|██▏       | 22/100 [00:03<00:12,  6.50it/s]
 23%|██▎       | 23/100 [00:03<00:12,  6.29it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.00it/s]
 25%|██▌       | 25/100 [00:03<00:12,  6.10it/s]
 26%|██▌       | 26/100 [00:04<00:12,  5.80it/s]
 27%|██▋       | 27/100 [00:04<00:12,  5.95it/s]
 28%|██▊       | 28/100 [00:04<00:12,  5.73it/s]
 29%|██▉       | 29/100 [00:04<00:12,  5.89it/s]
 30%|███       | 30/100 [00:04<00:11,  6.06it/s]
 31%|███       | 31/100 [00:04<00:11,  5.93it/s]
 32%|███▏      | 32/100 [00:05<00:11,  6.09it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.27it/s]
 34%|███▍      | 34/100 [00:05<00:10,  6.41it/s]
 35%|███▌      | 35/100 [00:05<00:10,  6.42it/s]
 36%|███▌      | 36/100 [00:05<00:10,  6.36it/s]
 37%|███▋      | 37/100 [00:05<00:10,  6.25it/s]
 38%|███▊      | 38/100 [00:06<00:09,  6.34it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.30it/s]
 40%|████      | 40/100 [00:06<00:09,  6.48it/s]
 41%|████      | 41/100 [00:06<00:09,  6.43it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.48it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.45it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.61it/s]
 45%|████▌     | 45/100 [00:07<00:08,  6.12it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.32it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.43it/s]
 48%|████▊     | 48/100 [00:07<00:08,  6.34it/s]
 49%|████▉     | 49/100 [00:07<00:08,  6.37it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.45it/s]
 51%|█████     | 51/100 [00:08<00:07,  6.59it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.51it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.35it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.36it/s]
 55%|█████▌    | 55/100 [00:08<00:07,  6.41it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.51it/s]
 57%|█████▋    | 57/100 [00:09<00:06,  6.23it/s]
 58%|█████▊    | 58/100 [00:09<00:06,  6.10it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.16it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.08it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.28it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.08it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.08it/s]
 64%|██████▍   | 64/100 [00:10<00:05,  6.23it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.28it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.48it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.44it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.61it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.52it/s]
 70%|███████   | 70/100 [00:11<00:04,  6.61it/s]
 71%|███████   | 71/100 [00:11<00:04,  6.67it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.65it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.18it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.38it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.58it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.62it/s]
 77%|███████▋  | 77/100 [00:12<00:03,  6.53it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.43it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.35it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.07it/s]
 81%|████████  | 81/100 [00:12<00:03,  6.14it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.44it/s]
 83%|████████▎ | 83/100 [00:13<00:02,  6.58it/s]
 84%|████████▍ | 84/100 [00:13<00:02,  6.43it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.70it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.82it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.88it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.73it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.68it/s]
 90%|█████████ | 90/100 [00:14<00:01,  6.65it/s]
 91%|█████████ | 91/100 [00:14<00:01,  6.64it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.50it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.48it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.29it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.29it/s]
 96%|█████████▌| 96/100 [00:15<00:00,  6.55it/s]
 97%|█████████▋| 97/100 [00:15<00:00,  6.48it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.69it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.35it/s]
100%|██████████| 100/100 [00:15<00:00,  6.25it/s]
100%|██████████| 100/100 [00:15<00:00,  6.37it/s]

  0%|          | 0/66320 [00:00<?, ?it/s]
  6%|▌         | 4104/66320 [00:00<00:01, 41037.01it/s]
 14%|█▍        | 9193/66320 [00:00<00:01, 43567.20it/s]
 24%|██▍       | 15933/66320 [00:00<00:01, 48737.22it/s]
 35%|███▌      | 23328/66320 [00:00<00:00, 54288.86it/s]
 47%|████▋     | 31041/66320 [00:00<00:00, 59580.83it/s]
 59%|█████▉    | 39132/66320 [00:00<00:00, 64696.43it/s]
 72%|███████▏  | 47807/66320 [00:00<00:00, 70037.23it/s]
 86%|████████▋ | 57243/66320 [00:00<00:00, 75905.94it/s]
100%|██████████| 66320/66320 [00:00<00:00, 74739.59it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/353979 [00:00<?, ?it/s]
  3%|▎         | 11287/353979 [00:00<00:03, 112865.26it/s]
  6%|▋         | 22688/353979 [00:00<00:02, 113205.01it/s]
 10%|▉         | 34368/353979 [00:00<00:02, 114258.97it/s]
 13%|█▎        | 46499/353979 [00:00<00:02, 116286.24it/s]
 17%|█▋        | 58775/353979 [00:00<00:02, 118153.52it/s]
 20%|██        | 71206/353979 [00:00<00:02, 119933.60it/s]
 24%|██▎       | 83663/353979 [00:00<00:02, 121286.05it/s]
 27%|██▋       | 96092/353979 [00:00<00:02, 122170.28it/s]
 31%|███       | 108718/353979 [00:00<00:01, 123367.41it/s]
 34%|███▍      | 121421/353979 [00:01<00:01, 124442.72it/s]
 38%|███▊      | 134325/353979 [00:01<00:01, 125784.80it/s]
 42%|████▏     | 147142/353979 [00:01<00:01, 126486.74it/s]
 45%|████▌     | 160116/353979 [00:01<00:01, 127444.87it/s]
 49%|████▉     | 172848/353979 [00:01<00:01, 127405.41it/s]
 52%|█████▏    | 185723/353979 [00:01<00:01, 127805.68it/s]
 56%|█████▌    | 198454/353979 [00:01<00:01, 126895.86it/s]
 60%|█████▉    | 211605/353979 [00:01<00:01, 128244.79it/s]
 63%|██████▎   | 224577/353979 [00:01<00:01, 128680.48it/s]
 67%|██████▋   | 237481/353979 [00:01<00:00, 128787.89it/s]
 71%|███████   | 250351/353979 [00:02<00:00, 128640.03it/s]
 74%|███████▍  | 263494/353979 [00:02<00:00, 129462.17it/s]
 78%|███████▊  | 276437/353979 [00:02<00:00, 129115.78it/s]
 82%|████████▏ | 289638/353979 [00:02<00:00, 129970.49it/s]
 85%|████████▌ | 302635/353979 [00:02<00:00, 129227.79it/s]
 89%|████████▉ | 315819/353979 [00:02<00:00, 129998.08it/s]
 93%|█████████▎| 328915/353979 [00:02<00:00, 130284.83it/s]
 97%|█████████▋| 341945/353979 [00:02<00:00, 129037.27it/s]
100%|██████████| 353979/353979 [00:02<00:00, 126420.16it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6412/150000 [00:00<00:02, 64119.60it/s]
  9%|▊         | 13042/150000 [00:00<00:02, 64756.95it/s]
 13%|█▎        | 19715/150000 [00:00<00:01, 65336.34it/s]
 18%|█▊        | 26418/150000 [00:00<00:01, 65833.69it/s]
 22%|██▏       | 33227/150000 [00:00<00:01, 66492.70it/s]
 27%|██▋       | 40095/150000 [00:00<00:01, 67133.06it/s]
 31%|███▏      | 47053/150000 [00:00<00:01, 67848.52it/s]
 36%|███▌      | 54018/150000 [00:00<00:01, 68378.26it/s]
 41%|████      | 60998/150000 [00:00<00:01, 68795.58it/s]
 45%|████▌     | 67950/150000 [00:01<00:01, 69010.35it/s]
 50%|████▉     | 74970/150000 [00:01<00:01, 69361.93it/s]
 55%|█████▍    | 82004/150000 [00:01<00:00, 69651.02it/s]
 59%|█████▉    | 89009/150000 [00:01<00:00, 69768.32it/s]
 64%|██████▍   | 96014/150000 [00:01<00:00, 69851.01it/s]
 69%|██████▊   | 103003/150000 [00:01<00:00, 69861.67it/s]
 73%|███████▎  | 110042/150000 [00:01<00:00, 70017.71it/s]
 78%|███████▊  | 117055/150000 [00:01<00:00, 70049.93it/s]
 83%|████████▎ | 124069/150000 [00:01<00:00, 70076.30it/s]
 87%|████████▋ | 131101/150000 [00:01<00:00, 70147.35it/s]
 92%|█████████▏| 138109/150000 [00:02<00:00, 70073.33it/s]
 97%|█████████▋| 145112/150000 [00:02<00:00, 69953.98it/s]
100%|██████████| 150000/150000 [00:02<00:00, 69097.32it/s]

  0%|          | 0/120 [00:00<?, ?it/s]
 2900 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-25 ...
Successfully removed 1478 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-26 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 10712 new domains found

Scrapping urls...
scrapping will be done in 120 phase  where 90 urls attempt in each phase

  1%|          | 1/120 [00:11<22:55, 11.56s/it]
  2%|▏         | 2/120 [00:34<29:29, 15.00s/it]
  2%|▎         | 3/120 [00:48<28:28, 14.60s/it]
  3%|▎         | 4/120 [01:05<29:29, 15.25s/it]
  4%|▍         | 5/120 [01:19<28:31, 14.88s/it]
  5%|▌         | 6/120 [01:49<37:04, 19.51s/it]
  6%|▌         | 7/120 [02:06<35:13, 18.70s/it]
  7%|▋         | 8/120 [02:17<30:58, 16.59s/it]
  8%|▊         | 9/120 [02:55<42:30, 22.97s/it]
  8%|▊         | 10/120 [03:06<35:36, 19.42s/it]
  9%|▉         | 11/120 [03:35<40:34, 22.33s/it]
 10%|█         | 12/120 [04:12<47:51, 26.59s/it]
 11%|█         | 13/120 [04:35<45:16, 25.39s/it]
 12%|█▏        | 14/120 [04:50<39:45, 22.51s/it]
 12%|█▎        | 15/120 [05:31<49:01, 28.02s/it]
 13%|█▎        | 16/120 [06:10<54:12, 31.27s/it]5.188.206.50 - - [29/May/2020 10:51:12] code 400, message Bad HTTP/0.9 request type ('\x03\x00\x00/*à\x00\x00\x00\x00\x00Cookie:')
5.188.206.50 - - [29/May/2020 10:51:12] "/*àCookie: mstshash=Administr" HTTPStatus.BAD_REQUEST -

 14%|█▍        | 17/120 [06:50<58:22, 34.01s/it]
 15%|█▌        | 18/120 [07:27<58:58, 34.69s/it]
 16%|█▌        | 19/120 [08:07<1:00:56, 36.21s/it]
 17%|█▋        | 20/120 [08:43<1:00:42, 36.42s/it]
 18%|█▊        | 21/120 [09:24<1:02:17, 37.75s/it]
 18%|█▊        | 22/120 [10:01<1:00:55, 37.30s/it]
 19%|█▉        | 23/120 [10:45<1:03:44, 39.43s/it]
 20%|██        | 24/120 [11:21<1:01:35, 38.49s/it]
 21%|██        | 25/120 [12:04<1:03:10, 39.90s/it]
 22%|██▏       | 26/120 [12:41<1:00:50, 38.83s/it]
 22%|██▎       | 27/120 [13:16<58:32, 37.77s/it]  
 23%|██▎       | 28/120 [13:56<58:58, 38.46s/it]
 24%|██▍       | 29/120 [14:32<57:21, 37.82s/it]
 25%|██▌       | 30/120 [14:59<51:37, 34.41s/it]
 26%|██▌       | 31/120 [15:35<51:56, 35.02s/it]
 27%|██▋       | 32/120 [16:16<53:50, 36.72s/it]
 28%|██▊       | 33/120 [16:48<51:14, 35.34s/it]
 28%|██▊       | 34/120 [17:28<52:43, 36.78s/it]
 29%|██▉       | 35/120 [18:08<53:31, 37.79s/it]
 30%|███       | 36/120 [18:51<54:56, 39.24s/it]
 31%|███       | 37/120 [19:09<45:31, 32.91s/it]
 32%|███▏      | 38/120 [19:47<46:47, 34.24s/it]
 32%|███▎      | 39/120 [20:29<49:24, 36.60s/it]
 33%|███▎      | 40/120 [20:56<45:15, 33.95s/it]
 34%|███▍      | 41/120 [21:32<45:30, 34.57s/it]
 35%|███▌      | 42/120 [21:56<40:51, 31.42s/it]
 36%|███▌      | 43/120 [22:19<37:00, 28.84s/it]
 37%|███▋      | 44/120 [23:00<41:12, 32.53s/it]
 38%|███▊      | 45/120 [23:40<43:11, 34.56s/it]
 38%|███▊      | 46/120 [24:04<38:37, 31.32s/it]
 39%|███▉      | 47/120 [24:41<40:29, 33.28s/it]
 40%|████      | 48/120 [25:18<40:59, 34.16s/it]
 41%|████      | 49/120 [25:53<40:51, 34.52s/it]
 42%|████▏     | 50/120 [26:37<43:26, 37.24s/it]
 42%|████▎     | 51/120 [26:55<36:11, 31.46s/it]
 43%|████▎     | 52/120 [27:16<32:09, 28.37s/it]
 44%|████▍     | 53/120 [27:52<34:18, 30.72s/it]
 45%|████▌     | 54/120 [28:15<31:09, 28.33s/it]
 46%|████▌     | 55/120 [29:00<36:09, 33.38s/it]
 47%|████▋     | 56/120 [29:24<32:37, 30.58s/it]
 48%|████▊     | 57/120 [29:48<30:00, 28.58s/it]
 48%|████▊     | 58/120 [30:29<33:33, 32.48s/it]
 49%|████▉     | 59/120 [31:00<32:30, 31.98s/it]27.56.245.116 - - [29/May/2020 11:15:34] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:15:34] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:15:46] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:15:47] "GET /getclusterurl/page/0 HTTP/1.1" 200 -

 50%|█████     | 60/120 [31:21<28:42, 28.71s/it]27.56.245.116 - - [29/May/2020 11:15:50] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:15:52] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:15:59] "GET /getClusterData/2020-05-02/2020-05-29/88 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:20] "GET /getClusterData/2020-05-02/2020-05-29/89 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:20] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -

 51%|█████     | 61/120 [31:53<29:06, 29.59s/it]27.56.245.116 - - [29/May/2020 11:16:22] "GET /getClusterData/2020-05-02/2020-05-29/90 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:22] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:24] "GET /getClusterData/2020-05-02/2020-05-29/91 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:24] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:28] "GET /getClusterData/2020-05-02/2020-05-29/92 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:29] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:32] "GET /getClusterData/2020-05-02/2020-05-29/93 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:32] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:35] "GET /getClusterData/2020-05-02/2020-05-29/94 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:35] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:38] "GET /getClusterData/2020-05-02/2020-05-29/95 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:38] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:40] "GET /getClusterData/2020-05-02/2020-05-29/96 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:40] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:43] "GET /getClusterData/2020-05-02/2020-05-29/97 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:43] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:45] "GET /getClusterData/2020-05-02/2020-05-29/98 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:16:45] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -

 52%|█████▏    | 62/120 [32:27<29:53, 30.92s/it]27.56.245.116 - - [29/May/2020 11:17:02] "GET /getClusterData/2020-05-02/2020-05-29/99 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:02] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:06] "GET /getClusterData/2020-05-02/2020-05-29/100 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:06] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:13] "GET /getClusterData/2020-05-02/2020-05-29/1 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:13] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:15] "GET /getClusterData/2020-05-02/2020-05-29/2 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:15] "GET /getclusterurl/2/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:23] "GET /getClusterData/2020-05-02/2020-05-29/3 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:23] "GET /getclusterurl/3/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:28] "GET /getClusterData/2020-05-02/2020-05-29/4 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:28] "GET /getclusterurl/4/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:30] "GET /getClusterData/2020-05-02/2020-05-29/5 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:30] "GET /getclusterurl/5/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:32] "GET /getClusterData/2020-05-02/2020-05-29/6 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:32] "GET /getclusterurl/6/page/0 HTTP/1.1" 200 -

 52%|█████▎    | 63/120 [33:05<31:26, 33.10s/it]27.56.245.116 - - [29/May/2020 11:17:35] "GET /getClusterData/2020-05-02/2020-05-29/7 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:35] "GET /getclusterurl/7/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:36] "GET /getClusterData/2020-05-02/2020-05-29/8 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:37] "GET /getclusterurl/8/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:48] "GET /getclusterurl/8/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:53] "GET /getOneDayClusterData/2020-05-29/9 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:53] "GET /getclusterurl/9/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:55] "GET /getOneDayClusterData/2020-05-29/8 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:17:56] "GET /getclusterurl/8/page/0 HTTP/1.1" 200 -

 53%|█████▎    | 64/120 [33:30<28:28, 30.51s/it]27.56.245.116 - - [29/May/2020 11:18:04] "GET /getClusterData/2020-05-02/2020-05-29/9 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:05] "GET /getclusterurl/9/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:09] "GET /getClusterData/2020-05-02/2020-05-29/10 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:09] "GET /getclusterurl/10/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:11] "GET /getClusterData/2020-05-02/2020-05-29/11 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:11] "GET /getclusterurl/11/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:15] "GET /getClusterData/2020-05-02/2020-05-29/12 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:15] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:19] "GET /getClusterData/2020-05-02/2020-05-29/13 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:19] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:21] "GET /getClusterData/2020-05-02/2020-05-29/14 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:21] "GET /getclusterurl/14/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:23] "GET /getClusterData/2020-05-02/2020-05-29/15 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:23] "GET /getclusterurl/15/page/0 HTTP/1.1" 200 -

 54%|█████▍    | 65/120 [33:56<26:51, 29.29s/it]27.56.245.116 - - [29/May/2020 11:18:24] "GET /getClusterData/2020-05-02/2020-05-29/16 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:25] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:26] "GET /getClusterData/2020-05-02/2020-05-29/17 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:26] "GET /getclusterurl/17/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:28] "GET /getClusterData/2020-05-02/2020-05-29/18 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:28] "GET /getclusterurl/18/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:30] "GET /getClusterData/2020-05-02/2020-05-29/17 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:30] "GET /getclusterurl/17/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:43] "GET /getClusterData/2020-05-02/2020-05-29/18 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:44] "GET /getclusterurl/18/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:46] "GET /getClusterData/2020-05-02/2020-05-29/19 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:46] "GET /getclusterurl/19/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:48] "GET /getClusterData/2020-05-02/2020-05-29/20 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:48] "GET /getclusterurl/20/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:50] "GET /getClusterData/2020-05-02/2020-05-29/21 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:50] "GET /getclusterurl/21/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:58] "GET /getClusterData/2020-05-02/2020-05-29/22 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:18:58] "GET /getclusterurl/22/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:04] "GET /getClusterData/2020-05-02/2020-05-29/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:04] "GET /getclusterurl/23/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:06] "GET /getClusterData/2020-05-02/2020-05-29/24 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:06] "GET /getclusterurl/24/page/0 HTTP/1.1" 200 -

 55%|█████▌    | 66/120 [34:40<30:15, 33.62s/it]27.56.245.116 - - [29/May/2020 11:19:08] "GET /getClusterData/2020-05-02/2020-05-29/25 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:09] "GET /getclusterurl/25/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:11] "GET /getClusterData/2020-05-02/2020-05-29/26 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:11] "GET /getclusterurl/26/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:27] "GET /getClusterData/2020-05-02/2020-05-29/27 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:29] "GET /getclusterurl/27/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:30] "GET /getClusterData/2020-05-02/2020-05-29/28 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:33] "GET /getclusterurl/28/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:34] "GET /getClusterData/2020-05-02/2020-05-29/29 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:40] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -

 56%|█████▌    | 67/120 [35:13<29:40, 33.59s/it]27.56.245.116 - - [29/May/2020 11:19:46] "GET /getOneDayClusterData/2020-05-29/30 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:50] "GET /getOneDayClusterData/2020-05-29/29 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:50] "GET /getclusterurl/30/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:19:50] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -

 57%|█████▋    | 68/120 [35:46<28:57, 33.41s/it]
 57%|█████▊    | 69/120 [36:20<28:23, 33.41s/it]27.56.245.116 - - [29/May/2020 11:20:51] "GET /getOneDayClusterData/2020-05-29/30 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:20:51] "GET /getclusterurl/30/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:20:52] "GET /getOneDayClusterData/2020-05-29/29 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:20:52] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:00] "GET /getOneDayClusterData/2020-05-29/30 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:01] "GET /getclusterurl/30/page/0 HTTP/1.1" 200 -

 58%|█████▊    | 70/120 [36:41<24:42, 29.65s/it]27.56.245.116 - - [29/May/2020 11:21:27] "GET /getOneDayClusterData/2020-05-29/31 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:27] "GET /getclusterurl/31/page/0 HTTP/1.1" 200 -

 59%|█████▉    | 71/120 [37:16<25:33, 31.29s/it]27.56.245.116 - - [29/May/2020 11:21:47] "GET /getClusterData/2020-05-02/2020-05-29/32 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:47] "GET /getclusterurl/32/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:49] "GET /getClusterData/2020-05-02/2020-05-29/33 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:49] "GET /getclusterurl/33/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:51] "GET /getClusterData/2020-05-02/2020-05-29/34 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:51] "GET /getclusterurl/34/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:53] "GET /getClusterData/2020-05-02/2020-05-29/35 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:53] "GET /getclusterurl/35/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:55] "GET /getClusterData/2020-05-02/2020-05-29/36 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:55] "GET /getclusterurl/36/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:57] "GET /getClusterData/2020-05-02/2020-05-29/37 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:57] "GET /getclusterurl/37/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:59] "GET /getClusterData/2020-05-02/2020-05-29/38 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:21:59] "GET /getclusterurl/38/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:00] "GET /getClusterData/2020-05-02/2020-05-29/39 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:00] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:02] "GET /getClusterData/2020-05-02/2020-05-29/40 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:02] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:03] "GET /getClusterData/2020-05-02/2020-05-29/39 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:03] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:10] "GET /getOneDayClusterData/2020-05-29/38 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:10] "GET /getclusterurl/38/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:11] "GET /getOneDayClusterData/2020-05-29/39 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:11] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -

 60%|██████    | 72/120 [37:48<25:23, 31.73s/it]27.56.245.116 - - [29/May/2020 11:22:17] "GET /getClusterData/2020-05-02/2020-05-29/40 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:17] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:20] "GET /getClusterData/2020-05-02/2020-05-29/41 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:20] "GET /getclusterurl/41/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:24] "GET /getClusterData/2020-05-02/2020-05-29/42 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:25] "GET /getclusterurl/42/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:27] "GET /getClusterData/2020-05-02/2020-05-29/43 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:27] "GET /getclusterurl/43/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:29] "GET /getClusterData/2020-05-02/2020-05-29/44 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:29] "GET /getclusterurl/44/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:31] "GET /getClusterData/2020-05-02/2020-05-29/45 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:31] "GET /getclusterurl/45/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:33] "GET /getClusterData/2020-05-02/2020-05-29/46 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:33] "GET /getclusterurl/46/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:35] "GET /getClusterData/2020-05-02/2020-05-29/47 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:35] "GET /getclusterurl/47/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:37] "GET /getClusterData/2020-05-02/2020-05-29/48 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:37] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:46] "GET /getOneDayClusterData/2020-05-29/47 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:46] "GET /getclusterurl/47/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:46] "GET /getOneDayClusterData/2020-05-29/48 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:46] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:55] "GET /getClusterData/2020-05-02/2020-05-29/49 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:22:56] "GET /getclusterurl/49/page/0 HTTP/1.1" 200 -

 61%|██████    | 73/120 [38:28<26:45, 34.16s/it]27.56.245.116 - - [29/May/2020 11:23:03] "GET /getOneDayClusterData/2020-05-29/48 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:04] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:06] "GET /getOneDayClusterData/2020-05-29/49 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:06] "GET /getclusterurl/49/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:14] "GET /getClusterData/2020-05-02/2020-05-29/50 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:14] "GET /getclusterurl/50/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:18] "GET /getClusterData/2020-05-02/2020-05-29/51 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:18] "GET /getclusterurl/51/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:20] "GET /getClusterData/2020-05-02/2020-05-29/52 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:20] "GET /getclusterurl/52/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:22] "GET /getClusterData/2020-05-02/2020-05-29/53 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:22] "GET /getclusterurl/53/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:24] "GET /getClusterData/2020-05-02/2020-05-29/54 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:24] "GET /getclusterurl/54/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:26] "GET /getClusterData/2020-05-02/2020-05-29/55 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:26] "GET /getclusterurl/55/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:28] "GET /getClusterData/2020-05-02/2020-05-29/56 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:28] "GET /getclusterurl/56/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:29] "GET /getClusterData/2020-05-02/2020-05-29/57 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:29] "GET /getclusterurl/57/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:31] "GET /getClusterData/2020-05-02/2020-05-29/58 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:31] "GET /getclusterurl/58/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:33] "GET /getClusterData/2020-05-02/2020-05-29/59 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:33] "GET /getclusterurl/59/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:36] "GET /getClusterData/2020-05-02/2020-05-29/60 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:36] "GET /getclusterurl/60/page/0 HTTP/1.1" 200 -

 62%|██████▏   | 74/120 [39:08<27:28, 35.83s/it]27.56.245.116 - - [29/May/2020 11:23:38] "GET /getClusterData/2020-05-02/2020-05-29/61 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:39] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:40] "GET /getClusterData/2020-05-02/2020-05-29/62 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:41] "GET /getclusterurl/62/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:42] "GET /getClusterData/2020-05-02/2020-05-29/63 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:42] "GET /getclusterurl/63/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:51] "GET /getOneDayClusterData/2020-05-29/62 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:51] "GET /getclusterurl/62/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:51] "GET /getOneDayClusterData/2020-05-29/63 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:23:52] "GET /getclusterurl/63/page/0 HTTP/1.1" 200 -

 62%|██████▎   | 75/120 [39:39<25:53, 34.53s/it]27.56.245.116 - - [29/May/2020 11:24:11] "GET /getClusterData/2020-05-02/2020-05-29/64 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:11] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:15] "GET /getClusterData/2020-05-02/2020-05-29/65 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:15] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:17] "GET /getClusterData/2020-05-02/2020-05-29/66 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:17] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:19] "GET /getClusterData/2020-05-02/2020-05-29/67 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:19] "GET /getclusterurl/67/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:27] "GET /getClusterData/2020-05-02/2020-05-29/68 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:27] "GET /getclusterurl/68/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:33] "GET /getClusterData/2020-05-02/2020-05-29/69 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:24:33] "GET /getclusterurl/69/page/0 HTTP/1.1" 200 -

 63%|██████▎   | 76/120 [40:14<25:13, 34.39s/it]27.56.245.116 - - [29/May/2020 11:25:04] "GET /getClusterData/2020-05-02/2020-05-29/70 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:04] "GET /getclusterurl/70/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:06] "GET /getClusterData/2020-05-02/2020-05-29/71 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:06] "GET /getclusterurl/71/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:11] "GET /getClusterData/2020-05-02/2020-05-29/72 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:11] "GET /getclusterurl/72/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:13] "GET /getClusterData/2020-05-02/2020-05-29/73 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:13] "GET /getclusterurl/73/page/0 HTTP/1.1" 200 -

 64%|██████▍   | 77/120 [40:51<25:14, 35.22s/it]27.56.245.116 - - [29/May/2020 11:25:22] "GET /getClusterData/2020-05-02/2020-05-29/74 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:22] "GET /getclusterurl/74/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:27] "GET /getClusterData/2020-05-02/2020-05-29/75 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:29] "GET /getClusterData/2020-05-02/2020-05-29/76 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:30] "GET /getclusterurl/75/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:30] "GET /getclusterurl/76/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:37] "GET /getOneDayClusterData/2020-05-29/75 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:37] "GET /getclusterurl/75/page/0 HTTP/1.1" 200 -

 65%|██████▌   | 78/120 [41:09<21:10, 30.25s/it]27.56.245.116 - - [29/May/2020 11:25:52] "GET /getOneDayClusterData/2020-05-29/76 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:25:53] "GET /getclusterurl/76/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:26:01] "GET /getOneDayClusterData/2020-05-29/77 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:26:01] "GET /getclusterurl/77/page/0 HTTP/1.1" 200 -

 66%|██████▌   | 79/120 [41:55<23:48, 34.85s/it]
 67%|██████▋   | 80/120 [42:31<23:29, 35.23s/it]27.56.245.116 - - [29/May/2020 11:27:03] "GET /getClusterData/2020-05-02/2020-05-29/78 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:03] "GET /getclusterurl/78/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:06] "GET /getClusterData/2020-05-02/2020-05-29/79 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:06] "GET /getclusterurl/79/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:08] "GET /getClusterData/2020-05-02/2020-05-29/80 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:08] "GET /getclusterurl/80/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:10] "GET /getClusterData/2020-05-02/2020-05-29/81 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:10] "GET /getclusterurl/81/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:12] "GET /getClusterData/2020-05-02/2020-05-29/82 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:12] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:13] "GET /getClusterData/2020-05-02/2020-05-29/83 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:14] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:16] "GET /getClusterData/2020-05-02/2020-05-29/84 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:16] "GET /getclusterurl/84/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:20] "GET /getClusterData/2020-05-02/2020-05-29/85 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:20] "GET /getclusterurl/85/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:26] "GET /getOneDayClusterData/2020-05-29/86 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:26] "GET /getclusterurl/86/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:26] "GET /getOneDayClusterData/2020-05-29/85 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:26] "GET /getclusterurl/85/page/0 HTTP/1.1" 200 -

 68%|██████▊   | 81/120 [43:07<23:05, 35.52s/it]27.56.245.116 - - [29/May/2020 11:27:52] "GET /getClusterData/2020-05-02/2020-05-29/86 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:52] "GET /getclusterurl/86/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:54] "GET /getClusterData/2020-05-02/2020-05-29/87 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:54] "GET /getclusterurl/87/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:57] "GET /getClusterData/2020-05-02/2020-05-29/88 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:57] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:59] "GET /getClusterData/2020-05-02/2020-05-29/89 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:27:59] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:01] "GET /getClusterData/2020-05-02/2020-05-29/90 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:01] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:01] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:01] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:02] "GET /getClusterData/2020-05-02/2020-05-29/91 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:02] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:04] "GET /getClusterData/2020-05-02/2020-05-29/92 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:04] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:06] "GET /getClusterData/2020-05-02/2020-05-29/93 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:06] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:07] "GET /getClusterData/2020-05-02/2020-05-29/94 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:07] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:09] "GET /getClusterData/2020-05-02/2020-05-29/95 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:09] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -

 68%|██████▊   | 82/120 [43:41<22:11, 35.05s/it]27.56.245.116 - - [29/May/2020 11:28:11] "GET /getClusterData/2020-05-02/2020-05-29/96 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:11] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:12] "GET /getClusterData/2020-05-02/2020-05-29/97 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:12] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:15] "GET /getClusterData/2020-05-02/2020-05-29/98 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:15] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:17] "GET /getClusterData/2020-05-02/2020-05-29/96 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:18] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:21] "GET /getOneDayClusterData/2020-05-29/95 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:21] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:22] "GET /getOneDayClusterData/2020-05-29/96 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:28:22] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -

 69%|██████▉   | 83/120 [44:12<20:52, 33.84s/it]27.56.245.116 - - [29/May/2020 11:29:01] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:08] "GET /getOneDayClusterData/2020-05-29/39 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:08] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -

 70%|███████   | 84/120 [44:47<20:28, 34.13s/it]27.56.245.116 - - [29/May/2020 11:29:28] "GET /getClusterData/2020-04-03/2020-05-29/39 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:42] "GET /getClusterData/2020-04-03/2020-05-30/39 HTTP/1.1" 200 -

 71%|███████   | 85/120 [45:20<19:44, 33.84s/it]27.56.245.116 - - [29/May/2020 11:29:50] "GET /getClusterData/2020-04-03/2020-05-30/1 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:50] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:53] "GET /getClusterData/2020-04-03/2020-05-30/2 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:53] "GET /getclusterurl/2/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:55] "GET /getClusterData/2020-04-03/2020-05-30/13 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:29:56] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:12] "GET /getClusterData/2020-04-03/2020-05-30/100 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:12] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:15] "GET /getClusterData/2020-04-03/2020-05-30/99 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:15] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:18] "GET /getClusterData/2020-04-03/2020-05-30/98 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:18] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:20] "GET /getClusterData/2020-04-03/2020-05-30/97 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:21] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -

 72%|███████▏  | 86/120 [45:54<19:10, 33.83s/it]27.56.245.116 - - [29/May/2020 11:30:31] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:30:31] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 72%|███████▎  | 87/120 [46:31<19:05, 34.71s/it]
 73%|███████▎  | 88/120 [47:07<18:45, 35.16s/it]
 74%|███████▍  | 89/120 [47:34<16:51, 32.62s/it]27.56.245.116 - - [29/May/2020 11:32:11] "GET /getClusterData/2020-04-05/2020-05-30/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:32:15] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:32:19] "GET /getClusterData/2020-05-05/2020-05-30/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:32:21] "GET /favicon.ico HTTP/1.1" 200 -

 75%|███████▌  | 90/120 [48:04<16:02, 32.09s/it]
 76%|███████▌  | 91/120 [48:26<14:00, 28.97s/it]27.56.245.116 - - [29/May/2020 11:33:18] "GET /getOneDayClusterData/2020-05-22/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:33:19] "GET /favicon.ico HTTP/1.1" 200 -

 77%|███████▋  | 92/120 [49:00<14:13, 30.47s/it]
 78%|███████▊  | 93/120 [49:20<12:18, 27.35s/it]
 78%|███████▊  | 94/120 [49:44<11:19, 26.14s/it]
 79%|███████▉  | 95/120 [50:05<10:14, 24.59s/it]
 80%|████████  | 96/120 [50:35<10:29, 26.25s/it]
 81%|████████  | 97/120 [51:05<10:34, 27.58s/it]
 82%|████████▏ | 98/120 [51:41<11:03, 30.16s/it]
 82%|████████▎ | 99/120 [52:13<10:41, 30.55s/it]27.56.245.116 - - [29/May/2020 11:36:59] "GET /getClusterData/2020-05-05/2020-05-27/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:37:01] "GET /favicon.ico HTTP/1.1" 200 -

 83%|████████▎ | 100/120 [52:41<09:58, 29.94s/it]27.56.245.116 - - [29/May/2020 11:37:44] "GET /getOneDayClusterData/2020-05-07/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 11:37:46] "GET /favicon.ico HTTP/1.1" 200 -

 84%|████████▍ | 101/120 [53:18<10:06, 31.95s/it]
 85%|████████▌ | 102/120 [53:43<08:58, 29.89s/it]
 86%|████████▌ | 103/120 [54:08<08:00, 28.25s/it]
 87%|████████▋ | 104/120 [54:28<06:55, 25.94s/it]
 88%|████████▊ | 105/120 [55:06<07:21, 29.45s/it]
 88%|████████▊ | 106/120 [55:43<07:26, 31.86s/it]
 89%|████████▉ | 107/120 [56:18<07:05, 32.75s/it]
 90%|█████████ | 108/120 [56:40<05:54, 29.55s/it]
 91%|█████████ | 109/120 [57:03<05:03, 27.55s/it]
 92%|█████████▏| 110/120 [57:43<05:13, 31.38s/it]
 92%|█████████▎| 111/120 [58:12<04:35, 30.58s/it]
 93%|█████████▎| 112/120 [58:31<03:36, 27.03s/it]
 94%|█████████▍| 113/120 [59:14<03:42, 31.82s/it]
 95%|█████████▌| 114/120 [59:37<02:56, 29.36s/it]
 96%|█████████▌| 115/120 [1:00:00<02:16, 27.35s/it]
 97%|█████████▋| 116/120 [1:00:36<01:59, 30.00s/it]
 98%|█████████▊| 117/120 [1:01:14<01:36, 32.33s/it]
 98%|█████████▊| 118/120 [1:01:50<01:06, 33.35s/it]
 99%|█████████▉| 119/120 [1:02:31<00:35, 35.83s/it]
100%|██████████| 120/120 [1:02:35<00:00, 26.05s/it]
100%|██████████| 120/120 [1:02:35<00:00, 31.29s/it]

  0%|          | 0/230406 [00:00<?, ?it/s]
  5%|▍         | 10911/230406 [00:00<00:02, 109102.04it/s]
 10%|▉         | 22050/230406 [00:00<00:01, 109777.14it/s]
 15%|█▍        | 33700/230406 [00:00<00:01, 111709.59it/s]
 20%|█▉        | 45291/230406 [00:00<00:01, 112936.72it/s]
 25%|██▍       | 57245/230406 [00:00<00:01, 114838.82it/s]
 30%|███       | 69391/230406 [00:00<00:01, 116747.99it/s]
 35%|███▌      | 81498/230406 [00:00<00:01, 118011.21it/s]
 41%|████      | 93684/230406 [00:00<00:01, 119137.53it/s]
 46%|████▌     | 106116/230406 [00:00<00:01, 120643.26it/s]
 51%|█████▏    | 118602/230406 [00:01<00:00, 121877.85it/s]
 57%|█████▋    | 131054/230406 [00:01<00:00, 122655.22it/s]
 62%|██████▏   | 143521/230406 [00:01<00:00, 123252.30it/s]
 68%|██████▊   | 156073/230406 [00:01<00:00, 123921.72it/s]
 73%|███████▎  | 168366/230406 [00:01<00:00, 122985.99it/s]
 78%|███████▊  | 180597/230406 [00:01<00:00, 120898.29it/s]
 84%|████████▍ | 193159/230406 [00:01<00:00, 122274.17it/s]
 89%|████████▉ | 205741/230406 [00:01<00:00, 123316.35it/s]
 95%|█████████▍| 218331/230406 [00:01<00:00, 124077.39it/s]
100%|██████████| 230406/230406 [00:01<00:00, 121012.95it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11071/150000 [00:00<00:01, 110707.47it/s]
 15%|█▍        | 22330/150000 [00:00<00:01, 111265.46it/s]
 23%|██▎       | 33943/150000 [00:00<00:01, 112680.65it/s]
 31%|███       | 45947/150000 [00:00<00:00, 114789.30it/s]
 39%|███▉      | 58155/150000 [00:00<00:00, 116882.56it/s]
 47%|████▋     | 70314/150000 [00:00<00:00, 118253.96it/s]
 55%|█████▌    | 82666/150000 [00:00<00:00, 119783.64it/s]
 63%|██████▎   | 95104/150000 [00:00<00:00, 121124.23it/s]
 72%|███████▏  | 107582/150000 [00:00<00:00, 122196.25it/s]
 80%|████████  | 120057/150000 [00:01<00:00, 122950.15it/s]
 88%|████████▊ | 132581/150000 [00:01<00:00, 123626.82it/s]
 97%|█████████▋| 145037/150000 [00:01<00:00, 123902.07it/s]
100%|██████████| 150000/150000 [00:01<00:00, 120955.95it/s]

0it [00:00, ?it/s]
149it [00:00, 1478.20it/s]
325it [00:00, 1551.52it/s]
518it [00:00, 1646.01it/s]
692it [00:00, 1667.66it/s]
928it [00:00, 1818.93it/s]
1133it [00:00, 1871.14it/s]
1370it [00:00, 1972.46it/s]
1641it [00:00, 2146.42it/s]
2002it [00:00, 2441.30it/s]
2348it [00:01, 2676.60it/s]
2830it [00:01, 3085.49it/s]
3168it [00:01, 2663.37it/s]

  0%|          | 0/7895 [00:00<?, ?it/s]
 21%|██▏       | 1696/7895 [00:00<00:00, 16959.13it/s]
 50%|████▉     | 3931/7895 [00:00<00:00, 18281.64it/s]
 77%|███████▋  | 6115/7895 [00:00<00:00, 18886.04it/s]
100%|██████████| 7895/7895 [00:00<00:00, 21663.79it/s]

  0%|          | 0/70257 [00:00<?, ?it/s]
  4%|▍         | 2904/70257 [00:00<00:02, 29038.92it/s]
  8%|▊         | 5629/70257 [00:00<00:02, 28475.94it/s]
 10%|█         | 7057/70257 [00:00<00:03, 15939.64it/s]
 14%|█▎        | 9566/70257 [00:00<00:03, 17897.33it/s]
 17%|█▋        | 12071/70257 [00:00<00:02, 19573.80it/s]
 21%|██        | 14582/70257 [00:00<00:02, 20959.88it/s]
 24%|██▍       | 16913/70257 [00:00<00:02, 21612.22it/s]
 28%|██▊       | 19341/70257 [00:00<00:02, 22348.36it/s]
 31%|███       | 21803/70257 [00:00<00:02, 22983.15it/s]
 35%|███▍      | 24290/70257 [00:01<00:01, 23516.40it/s]
 38%|███▊      | 26799/70257 [00:01<00:01, 23966.35it/s]
 42%|████▏     | 29202/70257 [00:01<00:01, 23706.28it/s]
 45%|████▌     | 31638/70257 [00:01<00:01, 23897.65it/s]
 48%|████▊     | 34056/70257 [00:01<00:01, 23980.64it/s]
 52%|█████▏    | 36509/70257 [00:01<00:01, 24140.70it/s]
 55%|█████▌    | 38926/70257 [00:01<00:01, 24043.08it/s]
 59%|█████▉    | 41332/70257 [00:01<00:01, 23389.96it/s]
 62%|██████▏   | 43676/70257 [00:01<00:01, 23201.86it/s]
 65%|██████▌   | 46001/70257 [00:02<00:01, 22933.18it/s]
 69%|██████▊   | 48298/70257 [00:02<00:00, 22822.86it/s]
 72%|███████▏  | 50598/70257 [00:02<00:00, 22873.93it/s]
 75%|███████▌  | 52961/70257 [00:02<00:00, 23094.87it/s]
 79%|███████▉  | 55476/70257 [00:02<00:00, 23675.02it/s]
 83%|████████▎ | 57981/70257 [00:02<00:00, 24070.22it/s]
 86%|████████▌ | 60546/70257 [00:02<00:00, 24522.08it/s]
 90%|████████▉ | 63134/70257 [00:02<00:00, 24913.65it/s]
 93%|█████████▎| 65688/70257 [00:02<00:00, 25098.27it/s]
 97%|█████████▋| 68230/70257 [00:02<00:00, 25192.55it/s]
100%|██████████| 70257/70257 [00:02<00:00, 23546.32it/s]

  0%|          | 0/70257 [00:00<?, ?it/s]
  8%|▊         | 5608/70257 [00:00<00:01, 52209.44it/s]
 14%|█▍        | 10123/70257 [00:00<00:01, 48638.69it/s]
 21%|██        | 14478/70257 [00:00<00:01, 45551.05it/s]
 27%|██▋       | 18740/70257 [00:00<00:01, 43052.85it/s]
 32%|███▏      | 22719/70257 [00:00<00:01, 41086.54it/s]
 38%|███▊      | 26608/70257 [00:00<00:01, 39761.93it/s]
 43%|████▎     | 30461/70257 [00:00<00:01, 38613.80it/s]
 49%|████▊     | 34157/70257 [00:00<00:00, 37744.98it/s]
 54%|█████▍    | 37813/70257 [00:00<00:00, 37169.80it/s]
 59%|█████▉    | 41554/70257 [00:01<00:00, 36663.36it/s]
 65%|██████▍   | 45344/70257 [00:01<00:00, 36676.93it/s]
 70%|███████   | 49409/70257 [00:01<00:00, 35256.93it/s]
 76%|███████▋  | 53624/70257 [00:01<00:00, 36303.53it/s]
 83%|████████▎ | 58114/70257 [00:01<00:00, 36984.68it/s]
 90%|████████▉ | 62972/70257 [00:01<00:00, 39835.99it/s]
 95%|█████████▌| 67036/70257 [00:01<00:00, 40073.19it/s]
100%|██████████| 70257/70257 [00:01<00:00, 39034.06it/s]

  0%|          | 0/3168 [00:00<?, ?it/s]
100%|██████████| 3168/3168 [00:00<00:00, 81815.89it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.15it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.08it/s]
  3%|▎         | 3/100 [00:00<00:13,  6.94it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.69it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.76it/s]
  6%|▌         | 6/100 [00:00<00:13,  6.78it/s]
  7%|▋         | 7/100 [00:01<00:13,  6.68it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.47it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.64it/s]
 10%|█         | 10/100 [00:01<00:13,  6.61it/s]
 11%|█         | 11/100 [00:01<00:13,  6.71it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.48it/s]
 13%|█▎        | 13/100 [00:01<00:13,  6.64it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.59it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.33it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.31it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.48it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.44it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.40it/s]
 20%|██        | 20/100 [00:03<00:12,  6.46it/s]
 21%|██        | 21/100 [00:03<00:12,  6.53it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.67it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.53it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.23it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.30it/s]
 26%|██▌       | 26/100 [00:04<00:12,  6.06it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.22it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.02it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.23it/s]
 30%|███       | 30/100 [00:04<00:10,  6.43it/s]
 31%|███       | 31/100 [00:04<00:10,  6.27it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.42it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.54it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.64it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.63it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.54it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.38it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.47it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.41it/s]
 40%|████      | 40/100 [00:06<00:09,  6.52it/s]
 41%|████      | 41/100 [00:06<00:09,  6.46it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.54it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.58it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.78it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.33it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.47it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.56it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.53it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.55it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.58it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.70it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.62it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.40it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.36it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.48it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.50it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.25it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.17it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.20it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.07it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.24it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  5.95it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.00it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.12it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.14it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.42it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.49it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.72it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.64it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.71it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.78it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.74it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.26it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.43it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.53it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.61it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.54it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.49it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.41it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.20it/s]
 81%|████████  | 81/100 [00:12<00:03,  6.27it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.53it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.71it/s]
 84%|████████▍ | 84/100 [00:13<00:02,  6.56it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.82it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.91it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  6.96it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.78it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.71it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.70it/s]
 91%|█████████ | 91/100 [00:14<00:01,  6.69it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.51it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.49it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.32it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.35it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.67it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.62it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.84it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.49it/s]
100%|██████████| 100/100 [00:15<00:00,  6.39it/s]
100%|██████████| 100/100 [00:15<00:00,  6.48it/s]

  0%|          | 0/70257 [00:00<?, ?it/s]
  6%|▌         | 3951/70257 [00:00<00:01, 39508.81it/s]
 13%|█▎        | 8889/70257 [00:00<00:01, 42028.74it/s]
 22%|██▏       | 15667/70257 [00:00<00:01, 47434.70it/s]
 33%|███▎      | 22912/70257 [00:00<00:00, 52914.18it/s]
 44%|████▎     | 30589/70257 [00:00<00:00, 58354.00it/s]
 55%|█████▌    | 38718/70257 [00:00<00:00, 63749.94it/s]
 67%|██████▋   | 47353/70257 [00:00<00:00, 69180.54it/s]
 80%|████████  | 56544/70257 [00:00<00:00, 74722.09it/s]
 95%|█████████▍| 66393/70257 [00:00<00:00, 80553.40it/s]
100%|██████████| 70257/70257 [00:00<00:00, 74970.22it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Script completed


  0%|          | 0/346693 [00:00<?, ?it/s]
  3%|▎         | 11060/346693 [00:00<00:03, 110595.89it/s]
  6%|▋         | 22402/346693 [00:00<00:02, 111427.02it/s]
 10%|▉         | 33688/346693 [00:00<00:02, 111850.57it/s]
 13%|█▎        | 45522/346693 [00:00<00:02, 113721.06it/s]
 17%|█▋        | 57705/346693 [00:00<00:02, 116037.02it/s]
 20%|██        | 69895/346693 [00:00<00:02, 117735.70it/s]
 24%|██▎       | 82162/346693 [00:00<00:02, 119170.85it/s]
 27%|██▋       | 94620/346693 [00:00<00:02, 120742.65it/s]
 31%|███       | 107139/346693 [00:00<00:01, 122040.97it/s]
 35%|███▍      | 119863/346693 [00:01<00:01, 123554.69it/s]
 38%|███▊      | 132490/346693 [00:01<00:01, 124356.54it/s]
 42%|████▏     | 145096/346693 [00:01<00:01, 124862.67it/s]
 46%|████▌     | 157769/346693 [00:01<00:01, 125416.48it/s]
 49%|████▉     | 170740/346693 [00:01<00:01, 126672.04it/s]
 53%|█████▎    | 183542/346693 [00:01<00:01, 127071.67it/s]
 57%|█████▋    | 196203/346693 [00:01<00:01, 126743.08it/s]
 60%|██████    | 209138/346693 [00:01<00:01, 127511.22it/s]
 64%|██████▍   | 221970/346693 [00:01<00:00, 127751.63it/s]
 68%|██████▊   | 234783/346693 [00:01<00:00, 127863.58it/s]
 71%|███████▏  | 247770/346693 [00:02<00:00, 128455.18it/s]
 75%|███████▌  | 260658/346693 [00:02<00:00, 128579.89it/s]
 79%|███████▉  | 273512/346693 [00:02<00:00, 127735.27it/s]
 83%|████████▎ | 286446/346693 [00:02<00:00, 128211.90it/s]
 86%|████████▋ | 299267/346693 [00:02<00:00, 123878.54it/s]
 90%|█████████ | 312239/346693 [00:02<00:00, 125573.28it/s]
 94%|█████████▎| 324824/346693 [00:02<00:00, 125575.76it/s]
 97%|█████████▋| 337816/346693 [00:02<00:00, 126845.33it/s]
100%|██████████| 346693/346693 [00:02<00:00, 124506.97it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6510/150000 [00:00<00:02, 65096.34it/s]
  9%|▉         | 13193/150000 [00:00<00:02, 65606.38it/s]
 13%|█▎        | 19939/150000 [00:00<00:01, 66149.48it/s]
 18%|█▊        | 26756/150000 [00:00<00:01, 66741.51it/s]
 22%|██▏       | 33627/150000 [00:00<00:01, 67319.09it/s]
 27%|██▋       | 40540/150000 [00:00<00:01, 67852.01it/s]
 32%|███▏      | 47559/150000 [00:00<00:01, 68536.43it/s]
 36%|███▋      | 54491/150000 [00:00<00:01, 68769.51it/s]
 41%|████      | 61503/150000 [00:00<00:01, 69166.89it/s]
 46%|████▌     | 68519/150000 [00:01<00:01, 69458.88it/s]
 50%|█████     | 75541/150000 [00:01<00:01, 69683.66it/s]
 55%|█████▌    | 82587/150000 [00:01<00:00, 69911.61it/s]
 60%|█████▉    | 89614/150000 [00:01<00:00, 70018.26it/s]
 64%|██████▍   | 96622/150000 [00:01<00:00, 70035.74it/s]
 69%|██████▉   | 103623/150000 [00:01<00:00, 70024.88it/s]
 74%|███████▍  | 110648/150000 [00:01<00:00, 70090.48it/s]
 78%|███████▊  | 117695/150000 [00:01<00:00, 70201.03it/s]
 83%|████████▎ | 124701/150000 [00:01<00:00, 70153.55it/s]
 88%|████████▊ | 131713/150000 [00:01<00:00, 70143.05it/s]
 92%|█████████▏| 138721/150000 [00:02<00:00, 70116.97it/s]
 97%|█████████▋| 145732/150000 [00:02<00:00, 70112.84it/s]
100%|██████████| 150000/150000 [00:02<00:00, 69397.63it/s]

  0%|          | 0/64 [00:00<?, ?it/s]
 3168 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-26 ...
Successfully removed 7895 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-27 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 5693 new domains found

Scrapping urls...
scrapping will be done in 64 phase  where 90 urls attempt in each phase

  2%|▏         | 1/64 [00:36<38:14, 36.42s/it]
  3%|▎         | 2/64 [01:03<34:47, 33.67s/it]
  5%|▍         | 3/64 [01:32<32:36, 32.07s/it]
  6%|▋         | 4/64 [01:43<25:48, 25.81s/it]
  8%|▊         | 5/64 [02:09<25:39, 26.10s/it]
  9%|▉         | 6/64 [02:41<26:47, 27.71s/it]
 11%|█         | 7/64 [03:11<27:04, 28.50s/it]
 12%|█▎        | 8/64 [03:48<28:53, 30.95s/it]
 14%|█▍        | 9/64 [04:24<29:50, 32.55s/it]
 16%|█▌        | 10/64 [05:04<31:17, 34.77s/it]
 17%|█▋        | 11/64 [05:43<31:43, 35.91s/it]
 19%|█▉        | 12/64 [06:19<31:18, 36.12s/it]
 20%|██        | 13/64 [06:47<28:37, 33.69s/it]
 22%|██▏       | 14/64 [07:24<28:41, 34.43s/it]
 23%|██▎       | 15/64 [08:03<29:21, 35.94s/it]
 25%|██▌       | 16/64 [08:34<27:39, 34.58s/it]
 27%|██▋       | 17/64 [09:15<28:32, 36.45s/it]
 28%|██▊       | 18/64 [09:52<27:55, 36.42s/it]
 30%|██▉       | 19/64 [10:33<28:28, 37.97s/it]
 31%|███▏      | 20/64 [11:09<27:18, 37.24s/it]
 33%|███▎      | 21/64 [11:43<26:07, 36.44s/it]
 34%|███▍      | 22/64 [12:19<25:24, 36.30s/it]
 36%|███▌      | 23/64 [13:00<25:46, 37.71s/it]
 38%|███▊      | 24/64 [13:33<24:14, 36.37s/it]
 39%|███▉      | 25/64 [14:18<25:10, 38.73s/it]
 41%|████      | 26/64 [14:54<24:04, 38.01s/it]
 42%|████▏     | 27/64 [15:35<23:57, 38.86s/it]
 44%|████▍     | 28/64 [16:18<24:03, 40.11s/it]
 45%|████▌     | 29/64 [16:56<22:59, 39.43s/it]
 47%|████▋     | 30/64 [17:31<21:36, 38.13s/it]
 48%|████▊     | 31/64 [18:15<21:55, 39.87s/it]
 50%|█████     | 32/64 [18:51<20:41, 38.78s/it]
 52%|█████▏    | 33/64 [19:09<16:51, 32.62s/it]
 53%|█████▎    | 34/64 [19:54<18:10, 36.34s/it]47.247.41.172 - - [29/May/2020 12:08:41] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
47.247.83.250 - - [29/May/2020 12:08:41] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:08:41] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 55%|█████▍    | 35/64 [20:18<15:42, 32.51s/it]
 56%|█████▋    | 36/64 [20:49<15:00, 32.18s/it]
 58%|█████▊    | 37/64 [21:31<15:49, 35.16s/it]
 59%|█████▉    | 38/64 [22:14<16:09, 37.29s/it]47.247.83.250 - - [29/May/2020 12:10:48] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:10:57] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:11:02] "GET /getOneDayClusterData/2020-05-29/16 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:11:05] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -

 61%|██████    | 39/64 [22:51<15:34, 37.37s/it]47.247.83.250 - - [29/May/2020 12:11:22] "GET /getClusterData/2020-05-01/2020-05-29/16 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:11:24] "GET /getClusterData/2020-05-01/2020-05-07/16 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:11:29] "GET /getClusterData/2020-05-01/2020-05-27/16 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:11:37] "GET /getClusterData/2020-04-02/2020-05-27/16 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:11:56] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:11:57] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:11:58] "GET /getClusterData/2020-04-02/2020-05-27/15 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:11:58] "GET /getclusterurl/15/page/0 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:11:59] "GET /getClusterData/2020-04-02/2020-05-27/16 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:12:00] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -

 62%|██████▎   | 40/64 [23:37<16:00, 40.01s/it]
 64%|██████▍   | 41/64 [24:05<13:57, 36.42s/it]
 66%|██████▌   | 42/64 [24:42<13:24, 36.58s/it]
 67%|██████▋   | 43/64 [25:19<12:47, 36.57s/it]49.36.48.61 - - [29/May/2020 12:14:01] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
49.36.48.61 - - [29/May/2020 12:14:07] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
49.36.48.61 - - [29/May/2020 12:14:07] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
49.36.48.61 - - [29/May/2020 12:14:07] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 69%|██████▉   | 44/64 [25:58<12:25, 37.29s/it]49.36.48.61 - - [29/May/2020 12:14:51] "GET /getclusterurl/page/0 HTTP/1.1" 200 -

 70%|███████   | 45/64 [26:37<12:01, 37.98s/it]
 72%|███████▏  | 46/64 [27:10<10:53, 36.33s/it]
 73%|███████▎  | 47/64 [27:44<10:04, 35.58s/it]
 75%|███████▌  | 48/64 [28:24<09:52, 37.03s/it]
 77%|███████▋  | 49/64 [29:02<09:18, 37.23s/it]
 78%|███████▊  | 50/64 [29:27<07:49, 33.51s/it]
 80%|███████▉  | 51/64 [29:46<06:20, 29.24s/it]
 81%|████████▏ | 52/64 [30:14<05:47, 28.93s/it]
 83%|████████▎ | 53/64 [30:58<06:05, 33.26s/it]
 84%|████████▍ | 54/64 [31:38<05:54, 35.42s/it]
 86%|████████▌ | 55/64 [32:08<05:04, 33.81s/it]
 88%|████████▊ | 56/64 [32:47<04:41, 35.20s/it]
 89%|████████▉ | 57/64 [33:18<03:59, 34.19s/it]
 91%|█████████ | 58/64 [33:55<03:29, 34.94s/it]47.247.83.250 - - [29/May/2020 12:22:45] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:22:47] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:22:47] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -

 92%|█████████▏| 59/64 [34:35<03:01, 36.40s/it]47.247.41.172 - - [29/May/2020 12:23:13] "GET /getOneDayClusterData/2020-05-29/16 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:23:13] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:23:19] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:23:19] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:23:21] "GET /getOneDayClusterData/2020-05-29/2 HTTP/1.1" 200 -
47.247.41.172 - - [29/May/2020 12:23:22] "GET /getclusterurl/2/page/0 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:23:25] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
47.247.83.250 - - [29/May/2020 12:23:25] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:23:36] "GET /getOneDayClusterData/2020-05-07/23 HTTP/1.1" 200 -

 94%|█████████▍| 60/64 [35:13<02:27, 36.83s/it]27.56.245.116 - - [29/May/2020 12:23:39] "GET /favicon.ico HTTP/1.1" 200 -

 95%|█████████▌| 61/64 [35:43<01:44, 34.76s/it]
 97%|█████████▋| 62/64 [36:02<01:00, 30.03s/it]27.56.245.116 - - [29/May/2020 12:24:41] "GET /getOneDayClusterData/2020-05-07/23 HTTP/1.1" 200 -

 98%|█████████▊| 63/64 [36:19<00:26, 26.32s/it]27.56.245.116 - - [29/May/2020 12:24:44] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:24:54] "GET /getOneDayClusterData/2020-05-30/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:24:57] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:25:03] "GET /getOneDayClusterData/2020-05-30/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:25:04] "GET /favicon.ico HTTP/1.1" 200 -

100%|██████████| 64/64 [36:42<00:00, 25.14s/it]
100%|██████████| 64/64 [36:42<00:00, 34.41s/it]

  0%|          | 0/224894 [00:00<?, ?it/s]
  5%|▍         | 11017/224894 [00:00<00:01, 110169.32it/s]
 10%|▉         | 22195/224894 [00:00<00:01, 110646.01it/s]
 15%|█▌        | 33935/224894 [00:00<00:01, 112586.51it/s]
 20%|██        | 45616/224894 [00:00<00:01, 113819.90it/s]
 26%|██▌       | 57669/224894 [00:00<00:01, 115751.43it/s]
 31%|███       | 69791/224894 [00:00<00:01, 117336.75it/s]
 36%|███▋      | 81798/224894 [00:00<00:01, 118142.02it/s]
 42%|████▏     | 93665/224894 [00:00<00:01, 118299.79it/s]
 47%|████▋     | 105998/224894 [00:00<00:00, 119765.02it/s]
 53%|█████▎    | 118406/224894 [00:01<00:00, 121026.52it/s]
 58%|█████▊    | 130899/224894 [00:01<00:00, 122171.23it/s]
 64%|██████▎   | 143360/224894 [00:01<00:00, 122891.86it/s]
 69%|██████▉   | 155795/224894 [00:01<00:00, 123323.28it/s]
 75%|███████▍  | 168027/224894 [00:01<00:00, 122450.44it/s]
 80%|████████  | 180203/224894 [00:01<00:00, 121121.91it/s]
 86%|████████▌ | 192808/224894 [00:01<00:00, 122559.35it/s]
 91%|█████████▏| 205352/224894 [00:01<00:00, 123409.16it/s]
 97%|█████████▋| 217872/224894 [00:01<00:00, 123940.87it/s]
100%|██████████| 224894/224894 [00:01<00:00, 120856.43it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11089/150000 [00:00<00:01, 110882.17it/s]
 15%|█▍        | 22387/150000 [00:00<00:01, 111502.78it/s]
 23%|██▎       | 34118/150000 [00:00<00:01, 113183.07it/s]
 31%|███       | 46208/150000 [00:00<00:00, 115392.01it/s]
 39%|███▉      | 58223/150000 [00:00<00:00, 116777.37it/s]
 47%|████▋     | 70477/150000 [00:00<00:00, 118445.68it/s]
 55%|█████▌    | 82842/150000 [00:00<00:00, 119959.76it/s]
 63%|██████▎   | 95184/150000 [00:00<00:00, 120974.60it/s]
 72%|███████▏  | 107617/150000 [00:00<00:00, 121959.18it/s]
 80%|████████  | 120057/150000 [00:01<00:00, 122679.52it/s]
 88%|████████▊ | 132594/150000 [00:01<00:00, 123471.80it/s]
 97%|█████████▋| 145132/150000 [00:01<00:00, 124037.29it/s]
100%|██████████| 150000/150000 [00:01<00:00, 121022.66it/s]

0it [00:00, ?it/s]
93it [00:00, 911.38it/s]
201it [00:00, 954.41it/s]
360it [00:00, 1083.23it/s]
495it [00:00, 1147.07it/s]
699it [00:00, 1317.74it/s]
896it [00:00, 1463.05it/s]
1146it [00:00, 1666.83it/s]
1325it [00:00, 1697.73it/s]
1512it [00:00, 1744.87it/s]
1769it [00:00, 1787.07it/s]

  0%|          | 0/7281 [00:00<?, ?it/s]
 15%|█▍        | 1070/7281 [00:00<00:00, 10686.89it/s]
 35%|███▌      | 2576/7281 [00:00<00:00, 11706.21it/s]
 57%|█████▋    | 4179/7281 [00:00<00:00, 12370.94it/s]
 88%|████████▊ | 6389/7281 [00:00<00:00, 14246.74it/s]
100%|██████████| 7281/7281 [00:00<00:00, 16499.93it/s]

  0%|          | 0/72088 [00:00<?, ?it/s]
  4%|▎         | 2582/72088 [00:00<00:03, 19093.52it/s]
  5%|▌         | 3730/72088 [00:00<00:05, 12982.66it/s]
  8%|▊         | 6118/72088 [00:00<00:04, 15041.07it/s]
 12%|█▏        | 8510/72088 [00:00<00:03, 16924.63it/s]
 15%|█▌        | 10848/72088 [00:00<00:03, 18452.17it/s]
 18%|█▊        | 13174/72088 [00:00<00:02, 19671.53it/s]
 22%|██▏       | 15521/72088 [00:00<00:02, 20674.65it/s]
 25%|██▍       | 17913/72088 [00:00<00:02, 21550.36it/s]
 28%|██▊       | 20366/72088 [00:00<00:02, 22363.26it/s]
 32%|███▏      | 22809/72088 [00:01<00:02, 22945.28it/s]
 35%|███▌      | 25308/72088 [00:01<00:01, 23521.99it/s]
 39%|███▊      | 27793/72088 [00:01<00:01, 23902.91it/s]
 42%|████▏     | 30276/72088 [00:01<00:01, 24171.09it/s]
 45%|████▌     | 32754/72088 [00:01<00:01, 24349.73it/s]
 49%|████▉     | 35246/72088 [00:01<00:01, 24516.32it/s]
 52%|█████▏    | 37702/72088 [00:01<00:01, 24359.76it/s]
 56%|█████▌    | 40142/72088 [00:01<00:01, 23911.42it/s]
 59%|█████▉    | 42538/72088 [00:01<00:01, 23701.52it/s]
 62%|██████▏   | 44912/72088 [00:02<00:01, 23479.14it/s]
 66%|██████▌   | 47263/72088 [00:02<00:01, 23474.21it/s]
 69%|██████▉   | 49613/72088 [00:02<00:00, 23306.97it/s]
 72%|███████▏  | 51979/72088 [00:02<00:00, 23411.73it/s]
 76%|███████▌  | 54467/72088 [00:02<00:00, 23833.20it/s]
 79%|███████▉  | 56962/72088 [00:02<00:00, 24156.97it/s]
 82%|████████▏ | 59433/72088 [00:02<00:00, 24318.13it/s]
 86%|████████▌ | 61959/72088 [00:02<00:00, 24591.80it/s]
 89%|████████▉ | 64467/72088 [00:02<00:00, 24732.91it/s]
 93%|█████████▎| 66968/72088 [00:02<00:00, 24815.20it/s]
 96%|█████████▋| 69488/72088 [00:03<00:00, 24926.71it/s]
100%|█████████▉| 72061/72088 [00:03<00:00, 25162.16it/s]
100%|██████████| 72088/72088 [00:03<00:00, 23182.91it/s]

  0%|          | 0/72088 [00:00<?, ?it/s]
  8%|▊         | 5851/72088 [00:00<00:01, 51627.61it/s]
 14%|█▍        | 10257/72088 [00:00<00:01, 47985.33it/s]
 20%|██        | 14668/72088 [00:00<00:01, 45062.88it/s]
 27%|██▋       | 19186/72088 [00:00<00:01, 43026.49it/s]
 33%|███▎      | 23532/72088 [00:00<00:01, 41132.08it/s]
 38%|███▊      | 27553/72088 [00:00<00:01, 39319.71it/s]
 44%|████▎     | 31521/72088 [00:00<00:01, 37919.65it/s]
 49%|████▉     | 35315/72088 [00:00<00:00, 36875.75it/s]
 54%|█████▍    | 39263/72088 [00:01<00:00, 36229.35it/s]
 60%|█████▉    | 43180/72088 [00:01<00:00, 36229.70it/s]
 65%|██████▌   | 47024/72088 [00:01<00:00, 36182.09it/s]
 71%|███████   | 51088/72088 [00:01<00:00, 36082.49it/s]
 77%|███████▋  | 55271/72088 [00:01<00:00, 36476.05it/s]
 83%|████████▎ | 59737/72088 [00:01<00:00, 37114.27it/s]
 89%|████████▉ | 64371/72088 [00:01<00:00, 39471.01it/s]
 95%|█████████▍| 68342/72088 [00:01<00:00, 39308.25it/s]
100%|██████████| 72088/72088 [00:01<00:00, 38683.14it/s]

  0%|          | 0/1769 [00:00<?, ?it/s]
100%|██████████| 1769/1769 [00:00<00:00, 73503.37it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:14,  6.93it/s]
  2%|▏         | 2/100 [00:00<00:14,  6.81it/s]
  3%|▎         | 3/100 [00:00<00:14,  6.71it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.56it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.63it/s]
  6%|▌         | 6/100 [00:00<00:14,  6.68it/s]
  7%|▋         | 7/100 [00:01<00:14,  6.63it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.48it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.63it/s]
 10%|█         | 10/100 [00:01<00:13,  6.63it/s]
 11%|█         | 11/100 [00:01<00:13,  6.79it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.56it/s]
 13%|█▎        | 13/100 [00:01<00:13,  6.68it/s]
 14%|█▍        | 14/100 [00:02<00:12,  6.67it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.43it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.38it/s]
 17%|█▋        | 17/100 [00:02<00:12,  6.60it/s]
 18%|█▊        | 18/100 [00:02<00:12,  6.60it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.62it/s]
 20%|██        | 20/100 [00:03<00:11,  6.70it/s]
 21%|██        | 21/100 [00:03<00:11,  6.78it/s]
 22%|██▏       | 22/100 [00:03<00:11,  6.90it/s]
 23%|██▎       | 23/100 [00:03<00:11,  6.66it/s]
 24%|██▍       | 24/100 [00:03<00:11,  6.37it/s]
 25%|██▌       | 25/100 [00:03<00:11,  6.45it/s]
 26%|██▌       | 26/100 [00:03<00:12,  6.16it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.32it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.05it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.25it/s]
 30%|███       | 30/100 [00:04<00:10,  6.45it/s]
 31%|███       | 31/100 [00:04<00:10,  6.30it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.37it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.50it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.61it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.65it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.55it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.41it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.54it/s]
 39%|███▉      | 39/100 [00:05<00:09,  6.41it/s]
 40%|████      | 40/100 [00:06<00:09,  6.55it/s]
 41%|████      | 41/100 [00:06<00:09,  6.50it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.60it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.69it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.87it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.40it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.54it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.62it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.59it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.53it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.56it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.70it/s]
 52%|█████▏    | 52/100 [00:07<00:07,  6.64it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.45it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.40it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.52it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.64it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.37it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.27it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.33it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.18it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.28it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.12it/s]
 63%|██████▎   | 63/100 [00:09<00:06,  6.16it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.38it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.44it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.68it/s]
 67%|██████▋   | 67/100 [00:10<00:04,  6.68it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.89it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.77it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.81it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.89it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.88it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.35it/s]
 74%|███████▍  | 74/100 [00:11<00:03,  6.52it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.74it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.85it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.76it/s]
 78%|███████▊  | 78/100 [00:11<00:03,  6.67it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.58it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.30it/s]
 81%|████████  | 81/100 [00:12<00:02,  6.42it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.74it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.88it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.70it/s]
 85%|████████▌ | 85/100 [00:12<00:02,  6.95it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.95it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  7.09it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.95it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.91it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.89it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.82it/s]
 92%|█████████▏| 92/100 [00:13<00:01,  6.71it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.66it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.49it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.56it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.83it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.74it/s]
 98%|█████████▊| 98/100 [00:14<00:00,  6.97it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.66it/s]
100%|██████████| 100/100 [00:15<00:00,  6.46it/s]
100%|██████████| 100/100 [00:15<00:00,  6.58it/s]

  0%|          | 0/72088 [00:00<?, ?it/s]
  6%|▌         | 4411/72088 [00:00<00:01, 44102.79it/s]
 14%|█▎        | 9864/72088 [00:00<00:01, 46785.33it/s]
 23%|██▎       | 16906/72088 [00:00<00:01, 52022.72it/s]
 33%|███▎      | 23953/72088 [00:00<00:00, 56455.98it/s]
 44%|████▍     | 31549/72088 [00:00<00:00, 61166.90it/s]
 55%|█████▍    | 39484/72088 [00:00<00:00, 65681.75it/s]
 66%|██████▌   | 47675/72088 [00:00<00:00, 69830.81it/s]
 78%|███████▊  | 56489/72088 [00:00<00:00, 74470.72it/s]
 91%|█████████ | 65361/72088 [00:00<00:00, 78238.52it/s]
100%|██████████| 72088/72088 [00:00<00:00, 74682.01it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
27.56.245.116 - - [29/May/2020 12:25:59] "GET /getOneDayClusterData/2020-05-30/23 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 12:26:01] "GET /favicon.ico HTTP/1.1" 200 -
Script completed


  0%|          | 0/316921 [00:00<?, ?it/s]
  3%|▎         | 11051/316921 [00:00<00:02, 110504.31it/s]
  7%|▋         | 22252/316921 [00:00<00:02, 110950.05it/s]
 11%|█         | 33728/316921 [00:00<00:02, 112064.67it/s]
 14%|█▍        | 45756/316921 [00:00<00:02, 114404.49it/s]
 18%|█▊        | 58089/316921 [00:00<00:02, 116940.26it/s]
 22%|██▏       | 70308/316921 [00:00<00:02, 118465.51it/s]
 26%|██▌       | 82758/316921 [00:00<00:01, 120211.15it/s]
 30%|███       | 95336/316921 [00:00<00:01, 121826.75it/s]
 34%|███▍      | 107713/316921 [00:00<00:01, 122400.03it/s]
 38%|███▊      | 120286/316921 [00:01<00:01, 123380.21it/s]
 42%|████▏     | 132929/316921 [00:01<00:01, 124278.22it/s]
 46%|████▌     | 145512/316921 [00:01<00:01, 124737.55it/s]
 50%|████▉     | 158273/316921 [00:01<00:01, 125583.35it/s]
 54%|█████▍    | 170820/316921 [00:01<00:01, 125545.86it/s]
 58%|█████▊    | 183715/316921 [00:01<00:01, 126545.62it/s]
 62%|██████▏   | 196323/316921 [00:01<00:00, 126300.64it/s]
 66%|██████▌   | 208921/316921 [00:01<00:00, 125523.10it/s]
 70%|██████▉   | 221722/316921 [00:01<00:00, 126257.04it/s]
 74%|███████▍  | 234475/316921 [00:01<00:00, 126635.30it/s]
 78%|███████▊  | 247300/316921 [00:02<00:00, 127112.04it/s]
 82%|████████▏ | 260236/316921 [00:02<00:00, 127775.15it/s]
 86%|████████▌ | 273010/316921 [00:02<00:00, 127207.51it/s]
 90%|█████████ | 286004/316921 [00:02<00:00, 128012.90it/s]
 94%|█████████▍| 298806/316921 [00:02<00:00, 127170.27it/s]
 98%|█████████▊| 311809/316921 [00:02<00:00, 128013.67it/s]
100%|██████████| 316921/316921 [00:02<00:00, 124213.94it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  4%|▍         | 6477/150000 [00:00<00:02, 64760.95it/s]
  9%|▉         | 13150/150000 [00:00<00:02, 65339.15it/s]
 13%|█▎        | 19912/150000 [00:00<00:01, 66004.84it/s]
 18%|█▊        | 26725/150000 [00:00<00:01, 66627.75it/s]
 22%|██▏       | 33595/150000 [00:00<00:01, 67234.08it/s]
 27%|██▋       | 40585/150000 [00:00<00:01, 68010.62it/s]
 32%|███▏      | 47573/150000 [00:00<00:01, 68558.75it/s]
 36%|███▋      | 54532/150000 [00:00<00:01, 68862.39it/s]
 41%|████      | 61615/150000 [00:00<00:01, 69439.07it/s]
 46%|████▌     | 68636/150000 [00:01<00:01, 69667.50it/s]
 50%|█████     | 75684/150000 [00:01<00:01, 69907.16it/s]
 55%|█████▌    | 82726/150000 [00:01<00:00, 70059.60it/s]
 60%|█████▉    | 89684/150000 [00:01<00:00, 69911.93it/s]
 64%|██████▍   | 96676/150000 [00:01<00:00, 69913.37it/s]
 69%|██████▉   | 103655/150000 [00:01<00:00, 69873.78it/s]
 74%|███████▍  | 110698/150000 [00:01<00:00, 70038.80it/s]
 78%|███████▊  | 117681/150000 [00:01<00:00, 69935.84it/s]
 83%|████████▎ | 124660/150000 [00:01<00:00, 69754.19it/s]
 88%|████████▊ | 131626/150000 [00:01<00:00, 69721.24it/s]
 92%|█████████▏| 138627/150000 [00:02<00:00, 69807.19it/s]
 97%|█████████▋| 145625/150000 [00:02<00:00, 69857.02it/s]
100%|██████████| 150000/150000 [00:02<00:00, 69333.02it/s]

  0%|          | 0/102 [00:00<?, ?it/s]
 1769 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-27 ...
Successfully removed 7281 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

Updating Database Using Cisco Ranklist Of 2020-05-28 ...

Fetching cisco ranklist...
Filtering cisco ranklist...
Filtering Already visited domains...

 9175 new domains found

Scrapping urls...
scrapping will be done in 102 phase  where 90 urls attempt in each phase

  1%|          | 1/102 [00:15<26:49, 15.94s/it]
  2%|▏         | 2/102 [00:52<36:43, 22.04s/it]
  3%|▎         | 3/102 [01:03<30:50, 18.70s/it]
  4%|▍         | 4/102 [01:14<26:48, 16.41s/it]
  5%|▍         | 5/102 [01:54<38:12, 23.64s/it]
  6%|▌         | 6/102 [02:33<45:00, 28.13s/it]
  7%|▋         | 7/102 [03:13<50:15, 31.74s/it]
  8%|▊         | 8/102 [03:47<50:41, 32.36s/it]
  9%|▉         | 9/102 [04:28<54:26, 35.12s/it]
 10%|▉         | 10/102 [05:03<53:51, 35.13s/it]
 11%|█         | 11/102 [05:33<50:51, 33.53s/it]
 12%|█▏        | 12/102 [06:18<55:13, 36.82s/it]
 13%|█▎        | 13/102 [06:58<56:08, 37.85s/it]
 14%|█▎        | 14/102 [07:38<56:23, 38.44s/it]
 15%|█▍        | 15/102 [08:19<57:05, 39.37s/it]
 16%|█▌        | 16/102 [08:41<48:48, 34.06s/it]
 17%|█▋        | 17/102 [09:25<52:21, 36.96s/it]
 18%|█▊        | 18/102 [10:05<53:02, 37.89s/it]
 19%|█▊        | 19/102 [10:45<53:31, 38.69s/it]
 20%|█▉        | 20/102 [11:23<52:14, 38.23s/it]
 21%|██        | 21/102 [12:03<52:25, 38.83s/it]
 22%|██▏       | 22/102 [12:43<52:25, 39.32s/it]
 23%|██▎       | 23/102 [13:21<51:13, 38.90s/it]
 24%|██▎       | 24/102 [13:55<48:36, 37.40s/it]
 25%|██▍       | 25/102 [14:31<47:29, 37.00s/it]
 25%|██▌       | 26/102 [15:03<45:04, 35.58s/it]
 26%|██▋       | 27/102 [15:42<45:29, 36.39s/it]
 27%|██▋       | 28/102 [16:19<45:06, 36.58s/it]
 28%|██▊       | 29/102 [16:55<44:24, 36.50s/it]
 29%|██▉       | 30/102 [17:38<46:18, 38.59s/it]
 30%|███       | 31/102 [18:21<47:04, 39.78s/it]
 31%|███▏      | 32/102 [18:49<42:10, 36.15s/it]122.168.90.167 - - [29/May/2020 12:45:53] "GET /getclusterurl/page/0 HTTP/1.1" 200 -

 32%|███▏      | 33/102 [19:27<42:18, 36.79s/it]122.168.90.167 - - [29/May/2020 12:45:54] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 12:45:54] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
122.168.90.167 - - [29/May/2020 12:45:55] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 33%|███▎      | 34/102 [19:59<40:12, 35.48s/it]
 34%|███▍      | 35/102 [20:36<39:52, 35.71s/it]
 35%|███▌      | 36/102 [21:11<39:13, 35.66s/it]
 36%|███▋      | 37/102 [21:58<42:12, 38.97s/it]
 37%|███▋      | 38/102 [22:38<41:57, 39.34s/it]
 38%|███▊      | 39/102 [23:20<42:14, 40.23s/it]
 39%|███▉      | 40/102 [23:58<40:54, 39.58s/it]
 40%|████      | 41/102 [24:32<38:28, 37.84s/it]
 41%|████      | 42/102 [24:55<33:19, 33.33s/it]
 42%|████▏     | 43/102 [25:35<34:38, 35.22s/it]
 43%|████▎     | 44/102 [26:08<33:33, 34.71s/it]
 44%|████▍     | 45/102 [26:37<31:16, 32.92s/it]
 45%|████▌     | 46/102 [27:04<29:04, 31.15s/it]
 46%|████▌     | 47/102 [27:47<31:56, 34.84s/it]
 47%|████▋     | 48/102 [28:26<32:26, 36.04s/it]
 48%|████▊     | 49/102 [28:59<30:56, 35.03s/it]
 49%|████▉     | 50/102 [29:41<32:17, 37.27s/it]
 50%|█████     | 51/102 [30:24<33:05, 38.94s/it]
 51%|█████     | 52/102 [31:01<31:47, 38.14s/it]
 52%|█████▏    | 53/102 [31:37<30:43, 37.63s/it]
 53%|█████▎    | 54/102 [32:13<29:46, 37.21s/it]
 54%|█████▍    | 55/102 [32:34<25:14, 32.23s/it]
 55%|█████▍    | 56/102 [33:10<25:39, 33.47s/it]
 56%|█████▌    | 57/102 [33:52<27:04, 36.11s/it]
 57%|█████▋    | 58/102 [34:27<26:03, 35.54s/it]
 58%|█████▊    | 59/102 [35:10<27:02, 37.73s/it]
 59%|█████▉    | 60/102 [35:48<26:30, 37.86s/it]
 60%|█████▉    | 61/102 [36:22<25:08, 36.80s/it]
 61%|██████    | 62/102 [36:57<24:09, 36.25s/it]
 62%|██████▏   | 63/102 [37:38<24:31, 37.72s/it]
 63%|██████▎   | 64/102 [38:12<23:11, 36.62s/it]
 64%|██████▎   | 65/102 [38:53<23:22, 37.92s/it]
 65%|██████▍   | 66/102 [39:36<23:39, 39.44s/it]
 66%|██████▌   | 67/102 [40:13<22:37, 38.79s/it]
 67%|██████▋   | 68/102 [40:50<21:31, 38.00s/it]
 68%|██████▊   | 69/102 [41:26<20:38, 37.52s/it]
 69%|██████▊   | 70/102 [41:57<18:58, 35.59s/it]
 70%|██████▉   | 71/102 [42:35<18:48, 36.41s/it]
 71%|███████   | 72/102 [43:18<19:05, 38.18s/it]
 72%|███████▏  | 73/102 [43:44<16:44, 34.64s/it]
 73%|███████▎  | 74/102 [44:17<15:58, 34.22s/it]
 74%|███████▎  | 75/102 [44:51<15:17, 33.97s/it]
 75%|███████▍  | 76/102 [45:29<15:16, 35.25s/it]
 75%|███████▌  | 77/102 [46:03<14:35, 35.03s/it]
 76%|███████▋  | 78/102 [46:35<13:37, 34.06s/it]
 77%|███████▋  | 79/102 [47:16<13:51, 36.16s/it]
 78%|███████▊  | 80/102 [47:39<11:47, 32.16s/it]
 79%|███████▉  | 81/102 [48:17<11:51, 33.88s/it]
 80%|████████  | 82/102 [48:58<12:01, 36.09s/it]
 81%|████████▏ | 83/102 [49:38<11:45, 37.15s/it]
 82%|████████▏ | 84/102 [49:56<09:28, 31.59s/it]
 83%|████████▎ | 85/102 [50:26<08:46, 30.95s/it]
 84%|████████▍ | 86/102 [51:03<08:46, 32.88s/it]
 85%|████████▌ | 87/102 [51:47<09:00, 36.01s/it]
 86%|████████▋ | 88/102 [52:04<07:07, 30.55s/it]
 87%|████████▋ | 89/102 [52:31<06:20, 29.27s/it]
 88%|████████▊ | 90/102 [53:12<06:36, 33.01s/it]
 89%|████████▉ | 91/102 [53:48<06:10, 33.65s/it]
 90%|█████████ | 92/102 [54:04<04:45, 28.55s/it]
 91%|█████████ | 93/102 [54:30<04:09, 27.73s/it]
 92%|█████████▏| 94/102 [55:06<04:02, 30.28s/it]
 93%|█████████▎| 95/102 [55:31<03:20, 28.65s/it]122.168.90.167 - - [29/May/2020 13:22:22] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:22:22] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 94%|█████████▍| 96/102 [55:57<02:47, 27.89s/it]
 95%|█████████▌| 97/102 [56:31<02:27, 29.55s/it]122.168.90.167 - - [29/May/2020 13:23:06] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:23:11] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:23:12] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -

 96%|█████████▌| 98/102 [57:04<02:02, 30.69s/it]
 97%|█████████▋| 99/102 [57:34<01:31, 30.35s/it]
 98%|█████████▊| 100/102 [58:10<01:04, 32.19s/it]
 99%|█████████▉| 101/102 [58:49<00:34, 34.23s/it]
100%|██████████| 102/102 [59:14<00:00, 31.40s/it]
100%|██████████| 102/102 [59:14<00:00, 34.85s/it]

  0%|          | 0/223039 [00:00<?, ?it/s]
  5%|▍         | 10864/223039 [00:00<00:01, 108639.59it/s]
 10%|▉         | 22117/223039 [00:00<00:01, 109777.74it/s]
 15%|█▌        | 33705/223039 [00:00<00:01, 111539.11it/s]
 20%|██        | 45259/223039 [00:00<00:01, 112709.38it/s]
 26%|██▌       | 57226/223039 [00:00<00:01, 114709.50it/s]
 31%|███       | 69385/223039 [00:00<00:01, 116688.82it/s]
 36%|███▋      | 81307/223039 [00:00<00:01, 117436.27it/s]
 42%|████▏     | 93403/223039 [00:00<00:01, 118471.01it/s]
 47%|████▋     | 105785/223039 [00:00<00:00, 120025.75it/s]
 53%|█████▎    | 118178/223039 [00:01<00:00, 121170.56it/s]
 59%|█████▊    | 130552/223039 [00:01<00:00, 121926.97it/s]
 64%|██████▍   | 142848/223039 [00:01<00:00, 122234.48it/s]
 70%|██████▉   | 155213/223039 [00:01<00:00, 122652.62it/s]
 75%|███████▌  | 167379/223039 [00:01<00:00, 121578.32it/s]
 80%|████████  | 179469/223039 [00:01<00:00, 120106.03it/s]
 86%|████████▌ | 191897/223039 [00:01<00:00, 121325.89it/s]
 92%|█████████▏| 204262/223039 [00:01<00:00, 122012.38it/s]
 97%|█████████▋| 216713/223039 [00:01<00:00, 122750.42it/s]
100%|██████████| 223039/223039 [00:01<00:00, 120111.86it/s]

  0%|          | 0/150000 [00:00<?, ?it/s]
  7%|▋         | 11016/150000 [00:00<00:01, 110156.69it/s]
 15%|█▍        | 22148/150000 [00:00<00:01, 110500.41it/s]
 23%|██▎       | 33764/150000 [00:00<00:01, 112136.76it/s]
 31%|███       | 45948/150000 [00:00<00:00, 114880.27it/s]
 39%|███▊      | 58028/150000 [00:00<00:00, 116591.66it/s]
 47%|████▋     | 70413/150000 [00:00<00:00, 118677.05it/s]
 55%|█████▌    | 82866/150000 [00:00<00:00, 120371.71it/s]
 64%|██████▎   | 95287/150000 [00:00<00:00, 121494.77it/s]
 72%|███████▏  | 107860/150000 [00:00<00:00, 122731.94it/s]
 80%|████████  | 120418/150000 [00:01<00:00, 123570.47it/s]
 89%|████████▊ | 132842/150000 [00:01<00:00, 123770.08it/s]
 97%|█████████▋| 145369/150000 [00:01<00:00, 124215.60it/s]
100%|██████████| 150000/150000 [00:01<00:00, 121268.90it/s]

0it [00:00, ?it/s]
50it [00:00, 496.42it/s]
132it [00:00, 562.34it/s]
239it [00:00, 654.79it/s]
343it [00:00, 735.69it/s]
470it [00:00, 838.46it/s]
570it [00:00, 879.23it/s]
662it [00:00, 889.36it/s]
807it [00:00, 1004.34it/s]
956it [00:00, 1110.61it/s]
1126it [00:01, 1229.28it/s]
1286it [00:01, 1319.88it/s]
1484it [00:01, 1466.24it/s]
1745it [00:01, 1687.69it/s]
1998it [00:01, 1874.90it/s]
2247it [00:01, 2009.41it/s]
2350it [00:01, 1496.30it/s]

  0%|          | 0/4205 [00:00<?, ?it/s]
  4%|▍         | 175/4205 [00:00<00:02, 1736.12it/s]
  9%|▉         | 374/4205 [00:00<00:02, 1804.96it/s]
 13%|█▎        | 566/4205 [00:00<00:01, 1836.65it/s]
 17%|█▋        | 713/4205 [00:00<00:02, 1698.01it/s]
 20%|██        | 847/4205 [00:00<00:02, 1569.84it/s]
 26%|██▌       | 1092/4205 [00:00<00:01, 1758.30it/s]
 32%|███▏      | 1343/4205 [00:00<00:01, 1931.05it/s]
 37%|███▋      | 1553/4205 [00:00<00:01, 1973.79it/s]
 42%|████▏     | 1750/4205 [00:00<00:01, 1951.47it/s]
 46%|████▋     | 1945/4205 [00:01<00:01, 1869.69it/s]
 51%|█████▏    | 2163/4205 [00:01<00:01, 1951.47it/s]
 56%|█████▌    | 2362/4205 [00:01<00:00, 1962.58it/s]
 63%|██████▎   | 2633/4205 [00:01<00:00, 2139.52it/s]
 68%|██████▊   | 2853/4205 [00:01<00:00, 2027.85it/s]
 75%|███████▍  | 3144/4205 [00:01<00:00, 2229.67it/s]
 80%|████████  | 3377/4205 [00:01<00:00, 2057.50it/s]
 86%|████████▌ | 3624/4205 [00:01<00:00, 2162.87it/s]
 92%|█████████▏| 3889/4205 [00:01<00:00, 2288.92it/s]
 98%|█████████▊| 4126/4205 [00:01<00:00, 2303.28it/s]
100%|██████████| 4205/4205 [00:02<00:00, 2083.62it/s]

  0%|          | 0/71257 [00:00<?, ?it/s]
  4%|▎         | 2566/71257 [00:00<00:02, 25657.03it/s]
  7%|▋         | 4839/71257 [00:00<00:02, 24699.45it/s]
  9%|▉         | 6633/71257 [00:00<00:02, 22189.00it/s]
 11%|█▏        | 8026/71257 [00:00<00:04, 14921.38it/s]
 15%|█▍        | 10369/71257 [00:00<00:03, 16745.21it/s]
 18%|█▊        | 12713/71257 [00:00<00:03, 18313.07it/s]
 21%|██        | 15039/71257 [00:00<00:02, 19561.05it/s]
 24%|██▍       | 17322/71257 [00:00<00:02, 20438.06it/s]
 28%|██▊       | 19725/71257 [00:00<00:02, 21396.79it/s]
 31%|███       | 22015/71257 [00:01<00:02, 21825.34it/s]
 34%|███▍      | 24443/71257 [00:01<00:02, 22506.28it/s]
 38%|███▊      | 27034/71257 [00:01<00:01, 23427.90it/s]
 42%|████▏     | 29602/71257 [00:01<00:01, 24058.39it/s]
 45%|████▌     | 32160/71257 [00:01<00:01, 24493.41it/s]
 49%|████▊     | 34625/71257 [00:01<00:01, 23550.37it/s]
 52%|█████▏    | 37156/71257 [00:01<00:01, 24049.74it/s]
 56%|█████▌    | 39684/71257 [00:01<00:01, 24404.85it/s]
 59%|█████▉    | 42227/71257 [00:01<00:01, 24703.58it/s]
 63%|██████▎   | 44707/71257 [00:01<00:01, 23856.31it/s]
 66%|██████▌   | 47106/71257 [00:02<00:01, 23175.16it/s]
 69%|██████▉   | 49437/71257 [00:02<00:00, 22749.68it/s]
 73%|███████▎  | 51723/71257 [00:02<00:00, 22475.26it/s]
 76%|███████▌  | 54201/71257 [00:02<00:00, 23118.47it/s]
 80%|███████▉  | 56697/71257 [00:02<00:00, 23641.18it/s]
 83%|████████▎ | 59134/71257 [00:02<00:00, 23854.38it/s]
 86%|████████▋ | 61595/71257 [00:02<00:00, 24073.71it/s]
 90%|█████████ | 64213/71257 [00:02<00:00, 24666.36it/s]
 94%|█████████▎| 66688/71257 [00:02<00:00, 24560.83it/s]
 97%|█████████▋| 69192/71257 [00:03<00:00, 24701.97it/s]
100%|██████████| 71257/71257 [00:03<00:00, 23017.26it/s]

  0%|          | 0/71257 [00:00<?, ?it/s]
  8%|▊         | 6033/71257 [00:00<00:01, 53595.68it/s]
 15%|█▌        | 10732/71257 [00:00<00:01, 50012.03it/s]
 21%|██▏       | 15176/71257 [00:00<00:01, 46639.04it/s]
 28%|██▊       | 19605/71257 [00:00<00:01, 45907.88it/s]
 33%|███▎      | 23407/71257 [00:00<00:01, 43216.60it/s]
 38%|███▊      | 27157/71257 [00:00<00:01, 41325.91it/s]
 43%|████▎     | 30751/71257 [00:00<00:01, 39547.23it/s]
 48%|████▊     | 34349/71257 [00:00<00:00, 38403.07it/s]
 53%|█████▎    | 37954/71257 [00:00<00:00, 37663.73it/s]
 58%|█████▊    | 41581/71257 [00:01<00:00, 37232.48it/s]
 63%|██████▎   | 45192/71257 [00:01<00:00, 36887.54it/s]
 68%|██████▊   | 48796/71257 [00:01<00:00, 36193.64it/s]
 74%|███████▎  | 52451/71257 [00:01<00:00, 36299.08it/s]
 79%|███████▉  | 56241/71257 [00:01<00:00, 36764.72it/s]
 85%|████████▌ | 60592/71257 [00:01<00:00, 37324.36it/s]
 92%|█████████▏| 65243/71257 [00:01<00:00, 39673.23it/s]
 97%|█████████▋| 69405/71257 [00:01<00:00, 40235.62it/s]
100%|██████████| 71257/71257 [00:01<00:00, 39972.49it/s]

  0%|          | 0/2350 [00:00<?, ?it/s]
100%|██████████| 2350/2350 [00:00<00:00, 78768.48it/s]

  0%|          | 0/100 [00:00<?, ?it/s]
  1%|          | 1/100 [00:00<00:13,  7.12it/s]
  2%|▏         | 2/100 [00:00<00:13,  7.09it/s]
  3%|▎         | 3/100 [00:00<00:13,  6.94it/s]
  4%|▍         | 4/100 [00:00<00:14,  6.72it/s]
  5%|▌         | 5/100 [00:00<00:14,  6.73it/s]
  6%|▌         | 6/100 [00:00<00:13,  6.72it/s]
  7%|▋         | 7/100 [00:01<00:13,  6.65it/s]
  8%|▊         | 8/100 [00:01<00:14,  6.51it/s]
  9%|▉         | 9/100 [00:01<00:13,  6.66it/s]
 10%|█         | 10/100 [00:01<00:13,  6.63it/s]
 11%|█         | 11/100 [00:01<00:13,  6.80it/s]
 12%|█▏        | 12/100 [00:01<00:13,  6.55it/s]
 13%|█▎        | 13/100 [00:01<00:13,  6.61it/s]
 14%|█▍        | 14/100 [00:02<00:13,  6.54it/s]
 15%|█▌        | 15/100 [00:02<00:13,  6.27it/s]
 16%|█▌        | 16/100 [00:02<00:13,  6.19it/s]
 17%|█▋        | 17/100 [00:02<00:13,  6.36it/s]
 18%|█▊        | 18/100 [00:02<00:13,  6.30it/s]
 19%|█▉        | 19/100 [00:02<00:12,  6.34it/s]
 20%|██        | 20/100 [00:03<00:12,  6.38it/s]
 21%|██        | 21/100 [00:03<00:12,  6.30it/s]
 22%|██▏       | 22/100 [00:03<00:12,  6.48it/s]
 23%|██▎       | 23/100 [00:03<00:12,  6.41it/s]
 24%|██▍       | 24/100 [00:03<00:12,  6.20it/s]
 25%|██▌       | 25/100 [00:03<00:12,  6.13it/s]
 26%|██▌       | 26/100 [00:04<00:12,  5.92it/s]
 27%|██▋       | 27/100 [00:04<00:11,  6.15it/s]
 28%|██▊       | 28/100 [00:04<00:11,  6.01it/s]
 29%|██▉       | 29/100 [00:04<00:11,  6.24it/s]
 30%|███       | 30/100 [00:04<00:10,  6.50it/s]
 31%|███       | 31/100 [00:04<00:10,  6.29it/s]
 32%|███▏      | 32/100 [00:04<00:10,  6.46it/s]
 33%|███▎      | 33/100 [00:05<00:10,  6.59it/s]
 34%|███▍      | 34/100 [00:05<00:09,  6.70it/s]
 35%|███▌      | 35/100 [00:05<00:09,  6.69it/s]
 36%|███▌      | 36/100 [00:05<00:09,  6.54it/s]
 37%|███▋      | 37/100 [00:05<00:09,  6.41it/s]
 38%|███▊      | 38/100 [00:05<00:09,  6.52it/s]
 39%|███▉      | 39/100 [00:06<00:09,  6.51it/s]
 40%|████      | 40/100 [00:06<00:09,  6.51it/s]
 41%|████      | 41/100 [00:06<00:09,  6.50it/s]
 42%|████▏     | 42/100 [00:06<00:08,  6.55it/s]
 43%|████▎     | 43/100 [00:06<00:08,  6.65it/s]
 44%|████▍     | 44/100 [00:06<00:08,  6.72it/s]
 45%|████▌     | 45/100 [00:06<00:08,  6.28it/s]
 46%|████▌     | 46/100 [00:07<00:08,  6.39it/s]
 47%|████▋     | 47/100 [00:07<00:08,  6.49it/s]
 48%|████▊     | 48/100 [00:07<00:07,  6.54it/s]
 49%|████▉     | 49/100 [00:07<00:07,  6.63it/s]
 50%|█████     | 50/100 [00:07<00:07,  6.69it/s]
 51%|█████     | 51/100 [00:07<00:07,  6.83it/s]
 52%|█████▏    | 52/100 [00:08<00:07,  6.67it/s]
 53%|█████▎    | 53/100 [00:08<00:07,  6.43it/s]
 54%|█████▍    | 54/100 [00:08<00:07,  6.46it/s]
 55%|█████▌    | 55/100 [00:08<00:06,  6.62it/s]
 56%|█████▌    | 56/100 [00:08<00:06,  6.69it/s]
 57%|█████▋    | 57/100 [00:08<00:06,  6.32it/s]
 58%|█████▊    | 58/100 [00:08<00:06,  6.22it/s]
 59%|█████▉    | 59/100 [00:09<00:06,  6.29it/s]
 60%|██████    | 60/100 [00:09<00:06,  6.22it/s]
 61%|██████    | 61/100 [00:09<00:06,  6.40it/s]
 62%|██████▏   | 62/100 [00:09<00:06,  6.20it/s]
 63%|██████▎   | 63/100 [00:09<00:05,  6.17it/s]
 64%|██████▍   | 64/100 [00:09<00:05,  6.37it/s]
 65%|██████▌   | 65/100 [00:10<00:05,  6.45it/s]
 66%|██████▌   | 66/100 [00:10<00:05,  6.62it/s]
 67%|██████▋   | 67/100 [00:10<00:05,  6.59it/s]
 68%|██████▊   | 68/100 [00:10<00:04,  6.82it/s]
 69%|██████▉   | 69/100 [00:10<00:04,  6.73it/s]
 70%|███████   | 70/100 [00:10<00:04,  6.69it/s]
 71%|███████   | 71/100 [00:10<00:04,  6.72it/s]
 72%|███████▏  | 72/100 [00:11<00:04,  6.72it/s]
 73%|███████▎  | 73/100 [00:11<00:04,  6.19it/s]
 74%|███████▍  | 74/100 [00:11<00:04,  6.41it/s]
 75%|███████▌  | 75/100 [00:11<00:03,  6.63it/s]
 76%|███████▌  | 76/100 [00:11<00:03,  6.68it/s]
 77%|███████▋  | 77/100 [00:11<00:03,  6.60it/s]
 78%|███████▊  | 78/100 [00:12<00:03,  6.53it/s]
 79%|███████▉  | 79/100 [00:12<00:03,  6.48it/s]
 80%|████████  | 80/100 [00:12<00:03,  6.29it/s]
 81%|████████  | 81/100 [00:12<00:02,  6.36it/s]
 82%|████████▏ | 82/100 [00:12<00:02,  6.65it/s]
 83%|████████▎ | 83/100 [00:12<00:02,  6.72it/s]
 84%|████████▍ | 84/100 [00:12<00:02,  6.62it/s]
 85%|████████▌ | 85/100 [00:13<00:02,  6.80it/s]
 86%|████████▌ | 86/100 [00:13<00:02,  6.94it/s]
 87%|████████▋ | 87/100 [00:13<00:01,  7.06it/s]
 88%|████████▊ | 88/100 [00:13<00:01,  6.88it/s]
 89%|████████▉ | 89/100 [00:13<00:01,  6.79it/s]
 90%|█████████ | 90/100 [00:13<00:01,  6.72it/s]
 91%|█████████ | 91/100 [00:13<00:01,  6.65it/s]
 92%|█████████▏| 92/100 [00:14<00:01,  6.51it/s]
 93%|█████████▎| 93/100 [00:14<00:01,  6.50it/s]
 94%|█████████▍| 94/100 [00:14<00:00,  6.41it/s]
 95%|█████████▌| 95/100 [00:14<00:00,  6.49it/s]
 96%|█████████▌| 96/100 [00:14<00:00,  6.73it/s]
 97%|█████████▋| 97/100 [00:14<00:00,  6.67it/s]
 98%|█████████▊| 98/100 [00:15<00:00,  6.84it/s]
 99%|█████████▉| 99/100 [00:15<00:00,  6.59it/s]
100%|██████████| 100/100 [00:15<00:00,  6.48it/s]
100%|██████████| 100/100 [00:15<00:00,  6.51it/s]

  0%|          | 0/71257 [00:00<?, ?it/s]
  6%|▌         | 4416/71257 [00:00<00:01, 44158.46it/s]
 14%|█▍        | 9851/71257 [00:00<00:01, 46788.96it/s]
 24%|██▍       | 17090/71257 [00:00<00:01, 52341.64it/s]
 35%|███▍      | 24624/71257 [00:00<00:00, 57617.29it/s]
 45%|████▌     | 32320/71257 [00:00<00:00, 62314.00it/s]
 57%|█████▋    | 40327/71257 [00:00<00:00, 66752.98it/s]
 68%|██████▊   | 48448/71257 [00:00<00:00, 70517.83it/s]
 80%|████████  | 57221/71257 [00:00<00:00, 74927.58it/s]
 94%|█████████▎| 66626/71257 [00:00<00:00, 79792.73it/s]
100%|██████████| 71257/71257 [00:00<00:00, 75426.71it/s]root dir
correct dir
Server terminal Started
Server Started ...
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
122.168.90.167 - - [29/May/2020 13:26:34] "GET /getOneDayClusterData/2020-05-06/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:26:34] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:26:37] "GET /getOneDayClusterData/2020-05-06/89 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:26:37] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
Script completed


 2350 new domains scrapped

Adjusting ranks...
performing updates on global_data.....
Adding new records...
Successfully added new records
Deleting records prior to 2020-04-28 ...
Successfully removed 4205 entry
updating ranks in globaldata...
Successfully update ranks
updating date...
Successfully update date
updating trends.......
updating cluster info...
Successfully update cluster no.
updating size...
updating rank...
Finding all keywords ...
updating keywords...
Trends Updated
updating visited domains...
clearing temporary data...
SUCCESS

122.168.90.167 - - [29/May/2020 13:26:49] "GET /getOneDayClusterData/2020-05-06/54 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:26:49] "GET /getclusterurl/54/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:26:54] "GET /getOneDayClusterData/2020-05-20/54 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:28:41] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:28:41] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:28:54] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:28:59] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:28:59] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:31:58] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:31:58] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:32:55] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:35:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:35:26] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:36:09] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:36:23] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:36:23] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:39:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:39:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:39:25] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:39:27] "GET /getOneDayClusterData/2020-05-29/66 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:39:27] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:48:56] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:48:56] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:49:00] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:49:00] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:51:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:51:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:56:50] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 13:56:50] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:39:00] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:39:13] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:26] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:29] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:29] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:38] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:39] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 14:53:39] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 15:33:40] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 15:52:55] "GET /favicon.ico HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:49:17] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:49:22] "GET /getOneDayClusterData/2020-05-29/84 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:49:22] "GET /getclusterurl/84/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:49:26] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:49:26] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:04] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:07] "GET /getOneDayClusterData/2020-05-29/38 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:07] "GET /getclusterurl/38/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:09] "GET /getOneDayClusterData/2020-05-29/37 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:09] "GET /getclusterurl/37/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 16:50:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 16:54:08] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
47.247.218.34 - - [29/May/2020 16:54:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 16:54:09] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 16:55:49] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:00:04] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:02:35] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:02:51] "GET /search/query/web HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:03] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:09] "GET /search/query/server HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:12] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:19] "GET /search/query/shopping HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:27] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:27] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 17:03:34] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.56.245.116 - - [29/May/2020 17:03:34] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:03:40] "GET /search/query/work/clusterno/12 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:12:45] "GET /https://stackoverflow.com//1 HTTP/1.1" 308 -
27.62.222.209 - - [29/May/2020 17:12:46] "GET /https://stackoverflow.com/1 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:13:02] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
27.62.222.209 - - [29/May/2020 17:13:02] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:13:02] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:13:12] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
157.34.18.197 - - [29/May/2020 17:13:22] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
157.34.18.197 - - [29/May/2020 17:13:22] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
157.34.18.197 - - [29/May/2020 17:13:22] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
157.34.18.197 - - [29/May/2020 17:13:30] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
157.34.18.197 - - [29/May/2020 17:13:54] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
213.217.0.184 - - [29/May/2020 17:20:16] "GET /photo/p/api/album.php HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:26:14] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:26:15] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:26:29] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:26:30] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:28:13] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:28:13] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:28:13] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
27.62.222.209 - - [29/May/2020 17:28:15] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:28:22] "GET /search/query/education/clusterno/12 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:28:34] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:28:50] "GET /search/query/educatioin HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
    resp = make_response(f(*args, **kwargs))
  File "/home/ec2-user/webcat/app.py", line 69, in query_search
    return json.dumps(search_by_query(q,cluster_no-1))
  File "/home/ec2-user/webcat/webtools/search.py", line 15, in search_by_query
    sites = sorted([(norm(row[1]-embd), row[0], row[3] if row[3] != -1 else DB_DEFAULT_RANK) for row in site_info_by_cluster(cluster, limit=limit)], key=lambda x: x[0])
  File "/home/ec2-user/webcat/webtools/search.py", line 15, in <listcomp>
    sites = sorted([(norm(row[1]-embd), row[0], row[3] if row[3] != -1 else DB_DEFAULT_RANK) for row in site_info_by_cluster(cluster, limit=limit)], key=lambda x: x[0])
TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
47.247.218.34 - - [29/May/2020 17:28:59] "GET /search/query/education HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:29:34] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:29:44] "GET /search/query/schol HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:29:57] "GET /search/query/school HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:31:20] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.218.34 - - [29/May/2020 17:31:27] "GET /search/domain/google HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:34:02] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:35:14] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:35:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:35:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:35:21] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.81.156 - - [29/May/2020 17:37:57] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:38:16] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.81.156 - - [29/May/2020 17:38:46] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:39:03] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
122.168.90.167 - - [29/May/2020 17:39:03] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:39:03] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.2.239 - - [29/May/2020 17:39:17] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.84.205 - - [29/May/2020 17:39:23] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
47.247.84.205 - - [29/May/2020 17:39:23] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
47.247.2.239 - - [29/May/2020 17:39:28] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
47.247.2.239 - - [29/May/2020 17:39:28] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.2.239 - - [29/May/2020 17:39:29] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.84.205 - - [29/May/2020 17:39:36] "GET /shopping/1 HTTP/1.1" 200 -
47.247.81.156 - - [29/May/2020 17:40:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.39.66 - - [29/May/2020 17:40:20] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
47.247.39.66 - - [29/May/2020 17:40:20] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:40:30] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:43:13] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:43:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:43:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:43:47] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:44:47] "GET /search/query/marketing HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:44:49] "GET /search/query/marketing HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:45:30] "GET /search/query/education HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:45:34] "GET /getOneDayClusterData/2020-05-29/94 HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:45:35] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:46:21] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
47.247.120.215 - - [29/May/2020 17:46:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:46:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:47:39] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:51:02] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 17:51:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:55:55] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
47.247.62.153 - - [29/May/2020 17:56:06] "GET /search/domain/education HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:56:08] "GET /search/domain/education HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:56:08] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
106.66.19.118 - - [29/May/2020 17:56:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:56:09] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 17:56:32] "GET /search/domain/google HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:56:35] "GET /facebook/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:56:39] "GET /facebook/1 HTTP/1.1" 200 -
one day cluster day data 2020-05-05 2020-05-06
one day cluster day data 2020-05-05 2020-05-06
one day cluster day data 2020-05-05 2020-05-06
one day cluster day data 2020-05-19 2020-05-20
one day cluster day data 2020-05-27 2020-05-28
favicon.ico
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
['web']
['server']
['shopping']
one day cluster day data 2020-05-27 2020-05-28
['work']
https://stackoverflow.com
photo/p/api/album.php
fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)

one day cluster day data 2020-05-27 2020-05-28
['education']
['educatioin']
['education']
['schol']
['school']
one day cluster day data 2020-05-27 2020-05-28
shopping
fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)

one day cluster day data 2020-05-27 2020-05-28
['marketing']
['marketing']
['education']
one day cluster day data 2020-05-27 2020-05-28
facebook
facebook
google
fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)
106.66.19.118 - - [29/May/2020 17:56:47] "GET /google/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:57:23] "GET /https://www.facebook.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:57:41] "GET /https://www.amazon.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:57:51] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:57:51] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:57:52] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:58:45] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:58:53] "GET /getOneDayClusterData/2020-05-29/18 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:58:53] "GET /getclusterurl/18/page/0 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:04] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:04] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:20] "GET /https://www.amazon.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:46] "GET /https://www.flipkart.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:47] "GET /https://www.flipkart.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 17:59:59] "GET /https://www.myntra.com/1 HTTP/1.1" 200 -
47.247.120.215 - - [29/May/2020 18:00:13] "GET /search/query/marketing HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 18:00:14] "GET /https://www.shein.com/1 HTTP/1.1" 200 -
106.66.19.118 - - [29/May/2020 18:00:25] "GET /https://www.compass.com/1 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:04:05] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:04:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:04:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:11:17] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:11:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:11:31] "GET /getOneDayClusterData/2020-05-29/66 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:11:31] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:11:34] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:11:34] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:11:35] "GET /getOneDayClusterData/2020-05-29/66 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:11:35] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:12:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:12:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:21] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
1.23.122.13 - - [29/May/2020 18:12:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:37] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:39] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:42] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:43] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:54] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:54] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:55] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:12:58] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:13:10] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:13:13] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:13:15] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:13:24] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:13:34] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:15:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:15:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:15:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:15:26] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:15:26] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:16:49] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:32] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:54] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:54] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:17:55] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:18:02] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:20:53] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:20:53] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:25] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:25] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:41] "GET /getclusterurl/13/page/1 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:46] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:22:47] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:23:05] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:23:06] "GET /getOneDayClusterData/2020-05-29/34 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 18:23:07] "GET /getclusterurl/34/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:18] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:19] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:31] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:32] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:23:58] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:03] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:03] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:10] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:11] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:12] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:12] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:33] "GET /getOneDayClusterData/2020-05-29/86 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:34] "GET /getclusterurl/86/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:40] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:41] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:24:48] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:22] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:22] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:23] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:43] "GET /getOneDayClusterData/2020-05-29/48 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:43] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:26:43] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:00] "GET /getOneDayClusterData/2020-05-29/47 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:00] "GET /getclusterurl/47/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:01] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:01] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:03] "GET /getOneDayClusterData/2020-05-29/2 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:03] "GET /getclusterurl/2/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:04] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:04] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:27:05] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:24] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:24] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:26] "GET /getOneDayClusterData/2020-05-29/5 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:26] "GET /getclusterurl/5/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:27] "GET /getOneDayClusterData/2020-05-29/7 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:27] "GET /getclusterurl/7/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:46] "GET /getOneDayClusterData/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:46:46] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:04] "GET /getOneDayClusterData/2020-05-29/96 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:04] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:12] "GET /getOneDayClusterData/2020-05-29/93 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:13] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:17] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:35] "GET /getClusterData/2020-05-19/2020-05-29/13 HTTP/1.1" 200 -
1.23.122.13 - - [29/May/2020 18:47:35] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:02] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:02] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:46] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:51] "GET /getOneDayClusterData/2020-05-29/62 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:51] "GET /getclusterurl/62/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:41:59] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:42:00] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:43:12] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:43:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:43:18] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 19:43:35] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:01:52] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:16] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:22] "GET /getOneDayClusterData/2020-05-29/62 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:22] "GET /getclusterurl/62/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:35] "GET /getOneDayClusterData/2020-05-29/65 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:02:35] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:11] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:36] "GET /getOneDayClusterData/2020-05-29/37 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:36] "GET /getclusterurl/37/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:58] "GET /getOneDayClusterData/2020-05-29/64 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:03:58] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:06] "GET /getOneDayClusterData/2020-05-29/65 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:06] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:35] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:35] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:36] "GET /getOneDayClusterData/2020-05-29/65 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:04:36] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:03] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:03] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:07] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:15] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:20] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:05:20] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:07:00] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:07:01] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:07:01] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
185.153.199.201 - - [29/May/2020 20:08:24] code 400, message Bad HTTP/0.9 request type ('\x03\x00\x00/*à\x00\x00\x00\x00\x00Cookie:')
185.153.199.201 - - [29/May/2020 20:08:24] "/*àCookie: mstshash=Administr" HTTPStatus.BAD_REQUEST -
122.168.90.167 - - [29/May/2020 20:33:20] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:25] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:25] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:34] "GET /getOneDayClusterData/2020-05-29/90 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:34] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:35] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:36] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:45] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:45] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:46] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:33:47] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:22] "GET /getOneDayClusterData/2020-05-29/40 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:22] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:39] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:39] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:52] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:34:52] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:35:27] "GET /getAllClusterDataOfSize/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:35:30] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:35:51] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:35:54] "GET /getOneDayClusterData/2020-05-29/32 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:35:54] "GET /getclusterurl/32/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:24] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:24] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:25] "GET /getOneDayClusterData/2020-05-29/32 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:26] "GET /getclusterurl/32/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:33] "GET /getOneDayClusterData/2020-05-29/12 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 20:36:33] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
185.153.199.201 - - [29/May/2020 20:40:10] code 400, message Bad HTTP/0.9 request type ('\x03\x00\x00/*à\x00\x00\x00\x00\x00Cookie:')
185.153.199.201 - - [29/May/2020 20:40:10] "/*àCookie: mstshash=Administr" HTTPStatus.BAD_REQUEST -
122.168.90.167 - - [29/May/2020 21:02:01] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:02] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:04] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:07] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:07] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:02:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
185.153.199.201 - - [29/May/2020 21:02:35] code 400, message Bad HTTP/0.9 request type ('\x03\x00\x00/*à\x00\x00\x00\x00\x00Cookie:')
185.153.199.201 - - [29/May/2020 21:02:35] "/*àCookie: mstshash=Administr" HTTPStatus.BAD_REQUEST -
122.168.90.167 - - [29/May/2020 21:11:52] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:11:59] "GET /getOneDayClusterData/2020-05-29/53 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:11:59] "GET /getclusterurl/53/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:12:56] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:13:03] "GET /getOneDayClusterData/2020-05-29/97 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:13:03] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:14:26] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:14:29] "GET /getOneDayClusterData/2020-05-29/96 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:14:29] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:14:36] "GET /getClusterData/2020-05-19/2020-05-28/96 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:09] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:25] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:28] "GET /getOneDayClusterData/2020-05-28/93 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:28] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:43] "GET /getClusterData/2020-05-07/2020-05-28/93 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:15:52] "GET /getClusterData/2020-05-01/2020-05-28/93 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:16:06] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:16:09] "GET /getOneDayClusterData/2020-05-28/95 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:16:09] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:16:18] "GET /getClusterData/2020-05-04/2020-05-28/95 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:13] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:20] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:23] "GET /getOneDayClusterData/2020-05-28/94 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:23] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:25] "GET /getClusterData/2020-05-04/2020-05-28/94 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:43] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:47] "GET /getOneDayClusterData/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:47] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:17:49] "GET /getClusterData/2020-04-29/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:18:01] "GET /getOneDayClusterData/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:18:05] "GET /getOneDayClusterData/2020-05-28/95 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:18:05] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:18:26] "GET /getClusterData/2020-04-29/2020-05-28/95 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:18:36] "GET /getOneDayClusterData/2020-05-28/95 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:19:23] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:19:29] "GET /getOneDayClusterData/2020-05-28/94 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:19:29] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:19:32] "GET /getClusterData/2020-04-29/2020-05-28/94 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:20:07] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:04] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:08] "GET /getOneDayClusterData/2020-05-28/100 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:08] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:17] "GET /getClusterData/2020-04-29/2020-05-28/100 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:35] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:37] "GET /getOneDayClusterData/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:37] "GET /getclusterurl/56/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:21:39] "GET /getClusterData/2020-04-29/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:07] "GET /getOneDayClusterData/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:26] "GET /getClusterData/2020-04-29/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:30] "GET /getOneDayClusterData/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:32] "GET /getClusterData/2020-04-29/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:35] "GET /getOneDayClusterData/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:37] "GET /getClusterData/2020-04-29/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:47] "GET /getOneDayClusterData/2020-05-28/56 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:53] "GET /getOneDayClusterData/2020-05-28/91 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:22:53] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:32:42] "GET /getClusterData/2020-04-29/2020-05-28/91 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:32:46] "GET /getOneDayClusterData/2020-05-28/91 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:00] "GET /getClusterData/2020-04-29/2020-05-28/91 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:05] "GET /getOneDayClusterData/2020-05-28/91 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:08] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:08] "GET /getAllClusterDataOfRank//2020-05-09 HTTP/1.1" 308 -
122.168.90.167 - - [29/May/2020 21:33:09] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:15] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:43] "GET /getOneDayClusterData/2020-05-28/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:33:43] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:34:50] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:34:50] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:06] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:11] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:11] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:36] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:40] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:36:40] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:07] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:12] "GET /search/query/myntra HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/ec2-user/webcat/venv/lib/python3.7/site-packages/flask_cors/decorator.py", line 128, in wrapped_function
    resp = make_response(f(*args, **kwargs))
  File "/home/ec2-user/webcat/app.py", line 69, in query_search
    return json.dumps(search_by_query(q,cluster_no-1))
  File "/home/ec2-user/webcat/webtools/search.py", line 15, in search_by_query
    sites = sorted([(norm(row[1]-embd), row[0], row[3] if row[3] != -1 else DB_DEFAULT_RANK) for row in site_info_by_cluster(cluster, limit=limit)], key=lambda x: x[0])
  File "/home/ec2-user/webcat/webtools/search.py", line 15, in <listcomp>
    sites = sorted([(norm(row[1]-embd), row[0], row[3] if row[3] != -1 else DB_DEFAULT_RANK) for row in site_info_by_cluster(cluster, limit=limit)], key=lambda x: x[0])
TypeError: unsupported operand type(s) for -: 'float' and 'NoneType'
122.168.90.167 - - [29/May/2020 21:37:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:22] "GET /search/domain/myntra HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:24] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:47] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:37:47] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:38:36] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:38:36] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:38:49] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:38:51] "GET /getOneDayClusterData/2020-05-28/34 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:38:51] "GET /getclusterurl/34/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:39:01] "GET /getClusterData/2020-04-29/2020-05-28/34 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:49:39] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:49:42] "GET /getOneDayClusterData/2020-05-29/87 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:49:42] "GET /getclusterurl/87/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:49:52] "GET /getClusterData/2020-05-02/2020-05-29/87 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:54:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:54:35] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:54:38] "GET /getOneDayClusterData/2020-05-28/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:54:38] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:54:40] "GET /getClusterData/2020-04-29/2020-05-28/88 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:06] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:13] "GET /getOneDayClusterData/2020-05-28/74 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:14] "GET /getclusterurl/74/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:22] "GET /getClusterData/2020-04-29/2020-05-28/74 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:52] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:58] "GET /getOneDayClusterData/2020-05-28/65 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:58] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:55:58] "GET /getClusterData/2020-04-02/2020-05-29/87 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:55:59] "GET /getClusterData/2020-04-29/2020-05-28/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:05] "GET /getClusterData/2020-04-02/2020-05-29/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:05] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:08] "GET /getClusterData/2020-04-02/2020-05-29/71 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:08] "GET /getclusterurl/71/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:10] "GET /getClusterData/2020-04-29/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:10] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:11] "GET /getClusterData/2020-04-02/2020-05-29/17 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:11] "GET /getclusterurl/17/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:14] "GET /getClusterData/2020-04-02/2020-05-29/16 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:14] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:18] "GET /getClusterData/2020-04-02/2020-05-29/15 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:18] "GET /getclusterurl/15/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:22] "GET /getClusterData/2020-04-02/2020-05-29/40 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:22] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:25] "GET /getClusterData/2020-04-02/2020-05-29/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:56:25] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:31] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:37] "GET /getOneDayClusterData/2020-05-28/89 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:37] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:56:38] "GET /getClusterData/2020-04-29/2020-05-28/89 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:58:43] "GET /getClusterData/2020-04-02/2020-05-29/94 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:58:43] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:58:57] "GET /getClusterData/2020-04-02/2020-05-29/75 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:58:57] "GET /getclusterurl/75/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:03] "GET /getClusterData/2020-04-02/2020-05-29/100 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:03] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:09] "GET /getClusterData/2020-04-02/2020-05-29/5 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:09] "GET /getclusterurl/5/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:12] "GET /getClusterData/2020-04-02/2020-05-29/37 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 21:59:12] "GET /getclusterurl/37/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 21:59:59] "GET /getOneDayClusterData/2020-05-28/89 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:23] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:34] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:38] "GET /getOneDayClusterData/2020-05-28/64 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:38] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:39] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:39] "GET /getOneDayClusterData/2020-05-28/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:40] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:00:43] "GET /getClusterData/2020-04-29/2020-05-28/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:01:42] "GET /getClusterData/2020-04-04/2020-05-28/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:15] "GET /getClusterData/2020-04-04/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:15] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:37] "GET /getClusterData/2020-04-04/2020-05-28/99 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:37] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:45] "GET /getClusterData/2020-04-04/2020-05-28/98 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:45] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:54] "GET /getClusterData/2020-04-04/2020-05-28/99 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:54] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:58] "GET /getClusterData/2020-04-04/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:02:58] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:03:02] "GET /getOneDayClusterData/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:03:08] "GET /getOneDayClusterData/2020-05-28/99 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:03:09] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:05:21] "GET /getOneDayClusterData/2020-05-28/1 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:05:21] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:05:46] "GET /getClusterData/2020-04-04/2020-05-28/1 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:08:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:08:57] "GET /getOneDayClusterData/2020-05-28/44 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:08:57] "GET /getclusterurl/44/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:09:00] "GET /getClusterData/2020-04-29/2020-05-28/44 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:09:06] "GET /getClusterData/2020-04-04/2020-05-28/44 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:10:57] "GET /getClusterData/2020-04-04/2020-05-28/24 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:10:57] "GET /getclusterurl/24/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:11:01] "GET /getOneDayClusterData/2020-05-28/24 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:11:05] "GET /getClusterData/2020-04-04/2020-05-28/24 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:11:31] "GET /getClusterData/2020-04-04/2020-05-28/13 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:11:32] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:11:32] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:06] "GET /getClusterData/2020-04-04/2020-05-28/14 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:06] "GET /getclusterurl/14/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:09] "GET /getOneDayClusterData/2020-05-28/14 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:11] "GET /getOneDayClusterData/2020-05-28/100 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:11] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:14] "GET /getOneDayClusterData/2020-05-28/99 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:14] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:17] "GET /getOneDayClusterData/2020-05-28/98 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:17] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:21] "GET /getOneDayClusterData/2020-05-28/97 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:21] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:21] "GET /getOneDayClusterData/2020-05-28/98 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:21] "GET /getclusterurl/98/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:31] "GET /netflix.com/1 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:34] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:37] "GET /getOneDayClusterData/2020-05-28/38 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:37] "GET /getclusterurl/38/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:40] "GET /getClusterData/2020-04-29/2020-05-28/38 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:42] "GET /getOneDayClusterData/2020-05-28/38 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:12:46] "GET /getClusterData/2020-04-29/2020-05-28/38 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:13:10] "GET /getClusterData/2020-04-04/2020-05-28/38 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:15:01] "GET /msg91.com/1 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:15:06] "GET /getClusterData/2020-04-04/2020-05-28/97 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:15:06] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:15:09] "GET /getOneDayClusterData/2020-05-28/97 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:15:12] "GET /getClusterData/2020-04-04/2020-05-28/97 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:16:07] "GET /getclusterurl/97/page/9 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:16:23] "GET /getclusterurl/97/page/25 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:16:34] "GET /getclusterurl/97/page/16 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:17:27] "GET /getclusterurl/97/page/106 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:17:33] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:17:42] "GET /getclusterurl/97/page/9 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:18:12] "GET /getclusterurl/97/page/15 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:18:20] "GET /getclusterurl/97/page/18 HTTP/1.1" 200 -
27.56.215.16 - - [29/May/2020 22:19:39] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:22:26] "GET /getclusterurl/97/page/9 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:24:52] "GET /getOneDayClusterData/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:04] "GET /walkover.in/1 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:12] "GET /getClusterData/2020-04-04/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:20] "GET /getClusterData/2020-04-04/2020-05-28/80 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:20] "GET /getclusterurl/80/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:25] "GET /getClusterData/2020-04-04/2020-05-28/88 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:25] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:30] "GET /getClusterData/2020-04-04/2020-05-28/80 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:25:30] "GET /getclusterurl/80/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:26:41] "GET /getClusterData/2020-05-09/2020-05-28/80 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:29] "GET /getClusterData/2020-05-09/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:29] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:41] "GET /getClusterData/2020-05-09/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:41] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:44] "GET /getClusterData/2020-05-09/2020-05-28/99 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:44] "GET /getclusterurl/99/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:46] "GET /getClusterData/2020-05-09/2020-05-28/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:27:47] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:28:07] "GET /getOneDayClusterData/2020-05-28/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:28:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:28:17] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:18] "GET /getOneDayClusterData/2020-05-28/95 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:18] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:23] "GET /getClusterData/2020-05-09/2020-05-28/95 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:32] "GET /getClusterData/2020-04-04/2020-05-28/95 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:45] "GET /getClusterData/2020-04-04/2020-05-28/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:45] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:48] "GET /getClusterData/2020-04-04/2020-05-28/95 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:48] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:53] "GET /getClusterData/2020-04-04/2020-05-28/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:29:53] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:30:03] "GET /getClusterData/2020-04-04/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:30:03] "GET /getclusterurl/97/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:30:10] "GET /getOneDayClusterData/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:17] "GET /getClusterData/2020-04-04/2020-05-28/97 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:22] "GET /getClusterData/2020-04-04/2020-05-28/96 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:22] "GET /getclusterurl/96/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:26] "GET /getClusterData/2020-04-04/2020-05-28/95 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:26] "GET /getclusterurl/95/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:29] "GET /getClusterData/2020-04-04/2020-05-28/94 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:29] "GET /getclusterurl/94/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:31] "GET /getClusterData/2020-04-04/2020-05-28/93 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:31] "GET /getclusterurl/93/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:33] "GET /getClusterData/2020-04-04/2020-05-28/92 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:34] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:36] "GET /getClusterData/2020-04-04/2020-05-28/91 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:36] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:38] "GET /getClusterData/2020-04-04/2020-05-28/90 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:38] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:40] "GET /getClusterData/2020-04-04/2020-05-28/89 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:40] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:42] "GET /getClusterData/2020-04-04/2020-05-28/90 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:42] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:52] "GET /getClusterData/2020-04-04/2020-05-28/89 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:52] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:32:57] "GET / HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:33:00] "GET /favicon.ico HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:22] "GET /getClusterData/2020-04-04/2020-05-28/88 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:22] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:24] "GET /getClusterData/2020-04-04/2020-05-28/87 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:24] "GET /getclusterurl/87/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:28] "GET /getOneDayClusterData/2020-05-28/87 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:33] "GET /getClusterData/2020-04-04/2020-05-28/87 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:36] "GET /getClusterData/2020-04-04/2020-05-28/86 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:36] "GET /getclusterurl/86/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:39] "GET /getClusterData/2020-04-04/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:34:39] "GET /getclusterurl/85/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:35:19] "GET /getOneDayClusterData/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:35:35] "GET /getclusterurl/85/page/1 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:35:52] "GET /getclusterurl/85/page/2 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:36:05] "GET /getclusterurl/85/page/3 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:36:11] "GET /getclusterurl/85/page/4 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:36:26] "GET /getclusterurl/85/page/5 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:36:50] "GET /getclusterurl/85/page/6 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:36:59] "GET /getClusterData/2020-04-04/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:14] "GET /getOneDayClusterData/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:16] "GET /getClusterData/2020-04-04/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:18] "GET /getClusterData/2020-04-04/2020-05-28/84 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:18] "GET /getclusterurl/84/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:22] "GET /getClusterData/2020-04-04/2020-05-28/83 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:22] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:24] "GET /getClusterData/2020-04-04/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:24] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:27] "GET /getClusterData/2020-04-04/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:27] "GET /getclusterurl/81/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:28] "GET /getClusterData/2020-04-04/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:28] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:42] "GET /getOneDayClusterData/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:46] "GET /getOneDayClusterData/2020-05-28/83 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:47] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:48] "GET /getOneDayClusterData/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:48] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:37:51] "GET /getClusterData/2020-04-04/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:06] "GET /getOneDayClusterData/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:09] "GET /getOneDayClusterData/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:09] "GET /getclusterurl/81/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:15] "GET /getClusterData/2020-04-04/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:21] "GET /getOneDayClusterData/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:23] "GET /getClusterData/2020-04-04/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:30] "GET /getClusterData/2020-04-04/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:30] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:36] "GET /getClusterData/2020-04-04/2020-05-28/83 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:36] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:39] "GET /getClusterData/2020-04-04/2020-05-28/84 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:39] "GET /getclusterurl/84/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:42] "GET /getClusterData/2020-04-04/2020-05-28/85 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:42] "GET /getclusterurl/85/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:43] "GET /getClusterData/2020-04-04/2020-05-28/84 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:38:43] "GET /getclusterurl/84/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:24] "GET /getClusterData/2020-04-04/2020-05-28/83 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:24] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:27] "GET /getClusterData/2020-04-04/2020-05-28/82 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:27] "GET /getclusterurl/82/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:29] "GET /getClusterData/2020-04-04/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:29] "GET /getclusterurl/81/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:36] "GET /getOneDayClusterData/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:47] "GET /getClusterData/2020-04-04/2020-05-28/81 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:50] "GET /getClusterData/2020-04-04/2020-05-28/79 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:50] "GET /getclusterurl/79/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:53] "GET /getClusterData/2020-04-04/2020-05-28/78 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:53] "GET /getclusterurl/78/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:57] "GET /getClusterData/2020-04-04/2020-05-28/77 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:39:57] "GET /getclusterurl/77/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:03] "GET /getOneDayClusterData/2020-05-28/77 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:08] "GET /getClusterData/2020-04-04/2020-05-28/77 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:11] "GET /getClusterData/2020-04-04/2020-05-28/76 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:12] "GET /getclusterurl/76/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:14] "GET /getClusterData/2020-04-04/2020-05-28/75 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:14] "GET /getclusterurl/75/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:16] "GET /getClusterData/2020-04-04/2020-05-28/74 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:16] "GET /getclusterurl/74/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:18] "GET /getClusterData/2020-04-04/2020-05-28/73 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:18] "GET /getclusterurl/73/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:20] "GET /getClusterData/2020-04-04/2020-05-28/72 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:20] "GET /getclusterurl/72/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:22] "GET /getClusterData/2020-04-04/2020-05-28/73 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:22] "GET /getclusterurl/73/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:25] "GET /getClusterData/2020-04-04/2020-05-28/71 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:25] "GET /getclusterurl/71/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:28] "GET /getClusterData/2020-04-04/2020-05-28/70 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:28] "GET /getclusterurl/70/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:30] "GET /getClusterData/2020-04-04/2020-05-28/69 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:30] "GET /getclusterurl/69/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:32] "GET /getClusterData/2020-04-04/2020-05-28/68 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:32] "GET /getclusterurl/68/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:33] "GET /getClusterData/2020-04-04/2020-05-28/67 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:34] "GET /getclusterurl/67/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:36] "GET /getClusterData/2020-04-04/2020-05-28/66 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:36] "GET /getclusterurl/66/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:38] "GET /getClusterData/2020-04-04/2020-05-28/65 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:38] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:39] "GET /getClusterData/2020-04-04/2020-05-28/64 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:39] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:42] "GET /getClusterData/2020-04-04/2020-05-28/63 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:42] "GET /getclusterurl/63/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:43] "GET /getClusterData/2020-04-04/2020-05-28/62 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:43] "GET /getclusterurl/62/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:45] "GET /getClusterData/2020-04-04/2020-05-28/61 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:45] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:47] "GET /getClusterData/2020-04-04/2020-05-28/60 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:47] "GET /getclusterurl/60/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:50] "GET /getClusterData/2020-04-04/2020-05-28/59 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:51] "GET /getclusterurl/59/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:52] "GET /getClusterData/2020-04-04/2020-05-28/58 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:53] "GET /getclusterurl/58/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:54] "GET /getClusterData/2020-04-04/2020-05-28/57 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:40:54] "GET /getclusterurl/57/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:00] "GET /getOneDayClusterData/2020-05-28/57 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:13] "GET /getClusterData/2020-04-04/2020-05-28/57 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:17] "GET /getClusterData/2020-04-04/2020-05-28/56 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:17] "GET /getclusterurl/56/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:19] "GET /getClusterData/2020-04-04/2020-05-28/55 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:19] "GET /getclusterurl/55/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:22] "GET /getClusterData/2020-04-04/2020-05-28/54 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:22] "GET /getclusterurl/54/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:25] "GET /getClusterData/2020-04-04/2020-05-28/53 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:25] "GET /getclusterurl/53/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:29] "GET /getClusterData/2020-04-04/2020-05-28/52 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:29] "GET /getclusterurl/52/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:31] "GET /getClusterData/2020-04-04/2020-05-28/51 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:31] "GET /getclusterurl/51/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:33] "GET /getClusterData/2020-04-04/2020-05-28/50 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:33] "GET /getclusterurl/50/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:35] "GET /getClusterData/2020-04-04/2020-05-28/49 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:35] "GET /getclusterurl/49/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:37] "GET /getClusterData/2020-04-04/2020-05-28/48 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:37] "GET /getclusterurl/48/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:38] "GET /getClusterData/2020-04-04/2020-05-28/47 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:38] "GET /getclusterurl/47/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:40] "GET /getClusterData/2020-04-04/2020-05-28/46 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:40] "GET /getclusterurl/46/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:42] "GET /getClusterData/2020-04-04/2020-05-28/45 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:42] "GET /getclusterurl/45/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:44] "GET /getClusterData/2020-04-04/2020-05-28/44 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:44] "GET /getclusterurl/44/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:41:50] "GET /getOneDayClusterData/2020-05-28/44 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:25] "GET /getClusterData/2020-04-04/2020-05-28/44 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:28] "GET /getClusterData/2020-04-04/2020-05-28/43 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:28] "GET /getclusterurl/43/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:32] "GET /getClusterData/2020-04-04/2020-05-28/42 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:32] "GET /getclusterurl/42/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:34] "GET /getClusterData/2020-04-04/2020-05-28/41 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:34] "GET /getclusterurl/41/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:36] "GET /getClusterData/2020-04-04/2020-05-28/40 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:36] "GET /getclusterurl/40/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:38] "GET /getClusterData/2020-04-04/2020-05-28/39 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:38] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:40] "GET /getClusterData/2020-04-04/2020-05-28/38 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:40] "GET /getclusterurl/38/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:42] "GET /getClusterData/2020-04-04/2020-05-28/37 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:42] "GET /getclusterurl/37/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:44] "GET /getClusterData/2020-04-04/2020-05-28/36 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:44] "GET /getclusterurl/36/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:46] "GET /getClusterData/2020-04-04/2020-05-28/35 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:46] "GET /getclusterurl/35/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:47] "GET /getClusterData/2020-04-04/2020-05-28/34 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:47] "GET /getclusterurl/34/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:49] "GET /getClusterData/2020-04-04/2020-05-28/33 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:49] "GET /getclusterurl/33/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:52] "GET /getClusterData/2020-04-04/2020-05-28/32 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:52] "GET /getclusterurl/32/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:54] "GET /getClusterData/2020-04-04/2020-05-28/31 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:54] "GET /getclusterurl/31/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:56] "GET /getClusterData/2020-04-04/2020-05-28/30 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:42:56] "GET /getclusterurl/30/page/0 HTTP/1.1" 200 -

https://www.facebook.com
https://www.amazon.com
one day cluster day data 2020-05-27 2020-05-28
https://www.amazon.com
https://www.flipkart.com
https://www.flipkart.com
https://www.myntra.com
['marketing']
https://www.shein.com
https://www.compass.com
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
['myntra']
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
netflix.com
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
msg91.com
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
walkover.in
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28

fetch_url_info error encoding with 'idna' codec failed (UnicodeError: label empty or too long)
Traceback (most recent call last):
  File "/usr/lib64/python3.7/encodings/idna.py", line 165, in encode
    raise UnicodeError("label empty or too long")
UnicodeError: label empty or too long

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/webcat/webtools/processurl.py", line 37, in fetch_url_info
    url_info = get_text_content(url)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 153, in get_text_content
    data =  get_url_and_content(url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 134, in get_url_and_content
    content = get_scrapped_text('https://'+url, dynamic, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 117, in get_scrapped_text
    content = get_static_text_content(url, timeout)
  File "/home/ec2-user/webcat/webutils/scrapper.py", line 72, in get_static_text_content
    res = requests.get(url, headers=headers, verify=False, timeout=timeout, allow_redirects=True)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 308, in connect
    conn = self._new_conn()
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ec2-user/webcat/venv/lib64/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
UnicodeError: encoding with 'idna' codec failed (UnicodeError: label empty or too long)

favicon.ico
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data 2020-05-27 2020-05-28
one day cluster day data106.207.233.136 - - [29/May/2020 22:43:01] "GET /getOneDayClusterData/2020-05-28/30 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:05] "GET /getOneDayClusterData/2020-05-28/31 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:05] "GET /getclusterurl/31/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:08] "GET /getOneDayClusterData/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:08] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:11] "GET /getClusterData/2020-04-04/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:20] "GET /getOneDayClusterData/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:38] "GET /getClusterData/2020-04-04/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:41] "GET /getOneDayClusterData/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:43:43] "GET /getClusterData/2020-04-04/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:21] "GET /getClusterData/2020-04-04/2020-05-28/28 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:21] "GET /getclusterurl/28/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:25] "GET /getClusterData/2020-04-04/2020-05-28/27 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:25] "GET /getclusterurl/27/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:28] "GET /getClusterData/2020-04-04/2020-05-28/26 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:28] "GET /getclusterurl/26/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:29] "GET /getClusterData/2020-04-04/2020-05-28/25 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:29] "GET /getclusterurl/25/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:33] "GET /getClusterData/2020-04-04/2020-05-28/24 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:33] "GET /getclusterurl/24/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:35] "GET /getClusterData/2020-04-04/2020-05-28/23 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:35] "GET /getclusterurl/23/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:39] "GET /getClusterData/2020-04-04/2020-05-28/22 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:39] "GET /getclusterurl/22/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:41] "GET /getClusterData/2020-04-04/2020-05-28/21 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:41] "GET /getclusterurl/21/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:43] "GET /getClusterData/2020-04-04/2020-05-28/20 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:43] "GET /getclusterurl/20/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:45] "GET /getClusterData/2020-04-04/2020-05-28/19 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:45] "GET /getclusterurl/19/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:47] "GET /getClusterData/2020-04-04/2020-05-28/18 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:47] "GET /getclusterurl/18/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:49] "GET /getClusterData/2020-04-04/2020-05-28/19 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:50] "GET /getclusterurl/19/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:53] "GET /getOneDayClusterData/2020-05-28/19 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:57] "GET /getOneDayClusterData/2020-05-28/20 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:46:57] "GET /getclusterurl/20/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:47:09] "GET /getClusterData/2020-04-04/2020-05-28/20 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:47:40] "GET /getClusterData/2020-04-04/2020-05-28/29 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:47:40] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:47:40] "GET /getclusterurl/29/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:07] "GET /getClusterData/2020-04-04/2020-05-28/27 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:07] "GET /getclusterurl/27/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:12] "GET /getClusterData/2020-04-04/2020-05-28/1 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:12] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:14] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:16] "GET /getClusterData/2020-04-04/2020-05-28/2 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:16] "GET /getclusterurl/2/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:21] "GET /getClusterData/2020-04-04/2020-05-28/3 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:21] "GET /getclusterurl/3/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:23] "GET /getClusterData/2020-04-04/2020-05-28/4 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:23] "GET /getclusterurl/4/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:25] "GET /getClusterData/2020-04-04/2020-05-28/5 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:25] "GET /getclusterurl/5/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:28] "GET /getClusterData/2020-04-04/2020-05-28/6 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:28] "GET /getclusterurl/6/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:31] "GET /getClusterData/2020-04-04/2020-05-28/7 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:31] "GET /getclusterurl/7/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:54] "GET /getClusterData/2020-04-04/2020-05-28/8 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:55] "GET /getclusterurl/8/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:58] "GET /getClusterData/2020-04-04/2020-05-28/9 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:49:58] "GET /getclusterurl/9/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:00] "GET /getClusterData/2020-04-04/2020-05-28/10 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:00] "GET /getclusterurl/10/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:02] "GET /getClusterData/2020-04-04/2020-05-28/11 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:03] "GET /getclusterurl/11/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:05] "GET /getClusterData/2020-04-04/2020-05-28/12 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:05] "GET /getclusterurl/12/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:07] "GET /getClusterData/2020-04-04/2020-05-28/13 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:07] "GET /getclusterurl/13/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:14] "GET /getOneDayClusterData/2020-05-28/13 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:50:21] "GET /getClusterData/2020-04-04/2020-05-28/13 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:04] "GET /getClusterData/2020-04-04/2020-05-28/14 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:04] "GET /getclusterurl/14/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:07] "GET /getClusterData/2020-04-04/2020-05-28/15 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:07] "GET /getclusterurl/15/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:11] "GET /getClusterData/2020-04-04/2020-05-28/16 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:11] "GET /getclusterurl/16/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:13] "GET /getClusterData/2020-04-04/2020-05-28/17 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:13] "GET /getclusterurl/17/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:15] "GET /getClusterData/2020-04-04/2020-05-28/18 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:15] "GET /getclusterurl/18/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:24] "GET /getClusterData/2020-04-04/2020-05-28/19 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:24] "GET /getclusterurl/19/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:30] "GET /getClusterData/2020-04-04/2020-05-28/20 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:30] "GET /getclusterurl/20/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:33] "GET /getClusterData/2020-04-04/2020-05-28/21 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:33] "GET /getclusterurl/21/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:36] "GET /getClusterData/2020-04-04/2020-05-28/22 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:36] "GET /getclusterurl/22/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:38] "GET /getClusterData/2020-04-04/2020-05-28/23 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:38] "GET /getclusterurl/23/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:43] "GET /getClusterData/2020-04-04/2020-05-28/24 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:43] "GET /getclusterurl/24/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:45] "GET /getClusterData/2020-04-04/2020-05-28/25 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:45] "GET /getclusterurl/25/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:47] "GET /getClusterData/2020-04-04/2020-05-28/26 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:47] "GET /getclusterurl/26/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:49] "GET /getClusterData/2020-04-04/2020-05-28/27 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:49] "GET /getclusterurl/27/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:50] "GET /getClusterData/2020-04-04/2020-05-28/26 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:51] "GET /getclusterurl/26/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:54] "GET /getOneDayClusterData/2020-05-28/26 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:59] "GET /getOneDayClusterData/2020-05-28/27 HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 22:51:59] "GET /getclusterurl/27/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 22:55:09] "GET /getOneDayClusterData/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [29/May/2020 22:55:09] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
106.207.233.136 - - [29/May/2020 23:28:17] "GET /search/query/travel HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 23:28:25] "GET /search/query/travle HTTP/1.1" 200 -
106.207.233.136 - - [29/May/2020 23:28:45] "GET /search/query/travel HTTP/1.1" 200 -
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
106.207.233.136 - - [30/May/2020 00:00:17] "GET /msg91.com/1 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:22] "GET /msg91.com/1 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:26] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:30] "GET /getOneDayClusterData/2020-05-29/1 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:30] "GET /getclusterurl/1/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:32] "GET /getOneDayClusterData/2020-05-29/77 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:33] "GET /getclusterurl/77/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:39] "GET /getOneDayClusterData/2020-05-29/100 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:00:39] "GET /getclusterurl/100/page/0 HTTP/1.1" 200 -
54.160.219.172 - - [30/May/2020 00:01:20] "GET /getClusterInfo/100 HTTP/1.1" 200 -
initializing...
loading database updates...
loading kmeans..
loading word vectors...
READY
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
3.84.245.248 - - [30/May/2020 00:06:33] "GET /getClusterInfo/100 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:45] "GET /getClusterData/2020-04-29/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:48] "GET /getOneDayClusterData/2020-05-28/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:50] "GET /getOneDayClusterData/2020-05-28/65 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:51] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:53] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:21:53] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:22:20] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:23] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:25] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:26] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:29] "GET /getClusterData/2020-04-30/2020-05-29/61 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:22:31] "GET /getOneDayClusterData/2020-05-29/83 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:22:32] "GET /getclusterurl/83/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:34] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:36] "GET /getClusterData/2020-04-30/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:38] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:22:39] "GET /getClusterData/2020-04-30/2020-05-29/83 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:42] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:42] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:44] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:47] "GET /getOneDayClusterData/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:22:47] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:22:52] "GET /getOneDayClusterData/2020-05-29/83 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:24:05] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:24:07] "GET /getOneDayClusterData/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:24:07] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:24:08] "GET /getClusterData/2020-04-30/2020-05-29/83 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:24:18] "GET /getOneDayClusterData/2020-05-29/83 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:24:35] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:24:35] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:36] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:39] "GET /getOneDayClusterData/2020-05-29/35 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:39] "GET /getclusterurl/35/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:40] "GET /getclusterurl/35/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:40] "GET /getclusterurl/35/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:25:42] "GET /getClusterData/2020-04-30/2020-05-29/35 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:25:42] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:25:43] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:25:43] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:25:48] "GET /getClusterData/2020-04-30/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:25:50] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:27:30] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:27:36] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:27:36] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:30:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:30:59] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:31:38] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:31:39] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:31:41] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:31:41] "GET /getOneDayClusterData/2020-05-29/88/getClusterData/2020-04-30/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:31:41] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:33:38] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:33:41] "GET /getOneDayClusterData/2020-05-29/86 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:33:41] "GET /getOneDayClusterData/2020-05-29/86/getClusterData/2020-04-30/2020-05-29/86 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:33:41] "GET /getclusterurl/86/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:34:42] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:34:43] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:35:11] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:35:13] "GET /getOneDayClusterData/2020-05-29/53 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:35:13] "GET /getOneDayClusterData/2020-05-29/53/getClusterData/2020-04-30/2020-05-29/53 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:35:13] "GET /getclusterurl/53/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:08] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:11] "GET /getOneDayClusterData/2020-05-29/51 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:11] "GET /getClusterData/2020-04-30/2020-05-29/51 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:11] "GET /getclusterurl/51/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:35] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:38] "GET /getOneDayClusterData/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:38] "GET /getClusterData/2020-04-30/2020-05-29/88 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:36:38] "GET /getclusterurl/88/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:00] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:02] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:02] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:02] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:10] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:13] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:13] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:37:13] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:38:00] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:38:02] "GET /getOneDayClusterData/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:38:02] "GET /getClusterData/2020-04-30/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:38:02] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:38:22] "GET /getOneDayClusterData/2020-05-28/100 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:39:57] "GET /getOneDayClusterData/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:39:57] "GET /getClusterData/2020-04-30/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:40:03] "GET /getOneDayClusterData/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:40:03] "GET /getClusterData/2020-04-30/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:41:05] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:41:09] "GET /getOneDayClusterData/2020-05-29/59 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:41:09] "GET /getClusterData/2020-04-30/2020-05-29/59 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:41:09] "GET /getclusterurl/59/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:13] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:16] "GET /getOneDayClusterData/2020-05-29/64 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:16] "GET /getClusterData/2020-04-30/2020-05-29/64 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:17] "GET /getclusterurl/64/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:37] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:41] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:41] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:42:41] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:42:53] "GET /getClusterData/2020-04-04/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:43:00] "GET /getOneDayClusterData/2020-05-28/100 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:46:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:46:14] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:51:19] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:51:22] "GET /getOneDayClusterData/2020-05-29/63 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:51:22] "GET /getClusterData/2020-04-30/2020-05-29/63 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:51:22] "GET /getclusterurl/63/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:28] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:28] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:34] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:37] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:37] "GET /getClusterData/2020-04-30/2020-05-29/61 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:51:38] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:52:36] "GET /getOneDayClusterData/2020-05-29/39 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:52:36] "GET /getClusterData/2020-04-30/2020-05-29/39 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:52:36] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:52:56] "GET /getOneDayClusterData/2020-05-29/39 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:52:56] "GET /getClusterData/2020-04-04/2020-05-29/39 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:28] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:31] "GET /getOneDayClusterData/2020-05-29/63 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:31] "GET /getClusterData/2020-04-30/2020-05-29/63 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:31] "GET /getclusterurl/63/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:48] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:51] "GET /getOneDayClusterData/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:51] "GET /getClusterData/2020-04-30/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:54:51] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:01] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:01] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:14] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:16] "GET /getOneDayClusterData/2020-05-29/65 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:16] "GET /getClusterData/2020-04-30/2020-05-29/65 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:16] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 00:55:17] "GET /getclusterurl/65/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:21] "GET /getOneDayClusterData/2020-05-29/90 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:21] "GET /getClusterData/2020-04-30/2020-05-29/90 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:21] "GET /getclusterurl/90/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:22] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:23] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 00:56:23] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:04:18] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:04:21] "GET /getOneDayClusterData/2020-05-29/60 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:04:21] "GET /getClusterData/2020-04-30/2020-05-29/60 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:04:22] "GET /getclusterurl/60/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:06] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:06] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:06] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:07] "GET /getOneDayClusterData/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:07] "GET /getClusterData/2020-04-30/2020-05-29/91 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:07] "GET /getclusterurl/91/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:07] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:08] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:08] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:08] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:12] "GET /getOneDayClusterData/2020-05-29/7 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:12] "GET /getclusterurl/7/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:24] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:24] "GET /getOneDayClusterData/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:24] "GET /getClusterData/2020-04-30/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:24] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:27] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:27] "GET /getOneDayClusterData/2020-05-29/39 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:27] "GET /getClusterData/2020-04-30/2020-05-29/39 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:05:27] "GET /getclusterurl/39/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:28] "GET /getOneDayClusterData/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:28] "GET /getClusterData/2020-04-30/2020-05-29/89 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:05:28] "GET /getclusterurl/89/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:07:14] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:08:00] "GET /getOneDayClusterData/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:08:00] "GET /getClusterData/2020-04-30/2020-05-29/92 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:08:00] "GET /getclusterurl/92/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:26] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:32] "GET /getOneDayClusterData/2020-05-29/68 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:32] "GET /getClusterData/2020-04-30/2020-05-29/68 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:32] "GET /getclusterurl/68/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:46] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:46] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:54] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:57] "GET /getOneDayClusterData/2020-05-29/69 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:57] "GET /getClusterData/2020-04-30/2020-05-29/69 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:11:57] "GET /getclusterurl/69/page/0 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:12:21] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:12:22] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:15:56] "GET /getOneDayClusterData/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:15:57] "GET /getClusterData/2020-04-30/2020-05-29/61 HTTP/1.1" 200 -
122.168.90.167 - - [30/May/2020 01:15:57] "GET /getclusterurl/61/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:31:36] "GET /getclusterurl/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:31:40] "GET /getOneDayClusterData/2020-05-29/36 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:31:40] "GET /getClusterData/2020-04-30/2020-05-29/36 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:31:40] "GET /getclusterurl/36/page/0 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:32:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -
106.207.233.136 - - [30/May/2020 01:32:33] "GET /getAllClusterDataOfRank/2020-05-09 HTTP/1.1" 200 -