Newsdata_extractive_summ.ipynb

{"cells":[{"cell_type":"code","execution_count":2,"metadata":{"id":"fOzvp34IZQ13"},"outputs":[],"source":["# Bertsum directory chdir\n","import os\n","os.chdir('/home/dongwon/korbertsum/src')"]},{"cell_type":"markdown","metadata":{"id":"gwxwDqcMsIaA"},"source":["# Train"]},{"cell_type":"markdown","metadata":{"id":"ryPPxFlnq2yw"},"source":["**Arguments**\n","\n","-mode train (train, validate, test) # 학습 모드 설정\n","\n","-encoder classifier (classifier, transformer, RNN) # 학습 방법 설정\n","\n","-dropout 0.1 # dropout rate\n","\n","-bert_data_path ../bert_data/korean # bert.pt파일 저장되어있는 경로(pt파일 앞의 단어까지 경로에 쓸 것)\n","\n","-model_path ../models/bert_classifier_sample # train의 경우 학습된 모델 저장\n","\n","-visible_gpus 0 # -1일 경우 cpu, 0이상일 경우 cuda 사용\n","\n","-train_steps 학습 진행할 총 step의 수 (예: 50000)\n","\n","-save_checkpoint_steps 몇 step마다 pt파일 형태의 checkpoint를 저장할지 지정 (예: 10000)\n","\n","-warmup_steps 학습할때 학습률이 점점 늘어났다가 줄어드는데 그 늘어나는게 몇 step까지 적용되는지 지정\n","\n","-batch_size: 한번에 몇개의 토큰(문장)을 읽을지 결정함"]},{"cell_type":"code","execution_count":7,"metadata":{"id":"za8H-D8aZrP0"},"outputs":[{"name":"stdout","output_type":"stream","text":["[2023-10-14 15:57:18,384 INFO] Device ID 0\n","[2023-10-14 15:57:18,384 INFO] Device cuda\n","[2023-10-14 15:57:18,526 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 15:57:18,526 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 15:57:19,674 INFO] Summarizer(\n","  (bert): Bert(\n","    (model): BertModel(\n","      (embeddings): BertEmbeddings(\n","        (word_embeddings): Embedding(30349, 768, padding_idx=0)\n","        (position_embeddings): Embedding(512, 768)\n","        (token_type_embeddings): Embedding(2, 768)\n","        (LayerNorm): BertLayerNorm()\n","        (dropout): Dropout(p=0.1, inplace=False)\n","      )\n","      (encoder): BertEncoder(\n","        (layer): ModuleList(\n","          (0-11): 12 x BertLayer(\n","            (attention): BertAttention(\n","              (self): BertSelfAttention(\n","                (query): Linear(in_features=768, out_features=768, bias=True)\n","                (key): Linear(in_features=768, out_features=768, bias=True)\n","                (value): Linear(in_features=768, out_features=768, bias=True)\n","                (dropout): Dropout(p=0.1, inplace=False)\n","              )\n","              (output): BertSelfOutput(\n","                (dense): Linear(in_features=768, out_features=768, bias=True)\n","                (LayerNorm): BertLayerNorm()\n","                (dropout): Dropout(p=0.1, inplace=False)\n","              )\n","            )\n","            (intermediate): BertIntermediate(\n","              (dense): Linear(in_features=768, out_features=3072, bias=True)\n","            )\n","            (output): BertOutput(\n","              (dense): Linear(in_features=3072, out_features=768, bias=True)\n","              (LayerNorm): BertLayerNorm()\n","              (dropout): Dropout(p=0.1, inplace=False)\n","            )\n","          )\n","        )\n","      )\n","      (pooler): BertPooler(\n","        (dense): Linear(in_features=768, out_features=768, bias=True)\n","        (activation): Tanh()\n","      )\n","    )\n","  )\n","  (encoder): TransformerInterEncoder(\n","    (pos_emb): PositionalEncoding(\n","      (dropout): Dropout(p=0.1, inplace=False)\n","    )\n","    (transformer_inter): ModuleList(\n","      (0-1): 2 x TransformerEncoderLayer(\n","        (self_attn): MultiHeadedAttention(\n","          (linear_keys): Linear(in_features=768, out_features=768, bias=True)\n","          (linear_values): Linear(in_features=768, out_features=768, bias=True)\n","          (linear_query): Linear(in_features=768, out_features=768, bias=True)\n","          (softmax): Softmax(dim=-1)\n","          (dropout): Dropout(p=0.1, inplace=False)\n","          (final_linear): Linear(in_features=768, out_features=768, bias=True)\n","        )\n","        (feed_forward): PositionwiseFeedForward(\n","          (w_1): Linear(in_features=768, out_features=512, bias=True)\n","          (w_2): Linear(in_features=512, out_features=768, bias=True)\n","          (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n","          (dropout_1): Dropout(p=0.1, inplace=False)\n","          (dropout_2): Dropout(p=0.1, inplace=False)\n","        )\n","        (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n","        (dropout): Dropout(p=0.1, inplace=False)\n","      )\n","    )\n","    (dropout): Dropout(p=0.1, inplace=False)\n","    (layer_norm): LayerNorm((768,), eps=1e-06, elementwise_affine=True)\n","    (wo): Linear(in_features=768, out_features=1, bias=True)\n","    (sigmoid): Sigmoid()\n","  )\n",")\n","gpu_rank 0\n","[2023-10-14 15:57:19,677 INFO] * number of parameters: 115657985\n","[2023-10-14 15:57:19,677 INFO] Start training...\n","[2023-10-14 15:57:19,677 INFO] Loading train dataset from /home/dongwon/korbertsum/bert_data/korean.train.0.bert.pt\n","[2023-10-14 15:57:20,058 INFO] Loaded train dataset from /home/dongwon/korbertsum/bert_data/korean.train.0.bert.pt, number of examples: 10001\n","[2023-10-14 15:58:26,511 INFO] Step 500/50000; xent: 5.58; lr: 0.0000014;  16 docs/s;     66 sec\n","[2023-10-14 15:59:33,335 INFO] Step 1000/50000; xent: 4.60; lr: 0.0000028;  16 docs/s;    133 sec\n","[2023-10-14 16:00:39,918 INFO] Step 1500/50000; xent: 4.53; lr: 0.0000042;  16 docs/s;    200 sec\n","[2023-10-14 16:01:44,813 INFO] Step 2000/50000; xent: 4.15; lr: 0.0000056;  16 docs/s;    265 sec\n","[2023-10-14 16:02:49,934 INFO] Step 2500/50000; xent: 3.94; lr: 0.0000070;  16 docs/s;    330 sec\n","[2023-10-14 16:03:56,419 INFO] Step 3000/50000; xent: 3.93; lr: 0.0000084;  16 docs/s;    396 sec\n","[2023-10-14 16:05:03,222 INFO] Step 3500/50000; xent: 3.83; lr: 0.0000098;  16 docs/s;    463 sec\n","[2023-10-14 16:06:09,281 INFO] Step 4000/50000; xent: 3.80; lr: 0.0000112;  16 docs/s;    529 sec\n","[2023-10-14 16:07:15,057 INFO] Step 4500/50000; xent: 3.77; lr: 0.0000126;  16 docs/s;    595 sec\n","[2023-10-14 16:07:48,213 INFO] Loading train dataset from /home/dongwon/korbertsum/bert_data/korean.train.2.bert.pt\n","[2023-10-14 16:07:50,030 INFO] Loaded train dataset from /home/dongwon/korbertsum/bert_data/korean.train.2.bert.pt, number of examples: 30014\n","[2023-10-14 16:08:18,176 INFO] Step 5000/50000; xent: 4.59; lr: 0.0000140;  12 docs/s;    658 sec\n","[2023-10-14 16:09:12,482 INFO] Step 5500/50000; xent: 5.71; lr: 0.0000154;   9 docs/s;    712 sec\n","[2023-10-14 16:10:06,286 INFO] Step 6000/50000; xent: 5.57; lr: 0.0000168;   9 docs/s;    766 sec\n","[2023-10-14 16:11:00,351 INFO] Step 6500/50000; xent: 5.65; lr: 0.0000182;   9 docs/s;    820 sec\n","[2023-10-14 16:11:54,484 INFO] Step 7000/50000; xent: 5.51; lr: 0.0000196;   9 docs/s;    874 sec\n","[2023-10-14 16:12:48,878 INFO] Step 7500/50000; xent: 5.54; lr: 0.0000210;   9 docs/s;    929 sec\n","[2023-10-14 16:13:43,352 INFO] Step 8000/50000; xent: 5.51; lr: 0.0000224;   9 docs/s;    983 sec\n","[2023-10-14 16:14:37,695 INFO] Step 8500/50000; xent: 5.56; lr: 0.0000217;   9 docs/s;   1038 sec\n","[2023-10-14 16:15:31,716 INFO] Step 9000/50000; xent: 5.62; lr: 0.0000211;   9 docs/s;   1092 sec\n","[2023-10-14 16:16:25,988 INFO] Step 9500/50000; xent: 5.68; lr: 0.0000205;   9 docs/s;   1146 sec\n","[2023-10-14 16:17:20,199 INFO] Step 10000/50000; xent: 5.50; lr: 0.0000200;   9 docs/s;   1200 sec\n","[2023-10-14 16:17:20,201 INFO] Saving checkpoint /home/dongwon/korbertsum/models/bert_classifier/model_step_10000.pt\n","[2023-10-14 16:18:15,425 INFO] Step 10500/50000; xent: 5.59; lr: 0.0000195;   9 docs/s;   1255 sec\n","[2023-10-14 16:19:09,182 INFO] Step 11000/50000; xent: 5.52; lr: 0.0000191;   9 docs/s;   1309 sec\n","[2023-10-14 16:20:03,123 INFO] Step 11500/50000; xent: 5.44; lr: 0.0000187;   9 docs/s;   1363 sec\n","[2023-10-14 16:20:57,510 INFO] Step 12000/50000; xent: 5.43; lr: 0.0000183;   9 docs/s;   1417 sec\n","[2023-10-14 16:21:51,479 INFO] Step 12500/50000; xent: 5.47; lr: 0.0000179;   9 docs/s;   1471 sec\n","[2023-10-14 16:22:45,664 INFO] Step 13000/50000; xent: 5.74; lr: 0.0000175;   9 docs/s;   1526 sec\n","[2023-10-14 16:23:40,110 INFO] Step 13500/50000; xent: 5.71; lr: 0.0000172;   9 docs/s;   1580 sec\n","[2023-10-14 16:24:35,396 INFO] Step 14000/50000; xent: 5.41; lr: 0.0000169;   9 docs/s;   1635 sec\n","[2023-10-14 16:25:31,161 INFO] Step 14500/50000; xent: 5.39; lr: 0.0000166;   9 docs/s;   1691 sec\n","[2023-10-14 16:26:27,269 INFO] Step 15000/50000; xent: 5.45; lr: 0.0000163;   9 docs/s;   1747 sec\n","[2023-10-14 16:27:23,478 INFO] Step 15500/50000; xent: 5.50; lr: 0.0000161;   9 docs/s;   1803 sec\n","[2023-10-14 16:28:19,523 INFO] Step 16000/50000; xent: 5.54; lr: 0.0000158;   9 docs/s;   1859 sec\n","[2023-10-14 16:29:15,101 INFO] Step 16500/50000; xent: 5.48; lr: 0.0000156;   9 docs/s;   1915 sec\n","[2023-10-14 16:30:11,081 INFO] Step 17000/50000; xent: 5.43; lr: 0.0000153;   9 docs/s;   1971 sec\n","[2023-10-14 16:31:06,671 INFO] Step 17500/50000; xent: 5.40; lr: 0.0000151;   9 docs/s;   2027 sec\n","[2023-10-14 16:32:02,484 INFO] Step 18000/50000; xent: 5.42; lr: 0.0000149;   9 docs/s;   2082 sec\n","[2023-10-14 16:32:58,592 INFO] Step 18500/50000; xent: 5.38; lr: 0.0000147;   9 docs/s;   2139 sec\n","[2023-10-14 16:33:53,639 INFO] Step 19000/50000; xent: 5.48; lr: 0.0000145;   9 docs/s;   2194 sec\n","[2023-10-14 16:34:47,399 INFO] Step 19500/50000; xent: 5.26; lr: 0.0000143;   9 docs/s;   2247 sec\n","[2023-10-14 16:35:41,795 INFO] Step 20000/50000; xent: 5.39; lr: 0.0000141;   9 docs/s;   2302 sec\n","[2023-10-14 16:35:41,796 INFO] Saving checkpoint /home/dongwon/korbertsum/models/bert_classifier/model_step_20000.pt\n","[2023-10-14 16:36:36,456 INFO] Step 20500/50000; xent: 5.44; lr: 0.0000140;   9 docs/s;   2356 sec\n","[2023-10-14 16:37:30,639 INFO] Step 21000/50000; xent: 5.32; lr: 0.0000138;   9 docs/s;   2411 sec\n","[2023-10-14 16:38:24,371 INFO] Step 21500/50000; xent: 5.43; lr: 0.0000136;   9 docs/s;   2464 sec\n","[2023-10-14 16:39:18,992 INFO] Step 22000/50000; xent: 5.29; lr: 0.0000135;   9 docs/s;   2519 sec\n","[2023-10-14 16:40:13,435 INFO] Step 22500/50000; xent: 5.34; lr: 0.0000133;   9 docs/s;   2573 sec\n","[2023-10-14 16:41:06,751 INFO] Step 23000/50000; xent: 5.35; lr: 0.0000132;   9 docs/s;   2627 sec\n","[2023-10-14 16:42:00,810 INFO] Step 23500/50000; xent: 5.26; lr: 0.0000130;   9 docs/s;   2681 sec\n","[2023-10-14 16:42:56,128 INFO] Step 24000/50000; xent: 5.43; lr: 0.0000129;   9 docs/s;   2736 sec\n","[2023-10-14 16:43:49,832 INFO] Step 24500/50000; xent: 6.29; lr: 0.0000128;   9 docs/s;   2790 sec\n","[2023-10-14 16:44:44,265 INFO] Step 25000/50000; xent: 6.24; lr: 0.0000126;   9 docs/s;   2844 sec\n","[2023-10-14 16:45:38,733 INFO] Step 25500/50000; xent: 5.42; lr: 0.0000125;   9 docs/s;   2899 sec\n","[2023-10-14 16:46:32,964 INFO] Step 26000/50000; xent: 5.45; lr: 0.0000124;   9 docs/s;   2953 sec\n","[2023-10-14 16:47:28,274 INFO] Step 26500/50000; xent: 5.33; lr: 0.0000123;   9 docs/s;   3008 sec\n","[2023-10-14 16:48:22,594 INFO] Step 27000/50000; xent: 5.53; lr: 0.0000122;   9 docs/s;   3063 sec\n","[2023-10-14 16:49:16,703 INFO] Step 27500/50000; xent: 5.29; lr: 0.0000121;   9 docs/s;   3117 sec\n","[2023-10-14 16:50:11,127 INFO] Step 28000/50000; xent: 5.29; lr: 0.0000120;   9 docs/s;   3171 sec\n","[2023-10-14 16:51:05,574 INFO] Step 28500/50000; xent: 5.28; lr: 0.0000118;   9 docs/s;   3226 sec\n","[2023-10-14 16:52:00,169 INFO] Step 29000/50000; xent: 5.51; lr: 0.0000117;   9 docs/s;   3280 sec\n","[2023-10-14 16:52:54,323 INFO] Step 29500/50000; xent: 5.36; lr: 0.0000116;   9 docs/s;   3334 sec\n","[2023-10-14 16:53:48,421 INFO] Step 30000/50000; xent: 5.30; lr: 0.0000115;   9 docs/s;   3388 sec\n","[2023-10-14 16:53:48,422 INFO] Saving checkpoint /home/dongwon/korbertsum/models/bert_classifier/model_step_30000.pt\n","[2023-10-14 16:54:43,663 INFO] Step 30500/50000; xent: 5.36; lr: 0.0000115;   9 docs/s;   3444 sec\n","[2023-10-14 16:55:38,539 INFO] Step 31000/50000; xent: 5.23; lr: 0.0000114;   9 docs/s;   3498 sec\n","[2023-10-14 16:56:32,930 INFO] Step 31500/50000; xent: 5.33; lr: 0.0000113;   9 docs/s;   3553 sec\n","[2023-10-14 16:57:26,745 INFO] Step 32000/50000; xent: 5.30; lr: 0.0000112;  10 docs/s;   3607 sec\n","[2023-10-14 16:58:21,971 INFO] Step 32500/50000; xent: 5.39; lr: 0.0000111;   9 docs/s;   3662 sec\n","[2023-10-14 16:59:16,085 INFO] Step 33000/50000; xent: 5.39; lr: 0.0000110;   9 docs/s;   3716 sec\n","[2023-10-14 17:00:11,425 INFO] Step 33500/50000; xent: 5.34; lr: 0.0000109;   9 docs/s;   3771 sec\n","[2023-10-14 17:01:05,921 INFO] Step 34000/50000; xent: 5.16; lr: 0.0000108;   9 docs/s;   3826 sec\n","[2023-10-14 17:01:27,830 INFO] Loading train dataset from /home/dongwon/korbertsum/bert_data/korean.train.1.bert.pt\n","[2023-10-14 17:01:31,664 INFO] Loaded train dataset from /home/dongwon/korbertsum/bert_data/korean.train.1.bert.pt, number of examples: 59997\n","[2023-10-14 17:02:08,399 INFO] Step 34500/50000; xent: 4.87; lr: 0.0000108;   9 docs/s;   3888 sec\n","[2023-10-14 17:03:09,601 INFO] Step 35000/50000; xent: 4.46; lr: 0.0000107;  11 docs/s;   3950 sec\n","[2023-10-14 17:04:10,611 INFO] Step 35500/50000; xent: 4.54; lr: 0.0000106;  10 docs/s;   4011 sec\n","[2023-10-14 17:05:12,395 INFO] Step 36000/50000; xent: 4.36; lr: 0.0000105;  11 docs/s;   4072 sec\n","[2023-10-14 17:06:12,602 INFO] Step 36500/50000; xent: 4.30; lr: 0.0000105;  11 docs/s;   4133 sec\n","[2023-10-14 17:07:12,899 INFO] Step 37000/50000; xent: 4.57; lr: 0.0000104;  11 docs/s;   4193 sec\n","[2023-10-14 17:08:13,038 INFO] Step 37500/50000; xent: 4.33; lr: 0.0000103;  11 docs/s;   4253 sec\n","[2023-10-14 17:09:14,314 INFO] Step 38000/50000; xent: 4.46; lr: 0.0000103;  11 docs/s;   4314 sec\n","[2023-10-14 17:10:14,994 INFO] Step 38500/50000; xent: 4.42; lr: 0.0000102;  10 docs/s;   4375 sec\n","[2023-10-14 17:11:17,162 INFO] Step 39000/50000; xent: 4.36; lr: 0.0000101;  11 docs/s;   4437 sec\n","[2023-10-14 17:12:19,415 INFO] Step 39500/50000; xent: 4.28; lr: 0.0000101;  11 docs/s;   4499 sec\n","[2023-10-14 17:13:21,240 INFO] Step 40000/50000; xent: 4.36; lr: 0.0000100;  11 docs/s;   4561 sec\n","[2023-10-14 17:13:21,241 INFO] Saving checkpoint /home/dongwon/korbertsum/models/bert_classifier/model_step_40000.pt\n","[2023-10-14 17:14:23,381 INFO] Step 40500/50000; xent: 4.29; lr: 0.0000099;  10 docs/s;   4623 sec\n","[2023-10-14 17:15:24,114 INFO] Step 41000/50000; xent: 4.39; lr: 0.0000099;  10 docs/s;   4684 sec\n","[2023-10-14 17:16:25,776 INFO] Step 41500/50000; xent: 4.31; lr: 0.0000098;  11 docs/s;   4746 sec\n","[2023-10-14 17:17:26,046 INFO] Step 42000/50000; xent: 4.35; lr: 0.0000098;  11 docs/s;   4806 sec\n","[2023-10-14 17:18:27,188 INFO] Step 42500/50000; xent: 4.40; lr: 0.0000097;  10 docs/s;   4867 sec\n","[2023-10-14 17:19:27,985 INFO] Step 43000/50000; xent: 4.36; lr: 0.0000096;  11 docs/s;   4928 sec\n","[2023-10-14 17:20:28,604 INFO] Step 43500/50000; xent: 4.27; lr: 0.0000096;  11 docs/s;   4989 sec\n","[2023-10-14 17:21:28,231 INFO] Step 44000/50000; xent: 4.32; lr: 0.0000095;  11 docs/s;   5048 sec\n","[2023-10-14 17:22:28,131 INFO] Step 44500/50000; xent: 4.24; lr: 0.0000095;  11 docs/s;   5108 sec\n","[2023-10-14 17:23:30,038 INFO] Step 45000/50000; xent: 4.37; lr: 0.0000094;  10 docs/s;   5170 sec\n","[2023-10-14 17:24:32,128 INFO] Step 45500/50000; xent: 4.25; lr: 0.0000094;  10 docs/s;   5232 sec\n","[2023-10-14 17:25:32,985 INFO] Step 46000/50000; xent: 4.40; lr: 0.0000093;  10 docs/s;   5293 sec\n","[2023-10-14 17:26:33,908 INFO] Step 46500/50000; xent: 4.39; lr: 0.0000093;  10 docs/s;   5354 sec\n","[2023-10-14 17:27:36,698 INFO] Step 47000/50000; xent: 4.20; lr: 0.0000092;  11 docs/s;   5417 sec\n","[2023-10-14 17:28:38,750 INFO] Step 47500/50000; xent: 4.20; lr: 0.0000092;  10 docs/s;   5479 sec\n","[2023-10-14 17:29:40,161 INFO] Step 48000/50000; xent: 4.16; lr: 0.0000091;  10 docs/s;   5540 sec\n","[2023-10-14 17:30:42,439 INFO] Step 48500/50000; xent: 4.24; lr: 0.0000091;  10 docs/s;   5602 sec\n","[2023-10-14 17:31:44,789 INFO] Step 49000/50000; xent: 4.26; lr: 0.0000090;  10 docs/s;   5665 sec\n","[2023-10-14 17:32:46,491 INFO] Step 49500/50000; xent: 4.27; lr: 0.0000090;  10 docs/s;   5726 sec\n","[2023-10-14 17:33:46,809 INFO] Step 50000/50000; xent: 4.32; lr: 0.0000089;  10 docs/s;   5787 sec\n","[2023-10-14 17:33:46,810 INFO] Saving checkpoint /home/dongwon/korbertsum/models/bert_classifier/model_step_50000.pt\n","[2023-10-14 17:33:47,603 INFO] Loading train dataset from /home/dongwon/korbertsum/bert_data/korean.train.0.bert.pt\n","[2023-10-14 17:33:47,964 INFO] Loaded train dataset from /home/dongwon/korbertsum/bert_data/korean.train.0.bert.pt, number of examples: 10001\n"]}],"source":["!python train.py -mode train -encoder transformer -dropout 0.1 -bert_data_path /home/dongwon/korbertsum/bert_data/korean -model_path /home/dongwon/korbertsum/models/bert_classifier -lr 2e-3 -visible_gpus 0 -gpu_ranks 0 -world_size 1 -report_every 500 -save_checkpoint_steps 10000 -batch_size 1000 -decay_method noam -train_steps 50000 -accum_count 1 -log_file /home/dongwon/korbertsum/logs/bert_classifier -use_interval true -warmup_steps 8000 -bert_model /home/dongwon/001_bert_morp_pytorch -bert_config_path /home/dongwon/001_bert_morp_pytorch/bert_config.json -temp_dir ."]},{"cell_type":"code","execution_count":9,"metadata":{"id":"uW5ogvPn-FvO"},"outputs":[{"name":"stdout","output_type":"stream","text":["[2023-10-14 17:39:33,152 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_10000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:39:33,436 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:39:33,436 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:39:34,767 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt\n","[2023-10-14 17:39:34,783 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt, number of examples: 498\n","gpu_rank 0\n","[2023-10-14 17:39:34,785 INFO] * number of parameters: 115657985\n","[2023-10-14 17:39:42,780 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt\n","[2023-10-14 17:39:42,931 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt, number of examples: 2999\n","[2023-10-14 17:40:56,693 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt\n","[2023-10-14 17:40:56,789 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt, number of examples: 1499\n","[2023-10-14 17:41:36,987 INFO] Validation xent: 5.39072 at step 10000\n","[2023-10-14 17:41:37,003 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_20000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:41:37,205 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:41:37,205 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:41:37,916 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt\n","[2023-10-14 17:41:37,932 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt, number of examples: 498\n","gpu_rank 0\n","[2023-10-14 17:41:37,933 INFO] * number of parameters: 115657985\n","[2023-10-14 17:41:45,807 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt\n","[2023-10-14 17:41:45,953 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt, number of examples: 2999\n","[2023-10-14 17:42:58,605 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt\n","[2023-10-14 17:42:58,689 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt, number of examples: 1499\n","[2023-10-14 17:43:38,495 INFO] Validation xent: 5.36066 at step 20000\n","[2023-10-14 17:43:38,497 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_30000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:43:38,612 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:43:38,612 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:43:39,296 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt\n","[2023-10-14 17:43:39,314 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt, number of examples: 498\n","gpu_rank 0\n","[2023-10-14 17:43:39,315 INFO] * number of parameters: 115657985\n","[2023-10-14 17:43:47,267 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt\n","[2023-10-14 17:43:47,417 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt, number of examples: 2999\n","[2023-10-14 17:45:01,212 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt\n","[2023-10-14 17:45:01,296 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt, number of examples: 1499\n","[2023-10-14 17:45:40,378 INFO] Validation xent: 5.13827 at step 30000\n","[2023-10-14 17:45:40,379 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_40000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:45:40,495 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:45:40,495 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:45:41,144 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt\n","[2023-10-14 17:45:41,161 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt, number of examples: 498\n","gpu_rank 0\n","[2023-10-14 17:45:41,162 INFO] * number of parameters: 115657985\n","[2023-10-14 17:45:48,959 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt\n","[2023-10-14 17:45:49,107 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt, number of examples: 2999\n","[2023-10-14 17:47:01,773 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt\n","[2023-10-14 17:47:01,857 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt, number of examples: 1499\n","[2023-10-14 17:47:41,275 INFO] Validation xent: 4.84305 at step 40000\n","[2023-10-14 17:47:41,277 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_50000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:47:41,397 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:47:41,397 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:47:42,052 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt\n","[2023-10-14 17:47:42,069 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.0.bert.pt, number of examples: 498\n","gpu_rank 0\n","[2023-10-14 17:47:42,070 INFO] * number of parameters: 115657985\n","[2023-10-14 17:47:50,044 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt\n","[2023-10-14 17:47:50,189 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.1.bert.pt, number of examples: 2999\n","[2023-10-14 17:49:03,080 INFO] Loading valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt\n","[2023-10-14 17:49:03,171 INFO] Loaded valid dataset from /home/dongwon/korbertsum/bert_data/korean.valid.2.bert.pt, number of examples: 1499\n","[2023-10-14 17:49:42,068 INFO] Validation xent: 4.84563 at step 50000\n","[2023-10-14 17:49:42,070 INFO] PPL [(4.843047584005124, '/home/dongwon/korbertsum/models/bert_classifier/model_step_40000.pt'), (4.845631812134679, '/home/dongwon/korbertsum/models/bert_classifier/model_step_50000.pt'), (5.138273820173654, '/home/dongwon/korbertsum/models/bert_classifier/model_step_30000.pt')]\n","[2023-10-14 17:49:42,070 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_40000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:49:42,202 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:49:42,202 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:49:42,853 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt\n","[2023-10-14 17:49:42,882 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt, number of examples: 500\n","gpu_rank 0\n","[2023-10-14 17:49:42,883 INFO] * number of parameters: 115657985\n","[2023-10-14 17:49:54,065 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt\n","[2023-10-14 17:49:54,217 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt, number of examples: 2995\n","[2023-10-14 17:51:07,096 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt\n","[2023-10-14 17:51:07,185 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt, number of examples: 1500\n","4995\n","4995\n","2023-10-14 17:51:47,083 [MainThread  ] [INFO ]  Writing summaries.\n","[2023-10-14 17:51:47,083 INFO] Writing summaries.\n","2023-10-14 17:51:47,084 [MainThread  ] [INFO ]  Processing summaries. Saving system files to ../temp/tmpt6uc040y/system and model files to ../temp/tmpt6uc040y/model.\n","[2023-10-14 17:51:47,084 INFO] Processing summaries. Saving system files to ../temp/tmpt6uc040y/system and model files to ../temp/tmpt6uc040y/model.\n","2023-10-14 17:51:47,084 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-51-46/candidate/.\n","[2023-10-14 17:51:47,084 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-51-46/candidate/.\n","2023-10-14 17:51:47,233 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpt6uc040y/system.\n","[2023-10-14 17:51:47,233 INFO] Saved processed files to ../temp/tmpt6uc040y/system.\n","2023-10-14 17:51:47,233 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-51-46/reference/.\n","[2023-10-14 17:51:47,233 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-51-46/reference/.\n","2023-10-14 17:51:47,382 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpt6uc040y/model.\n","[2023-10-14 17:51:47,382 INFO] Saved processed files to ../temp/tmpt6uc040y/model.\n","2023-10-14 17:51:47,394 [MainThread  ] [INFO ]  Written ROUGE configuration to ../temp/tmpr6ydbd5u/rouge_conf.xml\n","[2023-10-14 17:51:47,394 INFO] Written ROUGE configuration to ../temp/tmpr6ydbd5u/rouge_conf.xml\n","2023-10-14 17:51:47,394 [MainThread  ] [INFO ]  Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpr6ydbd5u/rouge_conf.xml\n","[2023-10-14 17:51:47,394 INFO] Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpr6ydbd5u/rouge_conf.xml\n","---------------------------------------------\n","1 ROUGE-1 Average_R: 0.77945 (95%-conf.int. 0.77537 - 0.78339)\n","1 ROUGE-1 Average_P: 0.82552 (95%-conf.int. 0.82209 - 0.82882)\n","1 ROUGE-1 Average_F: 0.78652 (95%-conf.int. 0.78376 - 0.78913)\n","---------------------------------------------\n","1 ROUGE-2 Average_R: 0.64476 (95%-conf.int. 0.64014 - 0.64906)\n","1 ROUGE-2 Average_P: 0.68565 (95%-conf.int. 0.68102 - 0.69008)\n","1 ROUGE-2 Average_F: 0.65168 (95%-conf.int. 0.64752 - 0.65537)\n","---------------------------------------------\n","1 ROUGE-L Average_R: 0.74277 (95%-conf.int. 0.73871 - 0.74688)\n","1 ROUGE-L Average_P: 0.78733 (95%-conf.int. 0.78367 - 0.79098)\n","1 ROUGE-L Average_F: 0.74984 (95%-conf.int. 0.74672 - 0.75296)\n","\n","[2023-10-14 17:52:21,777 INFO] Rouges at step 40000 \n",">> ROUGE-F(1/2/3/l): 78.65/65.17/74.98\n","ROUGE-R(1/2/3/l): 77.94/64.48/74.28\n","\n","[2023-10-14 17:52:21,777 INFO] Validation xent: 4.98864 at step 40000\n","[2023-10-14 17:52:21,779 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_50000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:52:21,913 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:52:21,913 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:52:22,877 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt\n","[2023-10-14 17:52:22,899 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt, number of examples: 500\n","gpu_rank 0\n","[2023-10-14 17:52:22,900 INFO] * number of parameters: 115657985\n","[2023-10-14 17:52:33,916 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt\n","[2023-10-14 17:52:34,070 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt, number of examples: 2995\n","[2023-10-14 17:53:45,781 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt\n","[2023-10-14 17:53:45,866 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt, number of examples: 1500\n","4995\n","4995\n","2023-10-14 17:54:24,526 [MainThread  ] [INFO ]  Writing summaries.\n","[2023-10-14 17:54:24,526 INFO] Writing summaries.\n","2023-10-14 17:54:24,527 [MainThread  ] [INFO ]  Processing summaries. Saving system files to ../temp/tmpo4267bn3/system and model files to ../temp/tmpo4267bn3/model.\n","[2023-10-14 17:54:24,527 INFO] Processing summaries. Saving system files to ../temp/tmpo4267bn3/system and model files to ../temp/tmpo4267bn3/model.\n","2023-10-14 17:54:24,527 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-54-24/candidate/.\n","[2023-10-14 17:54:24,527 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-54-24/candidate/.\n","2023-10-14 17:54:24,678 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpo4267bn3/system.\n","[2023-10-14 17:54:24,678 INFO] Saved processed files to ../temp/tmpo4267bn3/system.\n","2023-10-14 17:54:24,678 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-54-24/reference/.\n","[2023-10-14 17:54:24,678 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-54-24/reference/.\n","2023-10-14 17:54:24,829 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpo4267bn3/model.\n","[2023-10-14 17:54:24,829 INFO] Saved processed files to ../temp/tmpo4267bn3/model.\n","2023-10-14 17:54:24,841 [MainThread  ] [INFO ]  Written ROUGE configuration to ../temp/tmpgq2f_dss/rouge_conf.xml\n","[2023-10-14 17:54:24,841 INFO] Written ROUGE configuration to ../temp/tmpgq2f_dss/rouge_conf.xml\n","2023-10-14 17:54:24,841 [MainThread  ] [INFO ]  Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpgq2f_dss/rouge_conf.xml\n","[2023-10-14 17:54:24,841 INFO] Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmpgq2f_dss/rouge_conf.xml\n","---------------------------------------------\n","1 ROUGE-1 Average_R: 0.78720 (95%-conf.int. 0.78303 - 0.79118)\n","1 ROUGE-1 Average_P: 0.82138 (95%-conf.int. 0.81793 - 0.82499)\n","1 ROUGE-1 Average_F: 0.78817 (95%-conf.int. 0.78508 - 0.79093)\n","---------------------------------------------\n","1 ROUGE-2 Average_R: 0.65296 (95%-conf.int. 0.64801 - 0.65754)\n","1 ROUGE-2 Average_P: 0.68485 (95%-conf.int. 0.68017 - 0.68978)\n","1 ROUGE-2 Average_F: 0.65522 (95%-conf.int. 0.65107 - 0.65922)\n","---------------------------------------------\n","1 ROUGE-L Average_R: 0.75072 (95%-conf.int. 0.74642 - 0.75503)\n","1 ROUGE-L Average_P: 0.78424 (95%-conf.int. 0.78049 - 0.78816)\n","1 ROUGE-L Average_F: 0.75212 (95%-conf.int. 0.74868 - 0.75527)\n","\n","[2023-10-14 17:55:02,140 INFO] Rouges at step 50000 \n",">> ROUGE-F(1/2/3/l): 78.82/65.52/75.21\n","ROUGE-R(1/2/3/l): 78.72/65.30/75.07\n","\n","[2023-10-14 17:55:02,140 INFO] Validation xent: 4.99079 at step 50000\n","[2023-10-14 17:55:02,142 INFO] Loading checkpoint from /home/dongwon/korbertsum/models/bert_classifier/model_step_30000.pt\n","Namespace(encoder='transformer', mode='validate', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=True, test_from='', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 17:55:02,308 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 17:55:02,308 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 17:55:02,982 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt\n","[2023-10-14 17:55:03,004 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt, number of examples: 500\n","gpu_rank 0\n","[2023-10-14 17:55:03,005 INFO] * number of parameters: 115657985\n","[2023-10-14 17:55:13,952 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt\n","[2023-10-14 17:55:14,098 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt, number of examples: 2995\n","[2023-10-14 17:56:25,590 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt\n","[2023-10-14 17:56:25,672 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt, number of examples: 1500\n","4995\n","4995\n","2023-10-14 17:57:04,119 [MainThread  ] [INFO ]  Writing summaries.\n","[2023-10-14 17:57:04,119 INFO] Writing summaries.\n","2023-10-14 17:57:04,120 [MainThread  ] [INFO ]  Processing summaries. Saving system files to ../temp/tmpg5jq17g8/system and model files to ../temp/tmpg5jq17g8/model.\n","[2023-10-14 17:57:04,120 INFO] Processing summaries. Saving system files to ../temp/tmpg5jq17g8/system and model files to ../temp/tmpg5jq17g8/model.\n","2023-10-14 17:57:04,120 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-57-03/candidate/.\n","[2023-10-14 17:57:04,120 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-57-03/candidate/.\n","2023-10-14 17:57:04,277 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpg5jq17g8/system.\n","[2023-10-14 17:57:04,277 INFO] Saved processed files to ../temp/tmpg5jq17g8/system.\n","2023-10-14 17:57:04,277 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-17-57-03/reference/.\n","[2023-10-14 17:57:04,277 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-17-57-03/reference/.\n","2023-10-14 17:57:04,432 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpg5jq17g8/model.\n","[2023-10-14 17:57:04,432 INFO] Saved processed files to ../temp/tmpg5jq17g8/model.\n","2023-10-14 17:57:04,443 [MainThread  ] [INFO ]  Written ROUGE configuration to ../temp/tmp936hyaqm/rouge_conf.xml\n","[2023-10-14 17:57:04,443 INFO] Written ROUGE configuration to ../temp/tmp936hyaqm/rouge_conf.xml\n","2023-10-14 17:57:04,443 [MainThread  ] [INFO ]  Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmp936hyaqm/rouge_conf.xml\n","[2023-10-14 17:57:04,443 INFO] Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmp936hyaqm/rouge_conf.xml\n","---------------------------------------------\n","1 ROUGE-1 Average_R: 0.79124 (95%-conf.int. 0.78732 - 0.79496)\n","1 ROUGE-1 Average_P: 0.80954 (95%-conf.int. 0.80575 - 0.81304)\n","1 ROUGE-1 Average_F: 0.78394 (95%-conf.int. 0.78119 - 0.78664)\n","---------------------------------------------\n","1 ROUGE-2 Average_R: 0.65073 (95%-conf.int. 0.64630 - 0.65509)\n","1 ROUGE-2 Average_P: 0.66906 (95%-conf.int. 0.66443 - 0.67363)\n","1 ROUGE-2 Average_F: 0.64598 (95%-conf.int. 0.64198 - 0.64964)\n","---------------------------------------------\n","1 ROUGE-L Average_R: 0.75196 (95%-conf.int. 0.74804 - 0.75577)\n","1 ROUGE-L Average_P: 0.77013 (95%-conf.int. 0.76609 - 0.77404)\n","1 ROUGE-L Average_F: 0.74540 (95%-conf.int. 0.74226 - 0.74832)\n","\n","[2023-10-14 17:57:39,999 INFO] Rouges at step 30000 \n",">> ROUGE-F(1/2/3/l): 78.39/64.60/74.54\n","ROUGE-R(1/2/3/l): 79.12/65.07/75.20\n","\n","[2023-10-14 17:57:39,999 INFO] Validation xent: 5.31951 at step 30000\n"]}],"source":["!python train.py -mode validate -bert_data_path /home/dongwon/korbertsum/bert_data/korean -model_path /home/dongwon/korbertsum/models/bert_classifier  -visible_gpus 0  -gpu_ranks 0 -batch_size 1000  -log_file /home/dongwon/korbertsum/logs/bert_classifier  -result_path /home/dongwon/korbertsum/results/korean -bert_model /home/dongwon/001_bert_morp_pytorch -test_all"]},{"cell_type":"code","execution_count":12,"metadata":{"id":"WoaKzondr8jd"},"outputs":[{"name":"stdout","output_type":"stream","text":["[2023-10-14 18:05:40,909 INFO] Loading checkpoint from ../models/bert_classifier/model_step_40000.pt\n","Namespace(encoder='transformer', mode='test', bert_data_path='/home/dongwon/korbertsum/bert_data/korean', model_path='/home/dongwon/korbertsum/models/bert_classifier', result_path='/home/dongwon/korbertsum/results/korean', temp_dir='../temp', bert_config_path='../bert_config_uncased_base.json', batch_size=1000, use_interval=True, hidden_size=128, ff_size=512, heads=4, inter_layers=2, rnn_size=512, param_init=0, param_init_glorot=True, dropout=0.1, optim='adam', lr=1, beta1=0.9, beta2=0.999, decay_method='', warmup_steps=8000, max_grad_norm=0, save_checkpoint_steps=5, accum_count=1, world_size=1, report_every=1, train_steps=1000, recall_eval=False, visible_gpus='0', gpu_ranks=[0], log_file='/home/dongwon/korbertsum/logs/bert_classifier', dataset='', seed=666, bert_model='/home/dongwon/001_bert_morp_pytorch', test_all=False, test_from='../models/bert_classifier/model_step_40000.pt', train_from='', report_rouge=True, block_trigram=True)\n","[2023-10-14 18:05:41,155 INFO] loading archive file /home/dongwon/001_bert_morp_pytorch\n","[2023-10-14 18:05:41,155 INFO] Model config {\n","  \"attention_probs_dropout_prob\": 0.1,\n","  \"directionality\": \"bidi\",\n","  \"hidden_act\": \"gelu\",\n","  \"hidden_dropout_prob\": 0.1,\n","  \"hidden_size\": 768,\n","  \"initializer_range\": 0.02,\n","  \"intermediate_size\": 3072,\n","  \"max_position_embeddings\": 512,\n","  \"num_attention_heads\": 12,\n","  \"num_hidden_layers\": 12,\n","  \"pooler_fc_size\": 768,\n","  \"pooler_num_attention_heads\": 12,\n","  \"pooler_num_fc_layers\": 3,\n","  \"pooler_size_per_head\": 128,\n","  \"pooler_type\": \"first_token_transform\",\n","  \"type_vocab_size\": 2,\n","  \"vocab_size\": 30349\n","}\n","\n","[2023-10-14 18:05:42,465 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt\n","[2023-10-14 18:05:42,488 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.0.bert.pt, number of examples: 500\n","gpu_rank 0\n","[2023-10-14 18:05:42,490 INFO] * number of parameters: 115657985\n","[2023-10-14 18:05:53,782 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt\n","[2023-10-14 18:05:53,934 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.1.bert.pt, number of examples: 2995\n","[2023-10-14 18:07:05,733 INFO] Loading test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt\n","[2023-10-14 18:07:05,824 INFO] Loaded test dataset from /home/dongwon/korbertsum/bert_data/korean.test.2.bert.pt, number of examples: 1500\n","4995\n","4995\n","2023-10-14 18:07:45,424 [MainThread  ] [INFO ]  Writing summaries.\n","[2023-10-14 18:07:45,424 INFO] Writing summaries.\n","2023-10-14 18:07:45,425 [MainThread  ] [INFO ]  Processing summaries. Saving system files to ../temp/tmpbh_5jsn8/system and model files to ../temp/tmpbh_5jsn8/model.\n","[2023-10-14 18:07:45,425 INFO] Processing summaries. Saving system files to ../temp/tmpbh_5jsn8/system and model files to ../temp/tmpbh_5jsn8/model.\n","2023-10-14 18:07:45,425 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-18-07-45/candidate/.\n","[2023-10-14 18:07:45,425 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-18-07-45/candidate/.\n","2023-10-14 18:07:45,633 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpbh_5jsn8/system.\n","[2023-10-14 18:07:45,633 INFO] Saved processed files to ../temp/tmpbh_5jsn8/system.\n","2023-10-14 18:07:45,633 [MainThread  ] [INFO ]  Processing files in ../temp/rouge-tmp-2023-10-14-18-07-45/reference/.\n","[2023-10-14 18:07:45,633 INFO] Processing files in ../temp/rouge-tmp-2023-10-14-18-07-45/reference/.\n","2023-10-14 18:07:45,851 [MainThread  ] [INFO ]  Saved processed files to ../temp/tmpbh_5jsn8/model.\n","[2023-10-14 18:07:45,851 INFO] Saved processed files to ../temp/tmpbh_5jsn8/model.\n","2023-10-14 18:07:45,869 [MainThread  ] [INFO ]  Written ROUGE configuration to ../temp/tmprv7_iyov/rouge_conf.xml\n","[2023-10-14 18:07:45,869 INFO] Written ROUGE configuration to ../temp/tmprv7_iyov/rouge_conf.xml\n","2023-10-14 18:07:45,869 [MainThread  ] [INFO ]  Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmprv7_iyov/rouge_conf.xml\n","[2023-10-14 18:07:45,869 INFO] Running ROUGE with command /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/dongwon/rouge/pyrouge/tools/ROUGE-1.5.5/data -c 95 -m -r 1000 -n 2 -a ../temp/tmprv7_iyov/rouge_conf.xml\n","---------------------------------------------\n","1 ROUGE-1 Average_R: 0.77945 (95%-conf.int. 0.77537 - 0.78339)\n","1 ROUGE-1 Average_P: 0.82552 (95%-conf.int. 0.82209 - 0.82882)\n","1 ROUGE-1 Average_F: 0.78652 (95%-conf.int. 0.78376 - 0.78913)\n","---------------------------------------------\n","1 ROUGE-2 Average_R: 0.64476 (95%-conf.int. 0.64014 - 0.64906)\n","1 ROUGE-2 Average_P: 0.68565 (95%-conf.int. 0.68102 - 0.69008)\n","1 ROUGE-2 Average_F: 0.65168 (95%-conf.int. 0.64752 - 0.65537)\n","---------------------------------------------\n","1 ROUGE-L Average_R: 0.74277 (95%-conf.int. 0.73871 - 0.74688)\n","1 ROUGE-L Average_P: 0.78733 (95%-conf.int. 0.78367 - 0.79098)\n","1 ROUGE-L Average_F: 0.74984 (95%-conf.int. 0.74672 - 0.75296)\n","\n","[2023-10-14 18:08:21,604 INFO] Rouges at step 40000 \n",">> ROUGE-F(1/2/3/l): 78.65/65.17/74.98\n","ROUGE-R(1/2/3/l): 77.94/64.48/74.28\n","\n","[2023-10-14 18:08:21,604 INFO] Validation xent: 4.98864 at step 40000\n"]}],"source":["!python train.py -mode test -bert_data_path /home/dongwon/korbertsum/bert_data/korean -model_path /home/dongwon/korbertsum/models/bert_classifier  -visible_gpus 0  -gpu_ranks 0 -batch_size 1000  -log_file /home/dongwon/korbertsum/logs/bert_classifier  -result_path /home/dongwon/korbertsum/results/korean -bert_model /home/dongwon/001_bert_morp_pytorch -test_from ../models/bert_classifier/model_step_40000.pt"]},{"cell_type":"markdown","metadata":{"id":"rIZruOeCsL_5"},"source":["# Test"]},{"cell_type":"code","execution_count":10,"metadata":{},"outputs":[{"name":"stdout","output_type":"stream","text":["<_io.TextIOWrapper name='/home/dongwon/korbertsum/results/korean_step1000.gold' mode='r' encoding='UTF-8'>\n"]}],"source":["with open('/home/dongwon/korbertsum/results/korean_step40000.gold','r') as f:\n","    gold = f\n","    print(gold)"]},{"cell_type":"code","execution_count":12,"metadata":{"id":"Fr5Ht2L0pV5J"},"outputs":[],"source":["with open('/home/dongwon/korbertsum/results/korean_step40000.gold','r') as f:\n","    gold = f.readlines()"]},{"cell_type":"code","execution_count":6,"metadata":{"id":"0ff8k_UYpupz"},"outputs":[],"source":["with open('/home/dongwon/korbertsum/results/korean_step40000.candidate','r') as f:\n","    candidate = f.readlines()"]},{"cell_type":"code","execution_count":13,"metadata":{"id":"ieg_5YynplZd"},"outputs":[{"data":{"text/plain":["'인천/NNP 은/JX 산둥성/NNP 을/JKO 비롯하/VV 어/EC 산둥성/NNP 주요/NNG 도시/NNG 이/VCP ㄴ/ETM 칭다오/NNP (/SS 靑島/SH )/SS ·/SP 옌타이/NNP (/SS 煙臺/SH )/SS 시/NNG 와/JC 우호/NNG 도시/NNG 관계/NNG 이/VCP 다/EF<q>또/MAG 미래/NNG 인/NNG 적/XSN 자산/NNG 으로서/JKB 청/NNG 소년/NNG 및/MAJ 학생/NNG 들/XSN 의/JKG 교류/NNG 활성/NNG 화/XSN 필요/NNG 성/XSN 도/JX 강조/NNG 하/XSV 었/EP 다/EF<q>박남춘/NNP 시장/NNG 은/JX \"/SS 두/MM 지역/NNG 의/JKG 상생/NNG 발전/NNG 을/JKO 위하/VV ㄴ/ETM 기술/NNG 협력/NNG ,/SP 상호/NNG 인증/NNG ,/SP 문화/NNG ·/SP 관광/NNG 융합/NNG 콘텐츠/NNG 개발/NNG 및/MAJ 한중/NNP FTA/SL 지방/NNG 경제/NNG 협력/NNG 시범/NNG 사업/NNG 의/JKG 성과/NNG 를/JKO 발전/NNG 시키/XSV 어/EC 두/MM 나라/NNG 간/NNB FTA/SL 선도/NNG 지역/NNG 의/JKG 역할/NNG 을/JKO 지속/NNG 적/XSN 으로/JKB 실천/NNG 하/XSV ㄹ/ETM 필요/NNG 가/JKS 있/VA 다/EF \"/SS 며/EC \"/SS 미래/NNG 지향/NNG 적/XSN 가치/NNG 창출/NNG 을/JKO 위하/VV 어/EC 공동/NNG 과제/NNG 를/JKO 발굴/NNG 하/XSV 고/EC 추진/NNG 하/XSV 어/EC 나가/VX 자/EF \"/SS 고/JKQ 말/NNG 하/XSV 었/EP 다/EF\\n'"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["# 라벨 데이터\n","gold[3]"]},{"cell_type":"code","execution_count":8,"metadata":{"id":"V4JZJNYXpz2X"},"outputs":[{"data":{"text/plain":["'12/SN 일/NNB 시/NNG 에/JKB 따르/VV 면/EC 박남춘/NNP 인천/NNP 시장/NNG 은/JX 이날/NNG 산둥성/NNP 을/JKO 방문/NNG 하/XSV 어/EC 지난/NNG (/SS 濟南/SH )/SS 시/NNG 왕중린/NNP (/SS 王忠林/SH )/SS 서기/NNG 와/JC 산둥성/NNP 류자이/NNP (/SS 劉家義/SH )/SS 서기/NNG 를/JKO 차례/NNG 로/JKB 접견/NNG 하/XSV 며/EC 상호/NNG 협력/NNG 발전/NNG 방안/NNG 을/JKO 논의/NNG 하/XSV 었/EP 다/EF<q>박/NNP 시장/NNG ,/SP 류자/NNP 이/JKS 서기/NNG 와/JC 접견/NNG 공동/NNG 포럼/NNG ·/SP 청/NNG 소년/NNG 교류/NNG 제안/NNG 인천시/NNP 가/JKS 중국/NNP 산둥/NNP (/SS 山東/SH )/SS 성/NNG 과/JC 우호/NNG 관계/NNG 를/JKO 넘/VV 어/EC ‘/SS 친구/NNG 도시/NNG ’/SS 로/JKB 나아가/VV 는/ETM 첫/MM 발/NNG 을/JKO 내디디/VV 었/EP 다/EF<q>인천/NNP 은/JX 산둥성/NNP 을/JKO 비롯하/VV 어/EC 산둥성/NNP 주요/NNG 도시/NNG 이/VCP ㄴ/ETM 칭다오/NNP (/SS 靑島/SH )/SS ·/SP 옌타이/NNP (/SS 煙臺/SH )/SS 시/NNG 와/JC 우호/NNG 도시/NNG 관계/NNG 이/VCP 다/EF\\n'"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["# 예측 데이터\n","candidate[3]"]}],"metadata":{"accelerator":"GPU","colab":{"authorship_tag":"ABX9TyOHi3fjXHKga4urnmSF7c68","collapsed_sections":[],"machine_shape":"hm","name":"Newsdata_extractive_summ.ipynb","private_outputs":true,"provenance":[]},"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.4"}},"nbformat":4,"nbformat_minor":0}