training_special_tokens skip loss

[Hanzhang-lang]

self-rag/data_creation/train_special_tokens.py

Lines 213 to 224 in 1fcdc42

	if context_markups is not None:
	for i, (label_id_list, source_len) in enumerate(zip(labels, sources_tokenized["input_ids_lens"])):
	context_start = False
	for j, orig_token in enumerate(label_id_list[source_len:]):
	if context_start is False and orig_token == context_markups[0]:
	context_start = True
	start_idx = j+source_len
	for k, orig_token_2 in enumerate(label_id_list[start_idx:]):
	if orig_token_2 == context_markups[1]:
	end_idx = start_idx + k
	labels[i][start_idx+1:end_idx] = IGNORE_INDEX
	context_start = False

If multiple paragraph references(<paragraph>) appear here, will all the content in between be set to IGNORE_INDEX? Is this incorrect?