Development test suites and reports

This page contains a log of the (most relevant) tests of the EOP on both RTE and industrial data, performed with version 1.1.3 of the EOP release. Please add the summary of your test: dataset used, configuration details (LAP and resources used) and the results (accuracy), as well as a link to a directory with the configuration file used, the model and the generated results file.

---

• BIUTEE
• EditDistance
• TIE
• P1EDA
• AdArte

---

EDA/Group	Data	Lang	Lexical Resources	Results	Exp
BIUTEE
Bar Ilan	RTE-3	EN
Bar Ilan	RTE-3	EN	WN,CV		link
Bar Ilan	NICE	EN	no res (mixed split, balanced)	0.6309 (F1: 0.7171)
Bar Ilan	NICE	EN	no res (pure split, balanced	0.6204 (F1: 0.7119)
Bar Ilan	NICE	EN	no res (mixed split, not balanced)	0.7919 (F1: 0.6196)
Bar Ilan	NICE	EN	no res (pure split, not balanced)	0.8043 (F1: 0.3929)
Bar Ilan	NICE	EN	WN,VO (mixed split, balanced)	0.6406 (F1: 0.7227)
Bar Ilan	NICE	EN	WN,VO (pure split, balanced)	0.6209 (F1: 0.7125)
Bar Ilan	NICE	EN	WN,VO (mixed split, not balanced)	0.7911 (F1: 0.6270)
Bar Ilan	NICE	EN	WN,VO (pure split, not balanced)	0.8482 (F1: 0.2394)
EditDistance
FBK	RTE-3	DE		0.61250	link
FBK	RTE-3	DE	WN	0.61250	link
FBK	OMQ	DE	WN (balanced)	0.71698 (F+ : 0.70779)	link
FBK	OMQ	DE	WN (not balanced)	0.94542 (F+ : 0.24208)	link
FBK	RTE-3	EN		0.62000	link
FBK	RTE-3	EN	WN	0.63625	link
FBK	NICE	EN	WN (mixed split, balanced)	0.61698 (F+ : 0.64634)	link
FBK	NICE	EN	WN (pure split, balanced)	0.55677 (F+ : 0.66571)	link
FBK	NICE	EN	WN (mixed split, not balanced)	0.68415 (F+ : 0.53688)	link
FBK	NICE	EN	WN (pure split, not balanced)	0.63228 (F+ : 0.33439)	link
FBK	RTE-3	IT		0.62625	link
FBK	RTE-3	IT	WN	0.61625	link
FBK	ALMA	IT	(mixed split, balanced)	0.6926 (F+: 0.6001)	link
FBK	ALMA	IT	(pure split, balanced)	0.7038 (F+: 0.6531)	link
FBK	ALMA	IT	(pure split, not balanced)	0.8075 (F+ : 0.091)	link
FBK	ALMA	IT	WN (mixed split, balanced)	0.6908 (F+: 0.5982)	link
FBK	ALMA	IT	WN (pure split, balanced)	0.7084 (F+: 0.6562)	link
FBK	ALMA	IT	WN (pure split, not balanced)	0.9852 (F+ : 0.0444)	link
FBK	ALMA	IT	BAP (mixed split, balanced)		link
			ALMA_BootCat	0.6528 (F+: 0.5466)
			Wikipedia	0.6516 (F+ 0.5206)
FBK	ALMA	IT	BAP (pure split, balanced)		link
			ALMA_BootCat	0.6789 (F+: 0.6308)
			Wikipedia	0.6680 (F+ 0.5930)
FBK	ALMA	IT	DIRT (mixed split, balanced)		link
FBK	ALMA	IT	DIRT (pure split, balanced)		link
FBK	ALMA	IT	LIN-dependency (mixed split, balanced)		link
			ALMA_BootCat	0.6418 (F+: 0.5430)
			Wikipedia	0.6493 (F+ 0.5088)
FBK	ALMA	IT	LIN-dependency (pure split, balanced)		link
			ALMA_BootCat	0.6789 (F+: 0.6106)
			Wikipedia	0.6437 (F+ 0.5145)
FBK	ALMA	IT	LIN-dependency (mixed split, balanced)		link
			ALMA_BootCat	0.6931 (F+: 0.6232)
FBK	ALMA	IT	LIN-proximity (mixed split, balanced)		link
			ALMA_BootCat	0.6447 (F+: 0.5157)
			Wikipedia	0.6585 (F+ 0.5218)
FBK	ALMA	IT	LIN-proximity (pure split, balanced)		link
			ALMA_BootCat	0.6737 (F+: 0.6255)
			Wikipedia	0.6443 (F+ 0.5195)
TIE
DFKI	RTE-3	DE
DFKI	RTE-3	DE	DS,DBPos,TP,TPPos,TS	0.62875	link
DFKI	RTE-3	DE	GNPos,DBPos,TP,TPPos,TS	0.63125	link
DFKI	RTE-3	DE	GNPos,DS,DBPos,TP,TPPos,TS	0.63125	link
DFKI	OMQ	DE
DFKI	OMQ	DE	DS,DBPos,TP,TPPos,TS (balanced)	0.74371 (F+: 0.71945)	[link] (#TieOMQ1)
DFKI	OMQ	DE	GNPos,DBPos,TP,TPPos,TS (balanced)	0.74686 (F+: 0.72289)	[link] (#TieOMQ2)
DFKI	OMQ	DE	GNPos,DS,DBPos,TP,TPPos,TS (balanced)	0.74843 (F+: 0.72414)	[link] (#TieOMQ3)
DFKI	OMQ	DE	DS,DBPos,TP,TPPos,TS (not balanced)	0.97613 (F+: 0.23097)	[link] (#TieOMQ4)
DFKI	OMQ	DE	GNPos,DBPos,TP,TPPos,TS (not balanced)	0.97613 (F+: 0.23097)	[link] (#TieOMQ5)
DFKI	OMQ	DE	GNPos,DS,DBPos,TP,TPPos,TS (not balanced)	0.97613 (F+: 0.23097)	[link] (#TieOMQ6)
DFKI	RTE-3	EN		0.67375	link
DFKI	RTE-3	EN	VO,TP,TPPos,TS	0.67625	link
DFKI	RTE-3	EN	WN,TP,TPPos,TS	0.67625	link
DFKI	RTE-3	EN	WN,VO,TP,TPPos,TS	0.67500	link
DFKI	NICE	EN	base (mixed split, balanced)	0.57377 (F+: 0.65622)	[link] (#TieNice1)
DFKI	NICE	EN	VO,TP,TPPos,TS (dito)	0.61208 (F+: 0.64206)	[link] (#TieNice2)
DFKI	NICE	EN	WN,TP,TPPos,TS (dito)	0.61736 (F+: 0.65179)	[link] (#TieNice3)
DFKI	NICE	EN	WN,VO,TP,TPPos,TS (dito)	0.61755 (F+: 0.65178)	[link] (#TieNice4)
DFKI	NICE	EN	base (pure split, balanced)	0.58952 (F+: 0.67280)	[link] (#TieNice5)
DFKI	NICE	EN	VO,TP,TPPos,TS (dito)	0.62604 (F+: 0.68950)	[link] (#TieNice6)
DFKI	NICE	EN	WN,TP,TPPos,TS (dito)	0.62737 (F+: 0.69065)	[link] (#TieNice7)
DFKI	NICE	EN	WN,VO,TP,TPPos,TS (dito)	0.62850 (F+: 0.69130)	[link] (#TieNice8)
DFKI	NICE	EN	base (mixed split, no balanced)	0.48330 (F+: 0.32384)	[link] (#TieNice9)
DFKI	NICE	EN	VO,TP,TPPos,TS (dito)	0.59023 (F+: 0.30757)	[link] (#TieNice10)
DFKI	NICE	EN	WN,TP,TPPos,TS (dito)	0.61509 (F+: 0.30801)	[link] (#TieNice11)
DFKI	NICE	EN	WN,VO,TP,TPPos,TS (dito)	0.62092 (F+: 0.30638)	[link] (#TieNice12)
DFKI	NICE	EN	base (pure split, not balanced)	0.38277 (F+: 0.24408)	[link] (#TieNice13)
DFKI	NICE	EN	VO,TP,TPPos,TS (dito)	0.55226 (F+: 0.23625)	[link] (#TieNice14)
DFKI	NICE	EN	WN,TP,TPPos,TS (dito)	0.59490 (F+: 0.21376)	[link] (#TieNice15)
DFKI	NICE	EN	WN,VO,TP,TPPos,TS (dito)	0.58644 (F+: 0.21265)	[link] (#TieNice16)
DFKI	RTE-3	IT		0.6375 (F+: 0.6375)	[link] (#TRTEIT)
FBK	ALMA	IT	(mixed split, balanced)	0.5916 (F+: 0.6435)	[link] (#TAMB)
FBK	ALMA	IT	(pure split, balanced)	0.5577 (F+: 0.5947)	[link] (#TAPB)
AdArte
FBK	NICE	EN	WN,VO,CV(mixed split, balanced)	0.798 (F+ : 0.796)	link
FBK	NICE	EN	WN,VO,CV(pure split, balanced)	0.659 (F+ : 0.693)	link
FBK	NICE	EN	WN,VO,CV(mixed split, not balanced)	0.868 (F+ : 0.702)	link
FBK	NICE	EN	WN,VO,CV(pure split, not balanced)	0.702 (F+ : 0.271)	link
P1EDA
FBK	ALMA	IT	(mixed split, balanced)		[link] (#P1AMB)
			no resource	0.7445 (F+: 0.6942)
			paraphrase table	0.7474 (F+: 0.6975)
FBK	ALMA	IT	(pure split, balanced)	0.7344 (F+: 0.6949)	[link] (#P1APB)
FBK	NICE	EN	(mixed split, balanced)	0.6704 (F+: 0.6477)	[link] (#P1NMB)
FBK	NICE	EN	(pure split, balanced)		[link] (#P1NPB)
			no resource	0.6671 (F+: 0.6918)
			paraphrase table	0.6665 (F+: 0.6915)
DFKI	OMQ	DE	base: identical lemma alignment + meteor-paraphrase alignments (balanced)	0.7107 (F+: 0.6923)	[link] (#P1fnr_de_report)
			base + derivBase (balanced)	0.7327 (F+: 0.7240)	[link] (#P1fnr_de_1_report)
			base + transDM distsim (balanced)	0.6572 (F+: 0.6305)	[link] (#P1fnr_de_2_report)
			base + derivBase and distsim (balanced)	0.6808 (F+: 0.6622)	[link] (#P1fnr_de_3_report)
			base - meteor-paraphrase (balanced)	0.7091 (F+: 0.6848)	[link] (#P1fnr_de_4_report)

CV: CatVar, WN: WordNet, WP: Wikipedia, VO: VerbOcean, GNPos: GermaNet with POS tags, DS: DewakDistributional, DBPos: DerivBase with POS tags, TP: TreePattern, TPPos: TreePattern with POS tags, TS: TreeSkeleton

BIUTEE EDA

**EXPERIMENT:** B1

Configuration ID: BIUTEE_WordNet_CatVar_EN

Data Set: RTE-3
Language: EN
Lexical Resources: WN, CV
Preprocessing:
Results(Accuracy): ----

Author name: Omer Levy
Affiliation: Bar-Ilan University
Contact:

Download the configuration file, model and results

EditDistance EDA

**EXPERIMENT:** E1

Configuration ID: EditDistanceEDA_DE

Data Set: RTE-3
Language: DE
Lexical Resources: none
Preprocessing: TreeTagger
Results(Accuracy): 0.61250

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** E2

Configuration ID: EditDistanceEDA_DE_WordNet

Data Set: RTE-3
Language: DE
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.61250

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** E3

Configuration ID: EditDistanceEDA_EN

Data Set: RTE-3
Language: EN
Lexical Resources: none
Preprocessing: TreeTagger
Results(Accuracy): 0.62000

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** E4

Configuration ID: EditDistanceEDA_EN_WordNet

Data Set: RTE-3
Language: EN
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.63625

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** E5

Configuration ID: EditDistanceEDA_IT

Data Set: RTE-3
Language: IT
Lexical Resources: none
Preprocessing: TextPro
Results(Accuracy): 0.62625

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** E6

Configuration ID: EditDistanceEDA_IT_WordNet

Data Set: RTE-3
Language: IT
Lexical Resources: WN
Preprocessing: TextPro
Results(Accuracy): 0.61625

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** EOMQ1

Configuration ID: EditDistanceEDA_DE

Data Set: OMQ, OMQ_BALANCED_TH_SET
Language: DE
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.71698 (F1:0.70779, Pr:0.73154, Re:0.68553)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** EOMQ2

Configuration ID: EditDistanceEDA_DE

Data Set: OMQ, OMQ_UNBALANCED_TH_SET
Language: DE
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.94542 (F1:0.24208, Pr:0.18905, Re:0.33648)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** ENICE1

Configuration ID: EditDistanceEDA_EN

Data Set: NICE, NICE_revised_balanced_mixSplit_1X
Language: ENG
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.61698 (F1:0.64634, Pr:0.59341, Re:0.70964)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** ENICE2

Configuration ID: EditDistanceEDA_EN

Data Set: NICE, NICE_revised_balanced_pureSplit_1X
Language: ENG
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.55677 (F1:0.66571, Pr:0.53437, Re:0.88266)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** ENICE3

Configuration ID: EditDistanceEDA_EN

Data Set: NICE, NICE_revised_mixSplit_1X
Language: ENG
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.68414 (F1:0.53688, Pr:0.42841, Re:0.71892)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** ENICE4

Configuration ID: EditDistanceEDA_EN

Data Set: NICE, NICE_revised_pureSplit_1X
Language: ENG
Lexical Resources: WN
Preprocessing: TreeTagger
Results(Accuracy): 0.63228 (F1:0.33439, Pr:0.21707, Re:0.72762)

Author name: Roberto Zanoli
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: zanoli [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMB

Configuration ID: EditDistanceEDA_IT

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources:
Preprocessing: TreeTagger
Results(Accuracy): 0.6926 (F+: 0.6001)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APB

Configuration ID: EditDistanceEDA_IT

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources:
Preprocessing: TreeTagger
Results(Accuracy): 0.7038 (F+: 0.6531)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** API

Configuration ID: EditDistanceEDA_IT

Data Set: ALMA (revised July 2014) -- not balanced, pure (all instances from a cluster in the same partition)
Test set details: 515 positive/ 34355 negative
Language: IT
Lexical Resources:
Preprocessing: TreeTagger
Results(Accuracy): 0.8075 (F+: 0.0910)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBW

Configuration ID: EditDistanceEDA_IT_WordNet

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: WordNet
Preprocessing: TreeTagger
Results(Accuracy): 0.6908 (F+: 0.5982)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APBW

Configuration ID: EditDistanceEDA_IT_WN

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: WordNet
Preprocessing: TreeTagger
Results(Accuracy): 0.7084 (F+: 0.6562)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APIW

Configuration ID: EditDistanceEDA_IT_WN

Data Set: ALMA (revised July 2014) -- not balanced, pure (all instances from a cluster in the same partition)
Test set details: 515 positive/34355 negative
Language: IT
Lexical Resources: WordNet
Preprocessing: TreeTagger
Results(Accuracy): 0.9852 (F+: 0.0444)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBB

Configuration ID: EditDistanceEDA_BAP_IT

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: BAP-style

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6528 (F+: 5466)
• Wikipedia -- 0.6516 (F+: 0.5206)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APBB

Configuration ID: EditDistanceEDA_BAP_IT

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: BAP-style

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6789 (F+: 0.6308)
• Wikipedia -- 0.6680 (F+: 5930)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBD

Configuration ID: EditDistanceEDA_IT_DIRT

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: DIRT (from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data)
Preprocessing: TreeTagger
Results(Accuracy): 0.6314

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APBD

Configuration ID: EditDistanceEDA_IT_DIRT

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: DIRT (from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data)
Preprocessing: TreeTagger
Results(Accuracy): 0.7297

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBLd

Configuration ID: EditDistanceEDA_IT_LIN-deps

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: LIN dependency

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6418 (F+: 0.5430)
• Wikipedia -- 0.6493 (F+: 0.5088)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APBLd

Configuration ID: EditDistanceEDA_IT_LIN-deps

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: LIN dependency

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6789 (F+: 0.6308)
• Wikipedia -- 0.6680 (F+: 5930)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBLdTP

Configuration ID: EditDistanceEDA_IT_LIN-deps

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: LIN dependency

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data

Preprocessing: TextPro Results(Accuracy):

• ALMA_BootCat -- 0.6931 (F+: 0.6232)

Author name: Simone Magnolini Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: magnolini [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** AMBLp

Configuration ID: EditDistanceEDA_IT_LIN-prox

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: LIN proximity

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6447 (F+: 0.5157)
• Wikipedia -- 0.6585 (F+: 0.5218)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

**EXPERIMENT:** APBLp

Configuration ID: EditDistanceEDA_IT_LIN-prox

Data Set: ALMA (revised July 2014) -- balanced, pure (all instances from a cluster in the same partition)
Language: IT
Lexical Resources: LIN proximity

• ALMA_BootCat -- from a "custom-made" corpus for ALMA, using BootCat and query terms from ALMA data
• Wikipedia -- from a corpus of Wikipedia articles (Italian Wikipedia version of 20140228)

Preprocessing: TreeTagger
Results(Accuracy):

• ALMA_BootCat -- 0.6737 (F+: 0.6255)
• Wikipedia -- 0.6443 (F+: 0.5195)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [at] fbk.eu

Download the configuration file, model and results

TIE EDA

**EXPERIMENT:** T1

Configuration ID: MaxEntClassificationEDA_Base+DS+DBPos+TP+TPPos+TS_DE

Data Set: RTE-3
Language: DE
Lexical Resources: DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.62875

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T2

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DBPos+TP+TPPos+TS_DE

Data Set: RTE-3
Language: DE
Lexical Resources: GNPos,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.63125

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T3

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DS+DBPos+TP+TPPos+TS_DE

Data Set: RTE-3
Language: DE
Lexical Resources: GNPos,DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.63125

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ1

Configuration ID: MaxEntClassificationEDA_Base+DS+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_BALANCED_TH_SET
Language: DE
Lexical Resources: DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.74371 (F+: 0.71945)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ2

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_BALANCED_TH_SET
Language: DE
Lexical Resources: GNPos,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.74686 (F+: 0.72289)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ3

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DS+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_BALANCED_TH_SET
Language: DE
Lexical Resources: GNPos,DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.74843 (F+: 0.72414)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ4

Configuration ID: MaxEntClassificationEDA_Base+DS+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_UNBALANCED_TH_SET
Language: DE
Lexical Resources: DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.97613 (F+: 0.23097)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ5

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_UNBALANCED_TH_SET
Language: DE
Lexical Resources: GNPos,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.97613 (F+: 0.23097)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** TieOMQ6

Configuration ID: MaxEntClassificationEDA_Base+GNPos+DS+DBPos+TP+TPPos+TS_DE

Data Set: OMQ, OMQ_UNBALANCED_TH_SET
Language: DE
Lexical Resources: GNPos,DS,DBPos,TP,TPPos,TS
Preprocessing: MaltParserDE
Results(Accuracy): 0.97613 (F+: 0.23097)

Author name: Günter Neumann, Aleksandra Gabryszak
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de, aleksandra.gabryszak [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T4

Configuration ID: MaxEntClassificationEDA_Base+VO+TP+TPPos+TS_EN

Data Set: RTE-3
Language: EN
Lexical Resources: VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.67625

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T5

Configuration ID: MaxEntClassificationEDA_Base+WN+TP+TPPos+TS_EN

Data Set: RTE-3
Language: EN
Lexical Resources: WN,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.67625

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T6

Configuration ID: MaxEntClassificationEDA_Base+WN+VO+TP+TPPos+TS_EN

Data Set: RTE-3
Language: EN
Lexical Resources: WN,VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.675

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

**EXPERIMENT:** T7

Configuration ID: MaxEntClassificationEDA_Base+OpenNLP_EN

Data Set: RTE-3
Language: EN
Lexical Resources:
Preprocessing: OpenNLP tokenization
Results(Accuracy): 0.67375

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice1

Configuration ID: MaxEntClassificationEDA_Base+OpenNLP_EN

Data Set: NICE_revised_balanced_mixSplit_1X
Language: EN
Lexical Resources:
Preprocessing: OpenNLP tokenization
Results(Accuracy): 0.57377 (F+: 0.65622)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice2

Configuration ID: MaxEntClassificationEDA_Base+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_mixSplit_1X
Language: EN
Lexical Resources: VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.61208 (F+: 0.64206)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice3

Configuration ID: MaxEntClassificationEDA_Base+WN+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_mixSplit_1X
Language: EN
Lexical Resources: WN,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.61736 (F+: 0.65179)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice4

Configuration ID: MaxEntClassificationEDA_Base+WN+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_mixSplit_1X
Language: EN
Lexical Resources: WN,VO, TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.61755 (F+: 0.65178)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

|EXPERIMENT: TieNice5

Configuration ID: MaxEntClassificationEDA_Base+OpenNLP_EN

Data Set: NICE_revised_balanced_pureSplit_1X
Language: EN
Lexical Resources:
Preprocessing: OpenNLP tokenization
Results(Accuracy): 0.58952 (F+: 0.67280)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice6

Configuration ID: MaxEntClassificationEDA_Base+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_pureSplit_1X
Language: EN
Lexical Resources: VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.62604 (F+: 0.68950)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice7

Configuration ID: MaxEntClassificationEDA_Base+WN+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_pureSplit_1X
Language: EN
Lexical Resources: WN,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.62737 (F+: 0.69065)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice8

Configuration ID: MaxEntClassificationEDA_Base+WN+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_balanced_pureSplit_1X Language: EN
Lexical Resources: WN,VO, TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.62850 (F+: 0.69130)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice9

Configuration ID: MaxEntClassificationEDA_Base+OpenNLP_EN

Data Set: NICE_revised_mixSplit_1X
Language: EN
Lexical Resources:
Preprocessing: OpenNLP tokenization
Results(Accuracy): 0.48330 (F+: 0.32384)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice10

Configuration ID: MaxEntClassificationEDA_Base+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_mixSplit_1X
Language: EN
Lexical Resources: VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.59023 (F+: 0.30757)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice11

Configuration ID: MaxEntClassificationEDA_Base+WN+TP+TPPos+TS_EN

Data Set: NICE_revised_mixSplit_1X
Language: EN
Lexical Resources: WN,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.61509 (F+: 0.30801)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice12

Configuration ID: MaxEntClassificationEDA_Base+WN+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_mixSplit_1X Language: EN
Lexical Resources: WN,VO, TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.62092 (F+: 0.30638)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice13

Configuration ID: MaxEntClassificationEDA_Base+OpenNLP_EN

Data Set: NICE_revised_pureSplit_1X
Language: EN
Lexical Resources:
Preprocessing: OpenNLP tokenization
Results(Accuracy): 0.38277 (F+: 0.24408)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice14

Configuration ID: MaxEntClassificationEDA_Base+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_pureSplit_1X
Language: EN
Lexical Resources: VO,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.55226 (F+: 0.23625)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice15

Configuration ID: MaxEntClassificationEDA_Base+WN+TP+TPPos+TS_EN

Data Set: NICE_revised_pureSplit_1X
Language: EN
Lexical Resources: WN,TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.59490 (F+: 0.21376)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TieNice16

Configuration ID: MaxEntClassificationEDA_Base+WN+VO+TP+TPPos+TS_EN

Data Set: NICE_revised_pureSplit_1X Language: EN
Lexical Resources: WN,VO, TP,TPPos,TS
Preprocessing: MaltParserEN
Results(Accuracy): 0.58644 (F+: 0.21265)

Author name: Günter Neumann
Affiliation: Deutsche Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)
Contact Information: neumann [@] dfki.de

Download the configuration file, model and results

EXPERIMENT: TRTEIT

Configuration ID: MaxEntClassificationEDA_IT

Data Set: RTE3-Italian
Language: IT
Lexical Resources: Preprocessing: TreeTaggerIT
Results(Accuracy): 0.6375 (F+: 0.6555)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: TAMB

Configuration ID: MaxEntClassificationEDA_IT

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: Preprocessing: TreeTaggerIT
Results(Accuracy): 0.5916 (F+: 0.6435)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: TAMB

Configuration ID: MaxEntClassificationEDA_IT

Data Set: ALMA (revised July 2014) -- balanced, pure (from clusters)
Language: IT
Lexical Resources: Preprocessing: TreeTaggerIT
Results(Accuracy): 0.5577 (F+: 0.5947)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

---

P1EDA (Alignment EDA)

EXPERIMENT: P1AMB

Configuration ID: P1EDA_Base

Data Set: ALMA (revised July 2014) -- balanced, mixed (from clusters)
Language: IT
Lexical Resources: none / Italian paraphrase table
Preprocessing: TreeTaggerIT
Results(Accuracy):

• none : 0.7445 (F+: 0.6942)
• Italian paraphrase table: 0.7474 (F+: 0.6975)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: P1APB

Configuration ID: P1EDA_Base

Data Set: ALMA (revised July 2014) -- balanced, pure (from clusters)
Language: IT
Lexical Resources: Italian paraphrase table
Preprocessing: TreeTaggerIT
Results(Accuracy): 0.7344 (F+: 0.6949)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: P1NMB

Configuration ID: P1EDA_Base

Data Set: NICE (revised July 2014) -- balanced, mixed (from clusters)
Language: EN
Lexical Resources: English paraphrase table
Preprocessing: TreeTaggerEN
Results(Accuracy): 0.6704 (F+: 0.6477)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: P1NPB

Configuration ID: P1EDA_Base

Data Set: NICE (revised July 2014) -- balanced, pure (from clusters)
Language: EN
Lexical Resources: none / English paraphrase table
Preprocessing: TreeTaggerEN
Results(Accuracy):

• none: 0.6671 (F+: 0.6918)
• English paraphrase table: 0.6665 (F+: 0.6915)

Author name: Vivi Nastase
Affiliation: Fondazione Bruno Kessler (FBK)
Contact Information: nastase [@] fbk.eu

Download the configuration file, model and results

EXPERIMENT: P1fnr_de_report

Configuration ID: P1fnr_de_report

Data Set: OMQ (data set revision?) -- balanced
Language: DE
Lexical Resources: base (i.e. identical lemma alignment + meteor-paraphrase alignments)
Preprocessing: TreeTaggerDE
Results(Accuracy): 0.7107

Author name: Kathrin Eichler
Affiliation: DFKI Contact Information: kathrin.eichler [@] dfki.de

EXPERIMENT: P1fnr_de_1_report

Configuration ID: P1fnr_de_1_report

Data Set: OMQ (data set revision?) -- balanced
Language: DE
Lexical Resources: base + derivBase
Preprocessing: TreeTaggerDE
Results(Accuracy): 0.7327

Author name: Kathrin Eichler
Affiliation: DFKI Contact Information: kathrin.eichler [@] dfki.de

EXPERIMENT: P1fnr_de_2_report

Configuration ID: P1fnr_de_2_report

Data Set: OMQ (data set revision?) -- balanced
Language: DE
Lexical Resources: base + transDM distsim
Preprocessing: TreeTaggerDE
Results(Accuracy): 0.6572

Author name: Kathrin Eichler
Affiliation: DFKI Contact Information: kathrin.eichler [@] dfki.de

EXPERIMENT: P1fnr_de_3_report

Configuration ID: P1fnr_de_3_report

Data Set: OMQ (data set revision?) -- balanced
Language: DE
Lexical Resources: base + derivBase and distsim
Preprocessing: TreeTaggerDE
Results(Accuracy): 0.6808

Author name: Kathrin Eichler
Affiliation: DFKI Contact Information: kathrin.eichler [@] dfki.de

EXPERIMENT: P1fnr_de_4_report

Configuration ID: P1fnr_de_4_report

Data Set: OMQ (data set revision?) -- balanced
Language: DE
Lexical Resources: base - meteor-paraphrase (means == only identical lemma with no resources == bare minimum)
Preprocessing: TreeTaggerDE
Results(Accuracy): 0.7091

Author name: Kathrin Eichler
Affiliation: DFKI Contact Information: kathrin.eichler [@] dfki.de

---