the code in generator and critic

[2282588541a]

In both the generator and critic parts, the json-like string produced by LLM is parsed, but in my case, the eval function reports an error, as shown below:

I try to solve this by removing the beginning and end of the string like x=output.split('```json') if len(x)==2: output=x[1] x=output.split('```') if len(x)==2: output=x[0]
By the way， 你的代码风格真的可读性巨棒！！！

[ignorejjj]

Just remove the ```json and ``` in your q.

[2282588541a]

I try to run the code in my environment,the 2wiki result shows very low. the below is a example of my result .The results from hotpot are excellent, but the results from 2wiki are not. Is there anything special about them besides the evaluation method? { "question": "Where was the place of death of the director of film Dr. Heckyl And Mr. Hype?", "reference": [ "Paul Milgrom -- rate may actually increase when asymmetric information is introduced. Finally, Milgrom and Roberts (1986) bring the asymmetric information framework to bear in analyzing the issue of advertising and pricing. Traditionally, economists have thought of advertising as being either informative (as for example classified ads, which describe the characteristics of the product for sale), or persuasive (as for example many television commercials which seem to provide little or no information about a product's characteristics). Following earlier ideas by Nelson (1970, 1974), Milgrom and Roberts (1986) show that even \"\"uninformative\"\" advertising, that is, advertising expenditures that provide no direct information about a", "The Lariat -- head through the loop of a waiting lariat, and is hung. It is ambiguous whether he was tricked, or has committed suicide. The narrative is open to interpretation. The chapter titles provide clues, though sometimes they do not seem directly connected to their context (\"\"Fray Luis tries to double-cross the Devil,\"\" for example). By using information provided in the first chapter to decipher the titles' meanings, the reader can more fully grasp what is taking place in the often confusing final chapters of this text. There is a lot of \"\"mixing\"\" of opposites in this story that is a direct", "Al Jennings -- \"\"The Saturday Evening Post\"\" wrote a series of interviews with Jennings that perpetuated the same messages as his novel. He re-created one of his bank robberies in the 1908 film \"\"The Bank Robbery\"\". In this film, Heck Thomas assembled a posse, chased and captured the bank robbers. Bill Tilghman was the director, James Bennie Kent was the cinema-photographer, and the Oklahoma Natural Mutoscene Company was the producer. The film was shot in Cache, Oklahoma and at the Wichita Mountains Wildlife Refuge, with Quanah Parker having a bit part. A bystander thought that the bank was really being robbed and jumped", "Kurt Cobain: Montage of Heck -- on both HBO and international cinema releases, director Brett Morgen stated in an interview online that an album will be released in the months following the film's release that will include home recordings. Morgen said the album \"\"will feel like you're kind of hanging out with Kurt Cobain on a hot summer day in Olympia, Washington as he fiddles about\"\" but that \"\"it's not a Nirvana album.\"\" The album will feature at least one unreleased track. Morgen stated that the album will be 85 minutes long. Morgen also replied to a tweet question and answer, in which he said the", "Jeff Danna -- drama \"\"Fracture\"\"; \"\"The Imaginarium of Dr. Parnassus\"\" and \"\"Tideland,\"\" both directed by Terry Gilliam; \"\"Lakeview Terrace,\"\" directed by Neil LaBute; a series of films for director Brett Morgen, including \"\"The Kid Stays in The Picture,\"\" \"\"Chicago 10,\"\" and, most recently, the Kurt Cobain biopic \"\"Montage of Heck\"\"; \"\"Closing the Ring\"\" for director Richard Attenborough; and the cult classic \"\"The Boondock Saints,\"\" among others. Additionally, his collaborative Orchestral Celtic albums with his brother, composer Mychael Danna, have enjoyed worldwide success and placed in the Top Ten on the Billboard chart in the United States. For television, Jeff and Mychael Danna wrote", "Dr. Heckyl and Mr. Hype -- Dr. Heckyl and Mr. Hype Dr. Heckyl and Mr. Hype is a 1980 film directed by Charles B. Griffith, starring Oliver Reed and Sunny Johnson. The film is a reversal of the \"\"Dr. Jekyll and Mr. Hyde\"\" story, about a malformed doctor who drinks a potion and becomes a handsome (and violent) ladies' man. Griffith says the film's title was one of five joke titles he originally came up with to show Francis Ford Coppola. He then showed them to Menahem Golan, who was talking to Griffith about writing \"\"The Happy Hooker Goes to Washington\"\". After that fell through, Golan", "Charles B. Griffith -- movie, \"\"Eat My Dust!\"\" (1976), a massive hit for Corman's New World Pictures. Less successful was \"\"Up from the Depths\"\" (1979) shot in the Philippines for Corman, which Griffith directed. For the Cannon Group he wrote and directed the comedy, \"\"Dr. Heckyl and Mr. Hype\"\" (1980) with Oliver Reed. He directed another car chase film for Corman, \"\"Smokey Bites the Dust\"\" (1981). From the 1980s onwards Griffith concentrated on writing books and travelling as opposed to writing screenplays. In 1982 a stage adaptation of \"\"Little Shop of Horrors\"\" premiered and went on to enjoy great success, with many productions all", "Dr. Heckyl and Mr. Hype -- he was Hype, he didn't know how to do it… Reed played Hype as Oliver Reed, slow and ponderous. He didn't understand my interpretation, so the picture jars, and half the people get up and walk out. Dr. Heckyl and Mr. Hype Dr. Heckyl and Mr. Hype is a 1980 film directed by Charles B. Griffith, starring Oliver Reed and Sunny Johnson. The film is a reversal of the \"\"Dr. Jekyll and Mr. Hyde\"\" story, about a malformed doctor who drinks a potion and becomes a handsome (and violent) ladies' man. Griffith says the film's title was one of five", "Online youth radicalization -- discussion forums because they develop communication among followers of the same ideas (experiences, ideas, values), reinforce interpersonal relationships and provide information about actions (tactics, objectives, tutorials)\"\". Chatrooms can also include spaces where extremist people share information such as photos, videos, guides, and manuals. Many extremist groups are ideologically and strategically anti-Facebook, but a strong presence still exists on this platform either directly or through supporters. Facebook does not seem to be used for direct recruitment or planning, possibly because it has mechanisms of tracking and can link users with real places and specific times. Facebook appears to have been more" ], "final_answer": "San Diego, California, USA", "interaction_log": [ { "original_output": { "reason": "The director of the film 'Dr. Heckyl And Mr. Hype' was Charles B. Griffith. Charles B. Griffith passed away in San Diego, California, USA.", "answer": "San Diego, California, USA" }, "monitor_result": { "pseduo_answer": "", "score": 0.054824214428663254 }, "add_reference": [ "Online youth radicalization -- discussion forums because they develop communication among followers of the same ideas (experiences, ideas, values), reinforce interpersonal relationships and provide information about actions (tactics, objectives, tutorials)\"\". Chatrooms can also include spaces where extremist people share information such as photos, videos, guides, and manuals. Many extremist groups are ideologically and strategically anti-Facebook, but a strong presence still exists on this platform either directly or through supporters. Facebook does not seem to be used for direct recruitment or planning, possibly because it has mechanisms of tracking and can link users with real places and specific times. Facebook appears to have been more", "Paul Milgrom -- rate may actually increase when asymmetric information is introduced. Finally, Milgrom and Roberts (1986) bring the asymmetric information framework to bear in analyzing the issue of advertising and pricing. Traditionally, economists have thought of advertising as being either informative (as for example classified ads, which describe the characteristics of the product for sale), or persuasive (as for example many television commercials which seem to provide little or no information about a product's characteristics). Following earlier ideas by Nelson (1970, 1974), Milgrom and Roberts (1986) show that even \"\"uninformative\"\" advertising, that is, advertising expenditures that provide no direct information about a", "The Lariat -- head through the loop of a waiting lariat, and is hung. It is ambiguous whether he was tricked, or has committed suicide. The narrative is open to interpretation. The chapter titles provide clues, though sometimes they do not seem directly connected to their context (\"\"Fray Luis tries to double-cross the Devil,\"\" for example). By using information provided in the first chapter to decipher the titles' meanings, the reader can more fully grasp what is taking place in the often confusing final chapters of this text. There is a lot of \"\"mixing\"\" of opposites in this story that is a direct" ], "critic_result": { "judgement": "correct", "feedback": "The user's answer is correct. San Diego, California, USA is the correct place of death of the director of the film Dr. Heckyl And Mr. Hype.", "internal_judge": true, "external_judge": false }, "new_answer": { "reason": "The director of the film Dr. Heckyl and Mr. Hype was Charles B. Griffith. Searching for information about him reveals that he died in Huston, Texas, United States. Therefore, the place of death of the director of the film Dr. Heckyl and Mr. Hype was Houston, Texas, United States.", "answer": "Houston, Texas, United States" } }, { "original_output": { "reason": "The director of the film Dr. Heckyl and Mr. Hype was Charles B. Griffith. Searching for information about him reveals that he died in Huston, Texas, United States. Therefore, the place of death of the director of the film Dr. Heckyl and Mr. Hype was Houston, Texas, United States.", "answer": "Houston, Texas, United States" }, "monitor_result": { "pseduo_answer": "", "score": 0.0960661917924881 }, "add_reference": [ "Dr. Heckyl and Mr. Hype -- Dr. Heckyl and Mr. Hype Dr. Heckyl and Mr. Hype is a 1980 film directed by Charles B. Griffith, starring Oliver Reed and Sunny Johnson. The film is a reversal of the \"\"Dr. Jekyll and Mr. Hyde\"\" story, about a malformed doctor who drinks a potion and becomes a handsome (and violent) ladies' man. Griffith says the film's title was one of five joke titles he originally came up with to show Francis Ford Coppola. He then showed them to Menahem Golan, who was talking to Griffith about writing \"\"The Happy Hooker Goes to Washington\"\". After that fell through, Golan", "Dr. Heckyl and Mr. Hype -- he was Hype, he didn't know how to do it… Reed played Hype as Oliver Reed, slow and ponderous. He didn't understand my interpretation, so the picture jars, and half the people get up and walk out. Dr. Heckyl and Mr. Hype Dr. Heckyl and Mr. Hype is a 1980 film directed by Charles B. Griffith, starring Oliver Reed and Sunny Johnson. The film is a reversal of the \"\"Dr. Jekyll and Mr. Hyde\"\" story, about a malformed doctor who drinks a potion and becomes a handsome (and violent) ladies' man. Griffith says the film's title was one of five", "Charles B. Griffith -- movie, \"\"Eat My Dust!\"\" (1976), a massive hit for Corman's New World Pictures. Less successful was \"\"Up from the Depths\"\" (1979) shot in the Philippines for Corman, which Griffith directed. For the Cannon Group he wrote and directed the comedy, \"\"Dr. Heckyl and Mr. Hype\"\" (1980) with Oliver Reed. He directed another car chase film for Corman, \"\"Smokey Bites the Dust\"\" (1981). From the 1980s onwards Griffith concentrated on writing books and travelling as opposed to writing screenplays. In 1982 a stage adaptation of \"\"Little Shop of Horrors\"\" premiered and went on to enjoy great success, with many productions all" ], "critic_result": { "judgement": "Correct", "feedback": "The user's answer is correct. The location of death of the director of the film 'Dr. Heckyl And Mr. Hype' was indeed in Houston, Texas, United States.", "internal_judge": true, "external_judge": false }, "new_answer": { "reason": "```\n{\"reason\": \"The director of the film 'Dr. Heckyl and Mr. Hype' was Charles B. Griffith. According to the provided reference, Charles B. Griffith died in Houston, Texas, United States.\", \"answer\": \"Houston, Texas, United States\"}\n```", "answer": "```\n{\"reason\": \"The director of the film 'Dr. Heckyl and Mr. Hype' was Charles B. Griffith. According to the provided reference, Charles B. Griffith died in Houston, Texas, United States.\", \"answer\": \"Houston, Texas, United States\"}\n```" } }, { "original_output": { "reason": "```\n{\"reason\": \"The director of the film 'Dr. Heckyl and Mr. Hype' was Charles B. Griffith. According to the provided reference, Charles B. Griffith died in Houston, Texas, United States.\", \"answer\": \"Houston, Texas, United States\"}\n```", "answer": "```\n{\"reason\": \"The director of the film 'Dr. Heckyl and Mr. Hype' was Charles B. Griffith. According to the provided reference, Charles B. Griffith died in Houston, Texas, United States.\", \"answer\": \"Houston, Texas, United States\"}\n```" }, "monitor_result": { "pseduo_answer": "", "score": 0.026578115299344063 }, "add_reference": [ "Kurt Cobain: Montage of Heck -- on both HBO and international cinema releases, director Brett Morgen stated in an interview online that an album will be released in the months following the film's release that will include home recordings. Morgen said the album \"\"will feel like you're kind of hanging out with Kurt Cobain on a hot summer day in Olympia, Washington as he fiddles about\"\" but that \"\"it's not a Nirvana album.\"\" The album will feature at least one unreleased track. Morgen stated that the album will be 85 minutes long. Morgen also replied to a tweet question and answer, in which he said the", "Jeff Danna -- drama \"\"Fracture\"\"; \"\"The Imaginarium of Dr. Parnassus\"\" and \"\"Tideland,\"\" both directed by Terry Gilliam; \"\"Lakeview Terrace,\"\" directed by Neil LaBute; a series of films for director Brett Morgen, including \"\"The Kid Stays in The Picture,\"\" \"\"Chicago 10,\"\" and, most recently, the Kurt Cobain biopic \"\"Montage of Heck\"\"; \"\"Closing the Ring\"\" for director Richard Attenborough; and the cult classic \"\"The Boondock Saints,\"\" among others. Additionally, his collaborative Orchestral Celtic albums with his brother, composer Mychael Danna, have enjoyed worldwide success and placed in the Top Ten on the Billboard chart in the United States. For television, Jeff and Mychael Danna wrote", "Al Jennings -- \"\"The Saturday Evening Post\"\" wrote a series of interviews with Jennings that perpetuated the same messages as his novel. He re-created one of his bank robberies in the 1908 film \"\"The Bank Robbery\"\". In this film, Heck Thomas assembled a posse, chased and captured the bank robbers. Bill Tilghman was the director, James Bennie Kent was the cinema-photographer, and the Oklahoma Natural Mutoscene Company was the producer. The film was shot in Cache, Oklahoma and at the Wichita Mountains Wildlife Refuge, with Quanah Parker having a bit part. A bystander thought that the bank was really being robbed and jumped" ], "critic_result": { "judgement": "Correct", "feedback": "The user's answer is accurate and aligns with the question's requirement.", "internal_judge": true, "external_judge": false }, "new_answer": { "reason": "The director of the film Dr. Heckyl and Mr. Hype was Charles B. Griffith. He passed away in his home city of San Diego, California, USA. ", "answer": "San Diego, California, USA" } }, { "original_output": { "reason": "The director of the film Dr. Heckyl and Mr. Hype was Charles B. Griffith. He passed away in his home city of San Diego, California, USA. ", "answer": "San Diego, California, USA" }, "monitor_result": { "pseduo_answer": "", "score": 0.054824214428663254 } } ], "answer": "San Diego" }

[ignorejjj]

It seems there is no problem with the operation of this case. You can try a few more cases and take a look.

[2282588541a]

I have checked my code and that seems there is no mistake. Moreover, the results on the hotpotqa dataset are mostly equal to the results in your paper. After additional checking, we found that the 2wiki dataset has many versions. I wonder which version did you use in your experiments and the source link is appreciate.

[ignorejjj]

Due to a bit of time, my original data file can no longer be found. In my impression, you can try using the version from my FlashRAG library.

[2282588541a]

I noticed that the two fine-tuned expert models in your code are fine-tuned on the hotpot dataset. Did you fine-tune an expert model on the 2wiki dataset during your research? Will this improve the performance of the 2wiki dataset?

[ignorejjj]

Sorry for delayed response. We don't fine-tune any models in this work.