Paper Reading Agent Integration (see Sakana AI)

[BradKML]

Saw this repo recently, and that it is very inspiring that such agents can automatically dig papers and find new ideas to test, but they used Aider instead of OpenHands, and cannot write code out of scratch. Maybe some of the ideas can be borrowed to make adding features to other codebases easier? https://github.com/SakanaAI/AI-Scientist

[SmartManoj]

Added download_arxiv_pdf skill.

Aider can write code out of scratch.

The agent can use the AI-Scientist repo directly for containerization.

[BradKML]

Definitely worth checking out these ones at least according to the table (all of them have more than SemanticScholar) https://oa.mg/data

• OA.mg https://chrome.google.com/webstore/detail/oamg/ahnnebfmpaambhblikbbbhdfaknfjfcd
• IA Scholar https://github.com/internetarchive/fatcat https://github.com/internetarchive/fatcat-scholar
• UnPaywall https://github.com/michaelplews/python-unpaywall
• CORE.ac.uk (a bit less and closer to SemanticScholar) https://github.com/ronentk/sci-paper-miner

[SmartManoj]

Added support for OA.mg and integrated PyPaperBot which is again being developed.
Support for other websites should be added to that library for best of both worlds.

[BradKML]

Also looking at how PyPaperBot is different from the other major repos https://github.com/jonatasgrosman/findpapers

[SmartManoj]

Agent downloaded a paper using Selenium.

[BradKML]

Ideally, in order of priority:

1. Download/Scrape/Crawl/Fetch scientific/academic papers with established packages (e.g. PyPaperBot) for main sites (select top sites with most papers and those with the most rare papers)
2. Modify established packages to accommodate other paper sites based on API (or take inspiration from less popular repositories), and note if the paper APIs are rate-limited or not, make PRs if necessary
3. Use Playwright/Selenium/Appium for Python, or Puppeteer/Cypress/Jest/Mocha/Karma/Nightwatch for JS/TS, to download papers, and make sure that the methodology is stable and usable

[BradKML]

As a sidenote, I need to remind myself that this exists. Paper hints as lacking in (in descending order of severity) specificity, human-level helpfulness, and alignment. It had issues in communicating novelty, citations, and theoretical soundness (and for ML specifically, ablation studies), while overemphasizing broadening datasets and elaborating on implications. https://github.com/Weixin-Liang/LLM-scientific-feedback
LLM needs to be finetuned/selected to giving better feedback (and not overfit), yet it also requires human data to do so.

[SmartManoj]

This is Human evaluation table

[BradKML]

I know, and I am thinking how majority support 2-step (corresponds to AI eval as "quality"), even when the median as 3-step (corresponding to "attributable"), and the maximum is 4-step (one of the researcher preferred "information"). As you can see each of the researcher has their own biases in what they want (quality vs attributable vs information). So for conservative estimate, anything with more than 4-step is likely going to be too cumbersome to read, with 2-step and 3-step as an optional.

[SmartManoj]

# Percentages for Step 1 by individual annotators
individual_annotators = [3.0, 2.0, 13.0, 17.4]

# Aggregate calculation as the average of all annotators' percentages
aggregate = sum(individual_annotators) / len(individual_annotators)

print(f"Aggregate for Step 1: {aggregate:.2f}%")

1. Aggregate for Step 1: 8.85%?

---

2. Why are they taking the median preferred?

---

3. For GPT-4 Evaluation, first, what do you think about 9.9 vs 9.11 comparison?

---

Step 2 is best based on both evaluation here.

[BradKML]

1. Think there might be some rounding errors + irregularities
2. Median is better than the mean due to how sometimes distribution can be skewed towards the majority (which is 2-step). But the first and second rater in the table is already enough to draw some suspicion that 2 is not enough in some cases.
3. GPT-4 definitely fails in contextuality, when it is number it fails, but when it is software versioning it succeeds. Same is said regarding "number of Rs in strawberry" if it is spelling correction it did with in 2 Rs (counter is like "library", "accessary", "arbitrary", "dietary", "military", "ordinary", "sanitary"). Context matters in these cases.

[SmartManoj]

They should choose the mode instead of mean or median. The human annotators thought Step 2 was best for 30.8% of samples.

[BradKML]

As for paper crawling and recommendation services, Thinking which ones would be better when integrated based on price, accessibility, and function.

[BradKML]

SemanticScholar mainly focuses on the search engine side of the equation and not the related paper hunting side (with topical networks/graphs), so not sure how they can be related. It's a little bit reaching tho

[SmartManoj]

Could you provide links to a Paper and one of its related papers?

[BradKML]

Here is an example of how graph-centric tools look like, where topics and similar papers can be extracted https://semopenalex.org/resource/semopenalex:UniversalSearch
Will also make a list on the others later just in case

[BradKML]

@SmartManoj extra note: someone made a blog post on "literature mapping" and it could be of interest https://musingsaboutlibrarianship.blogspot.com/p/list-of-innovative-literature-mapping.html
There are three forms: (a) Citation network (bibliographic coupling and co-citation), (b) Co-authorship graph, and (c) keyword similarities. Thinking which service would be the most useful first

[SmartManoj]

https://www.connectedpapers.com/