GSoC idea: IYP automatic deployment

[romain-fontugne]

For the Internet Yellow Pages we compute every week a new
database that we want to make publicly available as soon as possible. The goal
of this project is to design and implement this automated deployment. This
requires multiple tests that will ensure the integrity of the new database,
automatically store the new database on a public repository, and hot swap databases.
Optionally we could keep older databases online, for example all databases for the
past month.

Expected results:

• Design an easy to maintain automated pipeline that would be easy to maintain
• Implement unit tests for each of the IYP dataset
• Implement automatic deployment and hot swap of IYP databases
• (optional) Maintain multiple instances of the database

[sudo-jarvis]

Hi @romain-fontugne, I am interested to work on this project. Can we discuss this further?

[romain-fontugne]

sure :)
Here is a good place to ask general questions, so that mentors don't receive the same questions again and again from potential contributors.

[sudo-jarvis]

@romain-fontugne Are there some beginner friendly issues that relate to this project that I can start working on ? I believe this will help me understand the better.

[romain-fontugne]

First thing for this project is to get familiar with the Internet Yellow Pages. Following the instructions in the readme to learn how to deploy a local instance of the IYP.

Also this project requires knowledge in docker. So please get familiar with that too.

[sudo-jarvis]

Ok I'll start reading about IYP and try to deploy it locally. I already am familiar with docker so that would help me.
Thanks a lot.

[Sayanjones]

Hi @romain-fontugne, I am also interested to work on this project. I gone through the project, I got to know that it requires to manage workflow, runs unit tests and Neo4j checks, packages with Docker.

[UncleWeeds]

Hey @romain-fontugne, myself Abhiram I would like to work on this project too

[ryanp8]

Hi @romain-fontugne,
My name is Ryan and I'm an undergraduate student at Northwestern University studying computer science and math. I've recently taken courses in Networking and Scalable Software Architectures, and I believe this project provides a great opportunity to apply concepts I've learned from these classes in a collaborative and real-world setting. Excited to get started!

[romain-fontugne]

hey everyone, you don't need to introduce yourself here :)
Please ask questions if you have any

[ryanp8]

Hey @romain-fontugne
I noticed that the nodes and edges of the graph have both <elementId> and <id> fields. Why are there two ids? Does the reason behind this relate to why <elementId>s appear to be time-based UUIDs while <id>s are not?

[m-appel]

Both ids are generated by neo4j. The normal id is deprecated so I assume it will be removed at some point. The elementId is just the replacement.

[ryanp8]

Hi @romain-fontugne , @m-appel
I'm working on setting up IYP locally and have been having trouble with storage. I have downloaded the database dump for 2024/01/10 with curl https://exp1.iijlab.net/wip/iyp/dumps/2024/01/10/iyp-2024-01-10.dump -o dumps/neo4j.dump. The instructions say to have ~30GB of available storage, but when I uncompress the data, the data directory that's created is over 60GB, which is causing issues with storage (I get the error Failed to load database 'neo4j': Unable to load database: IOException: No space left on deviceLoad failed for databases: 'neo4j'). Do you have any pointers for how to download and uncompress the data while staying at around the 30GB? Does it depend on the specific dump I download?

[romain-fontugne]

please use a more recent dump. The latest one takes 40GB on my machine

[ryanp8]

Thanks, the most recent dump works. Just curious, what causes such a large difference between dump sizes?

[m-appel]

We are actually not sure. While we did add quite a lot of data recently, the size of our dumps has actually been decreasing. Not sure if this has to do with better compression in newer versions of neo4j (we updated recently) or something else. For example, creating a dump, loading the dump, and then creating a new dump from that results in an even smaller dump... So we don't really know.

[ryanp8]

Hey @romain-fontugne , @m-appel ,
Sorry for the delay, just finished my final exams. Now that I have a local deployment of IYP are there any specific issues or tasks you recommend that I work on? There don't seem to be any IYP-specific ones marked as a good first issue

[romain-fontugne]

For this project the first task is to automate the deployment, so maybe you can start by thinking of the best ways to do that.

[ryanp8]

In the description, it says that ultimately the pipeline should automatically store the new database on a public repository. Is this in addition to, for example, deploying a Docker container running the database, or are we just pushing a dump of the database to a repo?

[romain-fontugne]

we just want to make the dump publicly available, similar to what we do now:
https://exp1.iijlab.net/wip/iyp/dumps/

[Niloth-p]

I'd like clarification on a few things.

1. It is stated that we'd want to hotswap databases.
   I assume that means we'll be creating a new database instance for fetching.
   But, if all we want to do is fetch new data and update the database, we could automate doing that with just a single instance.
   At the possible cost of performance degradation for users accessing the database during that time (around 2 hours every week).
   So, is that the reason we're going with using a different instance to fetch the data?
   Or is it to support hosting multiple database instances in parallel later?

2. "Optionally we could keep older databases online, for example all databases for the past month."
   Is that something that IYP users typically need?
   If so, would they benefit even more from being allowed to load any database (choose by date)? We could load it up from the archive dumps into a new instance. Of course, it would take a significant loading time of 2-5 mins. But, still more accessible for users than them setting it all up (several GB) in their local instances.
   Or was the scalability for that already considered?

[Niloth-p]

Thank you for the quick and detailed answers.

[Niloth-p]

What is the reason behind the decision to include a separate entry point in each crawler script?
Why risk much redundancy of code?

[romain-fontugne]

Each crawler can be run separately.

[Niloth-p]

Ah, I should have asked that better.
It's about the "fully-fleshed" entry points.

For example,
Is there a particular reason why we do not have a common logging configuration, but include it within each crawler?

Why can't we take out all the common logic, and just import them into the crawlers?

[romain-fontugne]

Ok, no particular reason for that. Yes I guess it could share a common configuration