Python Import Index

This Undergraduate research project was done by Kolby Moroz Liebl and Marafi Mergani with research department UAlberta Software Maintenance and Reuse (SMR) at the University of Alberta. Our supervisors were Mohayeminul Islam and Sarah Nadi

Instructions for using the tools we created locally will be provided below and instructions to deploy our "Flask API Interface" using docker compose

Readme Index

• Python Import Index
• Readme Index
• Requirements
• Setup Instructions Ubuntu
• Setup on SMR server using docker compose
• Initialize/Update dataset
• Run Flask API Interface
• Examples of interacting with the Flask API Interface
  • Examples of /library/ api
  • Examples of /importname/ api
  • Examples of /dump api

Requirements

• python 3.11+
• postgres
• git

Setup Instructions Ubuntu

We assume you already have Postgres installed.

Our interface to the database is using a rest api built using Flask

Clone the project

git clone https://github.com/ualberta-smr/Python-Import-Index.git

Enter the directory

cd Python-Import-Index

Initialize Python Virtual Environment

python3 -m venv venv
. venv/bin/activate

Install project library requirements

pip install -r requirements.txt

Setup on SMR server using docker compose

# git clone to any repo it doesn't matter
git clone https://github.com/ualberta-smr/Python-Import-Index.git
cd Python-Import-Index
# edit enviroment variables for postgres
nano docker-compose.yml
# Run the app, running this will run the program on system reboot as well
docker compose up -d --pull "always"

Click this http://localhost:6969 and you should see our how page

Initialize/Update dataset

Run this script when you want to Initialize/Update the dataset. If you want to speed this process up you can configure the amount of python processes collecting the data in the file

python run-this-to-initialize-and-update-dataset.py

Run Flask API Interface

Run this script to launch the python Flask interface for the project

python app.py

Examples of interacting with the Flask API Interface

Run this script to launch the python Flask interface for the project

python app.py

You should get

WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on http://127.0.0.1:5000
Press CTRL+C to quit

Take the url provided in this case http://127.0.0.1:5000

Examples of /library/ api

This api takes a library's name and provides the respective library import names

• libraries are comma separated
• optional version argument can be added “:x.y.z” e.x. “discord.py:0.11.0”

So you can enter http://127.0.0.1:5000/library/discord.py:0.11.0,setuptools into your browser and get

{
    "error": [],
    "result": [
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.11.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "67.8.0"
        },
        {
            "import_name": "setuptools",
            "library_name": "setuptools",
            "version": "67.8.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.0.0"
        },
        {
            "import_name": "setuptools",
            "library_name": "setuptools",
            "version": "68.0.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.1.0"
        },
        {
            "import_name": "setuptools",
            "library_name": "setuptools",
            "version": "68.1.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.2.0"
        },
        {
            "import_name": "setuptools",
            "library_name": "setuptools",
            "version": "68.2.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "69.0.0"
        },
        {
            "import_name": "setuptools",
            "library_name": "setuptools",
            "version": "69.0.0"
        }
    ]
}

Examples of /importname/ api

This api takes a library import names and provides the respective library names

• libraries are comma separated
• this api endpoint doesn't provide a version argument

So you can enter http://127.0.0.1:5000/importname/discord,pkg_resources into your browser and get

{
    "error": [],
    "result": [
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.10.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.11.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.12.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.13.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.14.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.15.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.16.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.4.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.5.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.6.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.7.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.8.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "0.9.0"
        },
        {
            "import_name": "discord",
            "library_name": "discord.py",
            "version": "2.3.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "67.8.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.0.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.1.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "68.2.0"
        },
        {
            "import_name": "pkg_resources",
            "library_name": "setuptools",
            "version": "69.0.0"
        }
    ]
}

Examples of /dump api

If you want a csv dump of our database run http://127.0.0.1:5000/dump in your browser