Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub #43

XiaoleZ · 2021-01-15T00:06:11Z

What it does?

Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub

Why it is import?

It makes the database setup more easier and it allows the user to run librarycomparisonswebsite locally

How to test it?

Follow the updated Readme.md file.

Docker Local Development Setup

This is a simple way to get a local librarycomparison web development server set up.

You'll need to have Docker and Docker Compose installed.

You first need to set up some of the configuration parameters in the file Config.json:
- Change the value of TOKEN to your own GitHub generated token. (How to create Github TOKEN)
- Change the value of SO_TOKEN to your stack exchange key. (How to create StackOverflow TOKEN)
You can update the MAXSIZE to 100 in Config.json for testing purpose.

Creating the image

1. Builds/Rebuilds the image (not start the containers) in the docker-compose.yml file:

docker-compose build --no-cache

2. Starts the containers

Starts the containers && Start the server

docker-compose up

Run metric script:

docker-compose run metric-script

createmetrics: Create the Metrics

(Optional) Open librarycomparisons website command shell:

docker-compose run --service-ports web

start: Starts the Django server. The librarycomparison web will run in the 8000 port by default.
migrate: Runs Django migrations
make: Runs Django makemigrations
createsuperuser: Runs Django createsuperuser

To access the website, use http://127.0.0.1:8000/comparelibraries/

3. Stops containers and removes containers, networks, volumes, and images created by up

docker-compose down

Remove volume: docker-compose down -v. Warning: this will permanently delete the contents in the db_data volume, wiping out any previous database you had there

4. Setup Metric Table if you create the docker volumn for the first time

docker exec  librarymetricscripts_db_1 /bin/sh -c  'mysql -uroot -p"mypwd"  libcomp  < docker-entrypoint-initdb.d/metric-setup.sql'

Accessing docker container mysql databases

docker exec -it MyContainer mysql -uroot -pMyPassword
eg: docker exec -it librarymetricscripts_db_1 mysql -uroot -p"mypwd"
Show MySQL Databases: show databases;

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| libcomp            |
| mysql              |
| performance_schema |
| sys                |
+--------------------+

Show MySQL Tables:

use libcomp;
show tables;

Show Table's schema

describe libcomp.Metric;

Show the values of Metric table

select * from libcomp.Metric;

…isonswebsite

snadi · 2021-01-29T23:38:25Z

updatemetrics.sh

-
-#pull latest repo changes
-git pull
+file_name="libcomp-bkp-${today}.sql"


can we at least put this in a DBBackups folder? Otherwise, it will be lots of files in the current directory

@XiaoleZ So I guess based on our conversation, you should revert back all changes related to updatemetrics.sh (i.e., it should be the original one before your changes since you won't be dealing with it)

docker/start.sh

snadi · 2021-02-03T21:43:06Z

hi @XiaoleZ As discussed, I've created a dockerhub organization https://hub.docker.com/orgs/ualbertasmr and added you as an owner. I've linked the ualberta-smr organization account to it. Can you see if you are able to push the container there now instead of using your personal account? Let me know if there are any issues!

XiaoleZ · 2021-02-03T21:44:33Z

@snadi Thank you. I will put the images in the org docker hub

XiaoleZ · 2021-02-12T15:41:58Z

scripts/Config.json

@@ -8,5 +8,6 @@
        "MAXSIZE": "1000",
        "POPULARITY_OUTPUT_FILE":"scripts/Popularity/popularity_results.txt",
        "TIME_SPAN":"365",
-        "SO_TOKEN":"enter your SO token"
+        "SO_TOKEN":"enter your SO token",
+        "OUTPUT_PATH": "scripts/"


When create the docker, set it to "OUTPUT_PATH": "../home/scripts/" The shared charts folder is in home/scripts/charts

scripts/IssueMetrics/performanceclassifier.py

snadi · 2021-03-03T22:05:00Z

Why are we changing output_path to ../home/scripts/ ?

snadi · 2021-03-03T22:18:20Z

I get this error when creating the metrics

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/main/scripts/__main__.py", line 4, in <module>
    app.run()
  File "/main/scripts/app.py", line 26, in run
    addlibraries()
  File "/main/scripts/addlibraries.py", line 27, in addlibraries
    if not domain.exists():
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 809, in exists
    return self.query.has_results(using=self.db)
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 535, in has_results
    return compiler.has_results()
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/usr/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/usr/local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
django.db.utils.ProgrammingError: (1146, "Table 'libcomp.Domain' doesn't exist")

I think that the domain tables haven't been created yet. So does this mean that the steps need to be done in reverse? That you need to run the migrate for the website first?

XiaoleZ · 2021-03-11T23:38:43Z

Update the PR: Tested for 2 library and display the charts successfully.

XiaoleZ · 2021-03-11T23:40:17Z

Why are we changing output_path to ../home/scripts/ ?

Because the charts is in ../home/scripts/ (not in the metricscript or web container). The views.py read the charts from ../home/scripts/ . So the charts can be display on the web.'

I move this change in the code. output_path = ../home/scripts/ is for the docker. If we want to run locally we need to change to the file we store the charts

…isonswebsite

XiaoleZ added 5 commits January 14, 2021 17:04

create docker-compose file for LibraryMetricScripts and librarycompar…

1afc1e1

…isonswebsite

update

e26a48b

enable the docker for combinding two repos

4042cb8

mysql problem to connect the host

12eb56c

run web successfully

45a1a64

XiaoleZ changed the title ~~WIP - Create Dockerfile for LibraryMetricScripts and librarycomparisonswebsite~~ Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub Jan 29, 2021

XiaoleZ added 2 commits January 29, 2021 01:35

update the readme

85a4a25

update the readme

b53c425

snadi reviewed Jan 29, 2021

View reviewed changes

XiaoleZ added 7 commits February 5, 2021 02:27

update readme

2c01a39

update readme

533beaf

revert the changes

8812cc5

revert the changes

fe02889

create the shared folder

fd925a7

create a shared folder in home/scripts/charts

b11ab27

update the app.py

0701eb9

XiaoleZ commented Feb 12, 2021

View reviewed changes

XiaoleZ added 5 commits February 25, 2021 19:55

add the readme

ddb6be1

update readme

2bc7e44

update readme

f78550b

update readme

0917ebc

add openJDK (java) in docker

f0d3db9

finish testing

bc081e6

XiaoleZ requested a review from snadi March 11, 2021 23:37

snadi mentioned this pull request Apr 20, 2021

Missing metrics in database #40

Open

This was linked to issues Apr 20, 2021

Missing metrics in database #40

Open

Add instructions to insert metric info in DB #39

Open

Other users will not be able to create the DB #27

Open

snadi and others added 26 commits May 20, 2021 11:00

Update libraries that changed repo urls

643a3b6

Address Issue #36 popularity calculation

181d84d

Add git stash first

09eb866

Fix thousands number format

d089e21

create docker-compose file for LibraryMetricScripts and librarycompar…

15a0f57

…isonswebsite

update

7d8dadf

enable the docker for combinding two repos

295fc4a

mysql problem to connect the host

b0e6835

run web successfully

af31859

update the readme

f4871cd

update the readme

68fb526

update readme

487d853

update readme

8a9f137

revert the changes

1e50747

revert the changes

ba276b9

create the shared folder

4db6569

create a shared folder in home/scripts/charts

fa24571

add the readme

a6d8608

update readme

f03cf2c

update readme

3ee8560

update readme

08cbb82

add openJDK (java) in docker

6d98a2e

finish testing

83388ce

udpate the readme

1f4d1d0

Sync with master

cb001b5

Add WIP notes for testing containers

6e55a04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub #43

Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub #43

XiaoleZ commented Jan 15, 2021 •

edited

Loading

snadi Jan 29, 2021

snadi Feb 5, 2021

snadi commented Feb 3, 2021

XiaoleZ commented Feb 3, 2021

XiaoleZ Feb 12, 2021

snadi commented Mar 3, 2021

snadi commented Mar 3, 2021

XiaoleZ commented Mar 11, 2021

XiaoleZ commented Mar 11, 2021 •

edited

Loading

Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub #43

Are you sure you want to change the base?

Create Dockerfile for LibraryMetricScripts and Pull librarycomparisonswebsite images from Docker Hub #43

Conversation

XiaoleZ commented Jan 15, 2021 • edited Loading

What it does?

Why it is import?

How to test it?

Docker Local Development Setup

Creating the image

1. Builds/Rebuilds the image (not start the containers) in the docker-compose.yml file:

2. Starts the containers

3. Stops containers and removes containers, networks, volumes, and images created by up

4. Setup Metric Table if you create the docker volumn for the first time

Accessing docker container mysql databases

snadi Jan 29, 2021

Choose a reason for hiding this comment

snadi Feb 5, 2021

Choose a reason for hiding this comment

snadi commented Feb 3, 2021

XiaoleZ commented Feb 3, 2021

XiaoleZ Feb 12, 2021

Choose a reason for hiding this comment

snadi commented Mar 3, 2021

snadi commented Mar 3, 2021

XiaoleZ commented Mar 11, 2021

XiaoleZ commented Mar 11, 2021 • edited Loading

XiaoleZ commented Jan 15, 2021 •

edited

Loading

XiaoleZ commented Mar 11, 2021 •

edited

Loading