Skip to content
This repository has been archived by the owner on Mar 1, 2021. It is now read-only.

Performance measurement suite #73

Merged
merged 27 commits into from
Dec 20, 2016
Merged

Performance measurement suite #73

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
eb4ed17
initial work based on graphhopper Measurement.java
kodonnell Oct 5, 2016
8a587be
getting multiple commits working ...
kodonnell Oct 5, 2016
d60985b
getting multiple commits working ...
kodonnell Oct 5, 2016
0b34d2d
ok, seems to be working ...
kodonnell Oct 5, 2016
ee586c4
fixed merge while rebasing
kodonnell Oct 14, 2016
77818af
tidying up
kodonnell Oct 5, 2016
727ad77
formatting msg
kodonnell Oct 5, 2016
b08edf8
formatting code
kodonnell Oct 5, 2016
0373ee5
rats, committed on the wrong branch ...
kodonnell Oct 5, 2016
48cd988
tab to space ...
kodonnell Oct 5, 2016
6a62784
merging master
kodonnell Oct 15, 2016
696820c
updated to new API change, plus fixed test bug
kodonnell Oct 15, 2016
3f3829a
fixed infinite loop
kodonnell Oct 15, 2016
d540a14
OK, perf tests working correctly ... hopefully
kodonnell Oct 15, 2016
e0bd9ad
merging
kodonnell Dec 3, 2016
7cb6478
removed separate tools module and set name same as GH
kodonnell Dec 4, 2016
7bb0ad8
tidy file name and bug fix
kodonnell Dec 4, 2016
1b28b04
commit header for each col
kodonnell Dec 4, 2016
f83558d
start with oldest commit first
kodonnell Dec 4, 2016
173bed8
playing with graphs
kodonnell Dec 4, 2016
5441639
show charts one by one
kodonnell Dec 4, 2016
2a990e1
show charts one by one
kodonnell Dec 4, 2016
758529c
Merge branch 'issue_68' of github.com:kodonnell/map-matching into iss…
kodonnell Dec 4, 2016
c38265d
woops, fixing trap
kodonnell Dec 4, 2016
38790e4
clear charts
kodonnell Dec 4, 2016
028add7
removing gnuplots
kodonnell Dec 6, 2016
9d78657
tidied up doco a bit
kodonnell Dec 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ nbactions.xml
.project
matching-vaadin/src/main/webapp/
matching-core/build.xml
/map-data/**/*
bin/
measurement*
119 changes: 84 additions & 35 deletions map-matching.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,51 +1,100 @@
#!/bin/bash

# bail if any errors ...
set -e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strict option might be useful for graphhopper.sh too


if [ "$JAVA_HOME" != "" ]; then
JAVA=$JAVA_HOME/bin/java
JAVA=$JAVA_HOME/bin/java
fi

if [ "$JAVA" = "" ]; then
JAVA=java
JAVA=java
fi

echo "using $JAVA"

function set_jar {
local module=$1
local pattern="matching-$module/target/graphhopper-map-matching-*-dependencies.jar"
if ! ls $pattern > /dev/null 2>&1; then
mvn --projects hmm-lib -DskipTests=true install
mvn --projects matching-$module,matching-core -DskipTests=true install assembly:single
fi
JAR=$(ls matching-$module/target/graphhopper-map-matching-*-dependencies.jar)
}

if [ "$1" = "action=start-server" ]; then
function set_jar_path {
JAR=$(ls matching-web/target/graphhopper-map-matching-web-*-dependencies.jar)
}

set_jar_path

if [ ! -f "$JAR" ]; then
mvn --projects hmm-lib -DskipTests=true install
mvn --projects matching-web,matching-core -DskipTests=true install assembly:single
set_jar_path
fi

ARGS="graph.location=./graph-cache jetty.resourcebase=matching-web/src/main/webapp"

set_jar "web"
ARGS="graph.location=./graph-cache jetty.resourcebase=matching-web/src/main/webapp"
elif [ "$1" = "action=test" ]; then
export MAVEN_OPTS="-Xmx400m -Xms400m"
mvn clean test verify
# return exit code of mvn
exit $?
elif [ "$1" = "action=measurement" ]; then
set_jar "tools"
fname="measurement.$(date +%Y%m%d_%H%M%S)"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we need the same naming as in gh core

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

ARGS="config=$CONFIG graph.location=$GRAPH datareader.file=$2 prepare.ch.weightings=fastest graph.flag_encoders=car prepare.min_network_size=10000 prepare.min_oneway_network_size=10000"
current_commit=$(git log -n 1 --pretty=oneline | cut -d' ' -f1)
function startMeasurement {
echo -e "\nperforming measurement for commit $current_commit"
echo $measurement_fname
echo "$JAVA" $JAVA_OPTS -cp "$JAR" com.graphhopper.matching.tools.Measurement $ARGS measurement.location="$measurement_fname"
"$JAVA" $JAVA_OPTS -cp "$JAR" com.graphhopper.matching.tools.Measurement $ARGS measurement.location="$measurement_fname"
}

export MAVEN_OPTS="-Xmx400m -Xms400m"
mvn clean test verify
# return exit code of mvn
exit $?

# use all <last_commits> versions starting from HEAD
last_commits=$3
if [ -z "$last_commits" ]; then
measurement_fname=$fname
startMeasurement
else
# the following checks out the last commits, and tests each one. The code looks a bit
# messier than that, as we merge the results into a single file (to make it easier to
# compare.
echo -e "commits (in order tested):\n--------------------------\n" >> "$fname"
values=${fname}.values
measurement_fname=${fname}.tmp
commits=$(git rev-list HEAD -n "$last_commits")
first=true
empty_pad=""
for commit in $commits; do
git checkout "$commit"
git log -n 1 --pretty=oneline >> "$fname"
mvn --projects matching-tools -DskipTests=true clean install assembly:single
rm -f "$measurement_fname"
startMeasurement
while read -r line; do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does this loop through?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my way of merging results across multiple commits into a single file. Outputs are written temporarily to $measurement_fname (see line 43 and 57) and read in line by line (see line 78). Then 66-77 is logic for stripping the key and value from each line, and

  • adding a new line if it's a new measurement key
  • adding the value to the next column if it's an existing key

This is taken care of in the $values file. My advice: try it and see = )

Copy link
Member

@karussell karussell Nov 14, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My advice: try it and see = )

Will definitely do this... before: why would I want to have multiple results in one file? If I have one file per commit I know exactly what is what and can give a computer those files as input (already have an bit hidden UI here - wait a bit and then click e.g. routing)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See first PR comment - I find it much easier to compare performance across multiple commits when all the values are side-by-side, etc. The previous approach meant you had to open e.g. 10 files, and manually eyeball through until you found the right value, then memorise it and compare against all others, etc. If you wanted to compare across multiple metrics, it was even worse. Of course, if you're making a separate UI, there's not much need.

It's very easy to keep the original raw files per commit, if you're that way inclined (I just delete them, i.e. line 85).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You convinced me to invest a bit time to make the UI working easily - wasn't easy to get it working locally but with nodejs this was rather simple:

graphhopper/graphhopper#894

Would you mind to make this working with this UI?

key=${line%%=*}
value=$(printf "%-20s" "${line##*=}")
if [ "$first" = true ] ; then
printf "%-30s%s\n" "$key" "$value" >> "$values"
else
if grep "$key" "$values"; then
sed -ri "s/($key.*)/\1$value/g" "$values"
else
# add a new row, using $empty_pad to get the column in the right place
printf "%-30s%s%s\n" "$key" "$empty_pad" "$value" >> "$values"
fi
fi
done < "$measurement_fname"
first=false
empty_pad=$(printf "%s%-20s" "$empty_pad" "")
done
echo -e "\nmeasurements:\n-------------\n" >> "$fname"
cat "$values" >> "$fname"
rm "$values"
rm "$measurement_fname"
# revert checkout
git checkout "$current_commit"
fi
# in either case, echo the file:
echo ""
cat "$fname"
exit 0
else
function set_jar_path {
JAR=$(ls matching-core/target/graphhopper-map-matching-*-dependencies.jar)
}

set_jar_path

if [ ! -f "$JAR" ]; then
mvn --projects hmm-lib -DskipTests=true install
mvn --projects matching-core -DskipTests=true install assembly:single
set_jar_path
fi

ARGS="$@"
set_jar "core"
ARGS="$@"
fi

exec "$JAVA" $JAVA_OPTS -jar $JAR $ARGS prepare.min_network_size=0 prepare.min_one_way_network_size=0
exec "$JAVA" $JAVA_OPTS -jar "$JAR" $ARGS prepare.min_network_size=0 prepare.min_one_way_network_size=0
46 changes: 46 additions & 0 deletions matching-tools/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<name>GraphHopper Map Matching Tools</name>
<groupId>com.graphhopper</groupId>
<artifactId>graphhopper-map-matching-tools</artifactId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you wanted to make it identical to the GH core modules. But is a separate module necessary just for one class?

My intend for the separate tools module in GH was to separate swing usages from core and other deps

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you wanted to make it identical to the GH core modules

I think I was just copying what was there. More than happy to move it. Let me know if it's in the wrong place.

<version>0.8-SNAPSHOT</version>
<parent>
<groupId>com.graphhopper</groupId>
<artifactId>graphhopper-map-matching-parent</artifactId>
<version>0.8-SNAPSHOT</version>
</parent>
<dependencies>
<dependency>
<groupId>com.graphhopper</groupId>
<artifactId>graphhopper-map-matching-core</artifactId>
<version>${project.parent.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.5.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.graphhopper.matching.tools.Measurement</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
</plugin>
</plugins>
</build>
</project>
Loading