Main commit

LICENSE.md

@@ -0,0 +1,17 @@
+Copyright © 2021 Fabio Schick
+
+This software is provided 'as-is', without any express or implied
+warranty. In no event will the authors be held liable for any damages
+arising from the use of this software.
+
+Permission is granted to anyone to use this software for any purpose,
+including commercial applications, and to alter it and redistribute it
+freely, subject to the following restrictions:
+
+1. The origin of this software must not be misrepresented; you must not
+   claim that you wrote the original software. If you use this software
+   in a product, an acknowledgment in the product documentation would be
+   appreciated but is not required.
+2. Altered source versions must be plainly marked as such, and must not be
+   misrepresented as being the original software.
+3. This notice may not be removed or altered from any source distribution.

README.md

@@ -1,2 +1,49 @@
 # igc2acmi
-a little script that converts XCSoar's and FLARM's ".igc" flight logs into Tacview's ".acmi" format.
+A set of tools to convert [XCSoar](https://github.com/XCSoar/XCSoar#readme)'s, [FLARM](https://flarm.com/)'s and multiple other vendors' `.igc` GPS flight logs into [Tacview](https://www.tacview.net/)'s `.acmi` format.
+
+## The Goal
+`.igc` is simple format, useful for recording the flight of a single aircraft, but has its limits.
+
+`.klm`, which can be parsed from `.igc` using other [online utilities](http://cunimb.net/igc2kml.php), may be more visually appealing when viewed in Google Earth, but still isn't as useful for flight debriefing as I'd like to.
+
+For these reasons, I decided to develop igc2acmi, it converts one or more `.igc` logs into [Tacview](https://www.tacview.net/)'s `.acmi` flight recording format, which is specifically designed to efficiently store multiple GPS tracks, its software also provides very useful debriefing features such as:
+- customizable labels to display AMSL/AGL Altitude, Vertical Speed, Calibrated Air Speed, True Airspeed, etc.
+- dynamic distance indication between 2 aircraft
+- custom [terrain textures](https://www.tacview.net/documentation/terrain/en/) and [3D shapes](https://www.tacview.net/documentation/staticobjects/en/), which can be used to display aviation charts and airspace models:
+![Tacview flight with custom terrain and airspace shapes](http://fabioschick.altervista.org/img/igc2acmi-Tacview-TerrainAirspace.png)
+- online debriefs, where multiple people can observe a flight from a shared perspective
+
+## The Programs
+the `ProgramName.exe` `-h`/`--help` command will explain the different command-line arguments of each program.
+
+### igc2acmi
+converts a single `.igc` file into `.acmi`, useful when you want to convert a specific file using specific settings.
+
+### convert-all-igcs
+converts all `.igc` files in the program's directory (or a given input path) into an equal number of `.acmi` files. Useful when you need to convert multiple flights at once, all with the same settings.
+
+### combine-igcs
+arguably the most complex utility of the 3, it finds all `.igc` files in the program's directory (or a given input path), sorts them by date of flight and combines all flights started on the same date into a single `.acmi` file, the timeline starts with the first GPS fix of the first flight and ends with the last fix of the last one.
+
+Aeroclubs may find the greatest utility in this feature, for example by collecting everyone's `.igc` (with pilot consent of course) every day and combining them, to store/archive flights for collective debriefs and/or longer-term analysis of pilot activity.
+
+## Known limitations
+Until now, I have only tested my project on `.igc` logs recorded by XCSoar (my own) and FLARM (a collegue's). All of the FLARM logs had an invalid date field of "000000" in their header, which the programs are not capable of handling (yet).
+
+Maybe that has been resolved by newer FLARM versions, or was caused by incorrect setup of the FLARM module. In the meantime, make sure your `.igc` files have a valid date in their "HFDTE" line, using the "DDMMYY" format (i.e: "HFDTE221121" for 22nd November 2021).
+
+## Sources and How to Contribute
+Development of this project was significantly aided by available documentation for the [igc](https://xp-soaring.github.io/igc_file_format/igc_format_2008.html) and [acmi](https://www.tacview.net/documentation/acmi/en/) file formats, many thanks to their authors.
+
+The [released executables](https://github.com/RUSHER0600/igc2acmi/releases) are compiled with [pyinstaller](https://www.pyinstaller.org/). I use a batch file at the project root with 3 lines like this (last argument differs) to build all programs:
+```winbatch
+pyinstaller --specpath "build" --distpath "build\dist" -p "." --clean -F -i NONE --exclude-module _bootlocale "igc2acmi\igc2acmi.py"
+```
+
+Feel free to fork this project for your own uses, any Pull Requests back to the original repository would be greatly appreciated!
+
+## To-Do List
+- [ ] Add `--version` command that displays version number (and maybe release date)
+- [ ] Implement parsing of `igc` [tasks](https://xp-soaring.github.io/igc_file_format/igc_format_2008.html#link_3.6) and [events](https://xp-soaring.github.io/igc_file_format/igc_format_2008.html#link_4.2) into `acmi` [events](https://www.tacview.net/documentation/acmi/en/#Events)
+- [ ] Improve and augment error handling, i.e: for flights with missing/invalid date fields in their header
+- [ ] Add a way for the user to determine the output file's name or name template

combine-igcs/combine-igcs.py

@@ -0,0 +1,185 @@
+import sys, argparse, pathlib as pl, traceback as tb, datetime as dt
+sys.path.append(str(pl.Path(__file__).resolve().parents[1]))
+from common import functions as fnc, crashdumping as dump, args as a, console as c
+
+def listIGCsByDate(path, igcs): # builds a dict of igc files, the key is the date of flight, the value is a list containing all igc filepaths recorded on that date
+	ret = {}
+	for igc in igcs:
+		date = fnc.getFlightDate(path+igc+".igc")
+		if date in ret:
+			ret[date].append(path+igc+".igc")
+		else:
+			ret[date] = [path+igc+".igc"]
+	return ret
+
+def getLowestRefTime(igcs): # from a list of igc filepaths, find the one with the lowest first fix time and return that time as a time object
+	lowest = dt.time(23, 59, 59)
+
+	for igcpath in igcs:
+		with open(igcpath) as igc:
+			for line in igc:
+				if line.startswith("B"):
+					fixTime = dt.time(
+						int(line[1:3]),
+						int(line[3:5]),
+						int(line[5:7])
+					)
+					break
+			if fixTime < lowest:
+				lowest = fixTime
+
+	return lowest
+
+def getLastFix(acs): # returns the highest occuring "lastfix" value among all aircraft dicts
+	lastfix = 0
+	for id in acs:
+		if acs[id]["lastfix"] > lastfix:
+			lastfix = acs[id]["lastfix"]
+
+	return lastfix
+
+def getFixes(filename, refTime): # returns a list of fix dictionaries: key is the time offset from reference time in seconds, values are lat/lon/alt
+	ret = {}
+	with open(filename) as igc:
+		for line in igc:
+			if line.startswith("B"):
+				fix = fnc.parseFixLocation(line)
+				fixTime = fnc.parseFixTime(line)
+				ret[fnc.timeDiff(refTime, fixTime)] = {
+					"lat": fix[0],
+					"lon": fix[1],
+					"alt": fix[2]
+				}
+	return ret
+
+def buildAircraft(igcs, refTime): # builds the main data dict of aircraft, with one element for each IGC file read.
+	i = 0
+	ac = {}
+
+	for igcPath in igcs:
+		i += 1
+		fixes = getFixes(igcPath, refTime)
+		ac[1000+i] = {
+			"header": fnc.getFlightHeader(igcPath),
+			"fixes": fixes,
+			"lastfix": list(fixes.keys())[-1]
+		}
+
+	return ac
+
+
+def main():
+	parser = argparse.ArgumentParser(
+		"combine-igcs.exe",
+		description = "Use it to combine multiple .igc files recorded on the same day into a single .acmi, containing all aircraft"
+	)
+	# import common arguments
+	parser = a.addDefaultArgs(parser)
+	# set program-specific arguments
+	parser.add_argument(
+		"-i",
+		"--input",
+		type = str,
+		default = "",
+		help = "Input path for igc files"
+	)
+	parser.add_argument(
+		"-n",
+		"--name",
+		type = str,
+		default = "",
+		help = "Desired filename for ACMI file, will only be applied to the first file to be handled"
+	)
+	# parse all arguments
+	args = parser.parse_args()
+
+	try:
+		# define input path
+		inPath = a.parseArgPath(args.input, "Input")
+		# define output path
+		outPath = a.parseArgPath(args.output, "Output")
+
+		# list available .igc files by their date and parse an acmi for each day
+		igcs = listIGCsByDate(inPath, fnc.listIGCs(inPath))
+		if None in igcs: # if .igc files with no valid date header line have been found, list them as corrupted and remove them from the dictionary
+			c.warn("The following files have no valid date header line and will not be converted:")
+			for file in igcs[None]:
+				print("\""+file+"\"")
+			igcs.pop(None)
+
+		for date in igcs:
+			try:
+				# initialize data structure of flights on x date
+				refTime = dt.datetime.combine(
+					dt.datetime.strptime(str(date), "%y%m%d"),
+					getLowestRefTime(igcs[date])
+				)
+				c.info("Handling flights of date: "+refTime.strftime("%d %b %Y"))
+				acs = buildAircraft(igcs[date], refTime.time())
+
+				# default to ACMI name "YYYY-MM-DD_Callsign1_Callsign2_..." if filename argument is not set or already taken
+				if args.name != "" and not sorted(pl.Path(args.output).glob(args.name+".*")):
+					acmiName = args.name
+				else:
+					acmiName = refTime.strftime("%Y-%m-%d")
+					for id in acs:
+						if acs[id]["header"]["callsign"] not in acmiName:
+							acmiName += "_"+acs[id]["header"]["callsign"]
+
+				# init ACMI file for date x
+				acmi = open(outPath+acmiName+".tmpacmi", "w")
+				acmi.write(fnc.acmiFileInit(refTime))
+
+				# init Aircraft objects on ACMI
+				for id in acs:
+					acmi.write(fnc.acmiObjInit(acs[id]["header"], id))
+
+				# write Aircraft fix positions to ACMI, grouped by time offset to refTime
+				i = 0
+				while i <= getLastFix(acs):
+					rec = ""
+					for id in acs:
+						if i in acs[id]["fixes"]:
+							fix = acs[id]["fixes"][i]
+							rec += str(id)+",T="+fix["lon"]+"|"+fix["lat"]+"|"+fix["alt"]+"\n"
+							if i == acs[id]["lastfix"]: # if current time offset is last fix of aircraft, pass ACMI object deletion line to record
+								rec += "-"+str(id)+"\n"
+					if rec != "":
+						acmi.write("#"+str(i)+"\n"+rec)
+					i += 1
+
+				acmi.close()
+
+				# handle other optional arguments
+				if args.remove:
+					for igc in igcs[date]:
+						pl.Path(igc).unlink()
+						c.info("Removed "+pl.Path(igc).name)
+				if args.nozip:
+					ext = ".txt.acmi"
+					pl.Path(outPath+acmiName+".tmpacmi").replace(outPath+acmiName+ext)
+				else:
+					ext = ".zip.acmi"
+					fnc.zipAcmi(outPath+acmiName)
+
+				c.info("Combined the following files into \""+outPath+acmiName+ext+"\":")
+				for igc in igcs[date]:
+					print("\""+igc+"\"")
+
+			except:
+				if "acmi" in locals(): # clean up initialized ".tmpacmi" file in case of a crash
+					if not acmi.closed: # if tmpacmi is still open, close it
+						acmi.close()
+					pl.Path(outPath+acmiName+".tmpacmi").unlink()
+				if args.debug: # if debug flag is set, print normal error message
+					c.err("Unable to combine flights of date "+refTime.strftime("%d %b %Y")+", error:\n"+tb.format_exc())
+				else: # default crashdump writing
+					c.err("Unable to convert flights of date "+refTime.strftime("%d %b %Y")+", a crashlog for it has been saved in \""+dump.fileDump(acmiName, "combine-igcs")+"\"")
+
+	except:
+		if args.debug: # if debug flag is set, print normal error message
+			c.err(tb.format_exc())
+		else: # default crashdump writing
+			c.err("An error occured in this program's execution, a crash report has been saved to \""+dump.progDump(args, "combine-igcs")+"\"")
+
+main()

common/args.py

@@ -0,0 +1,43 @@
+import pathlib as pl
+from common import console as c
+
+# common arguments that every program uses
+def addDefaultArgs(parser):
+	parser.add_argument(
+		"-r",
+		"--remove",
+		action = "store_true",
+		help = 'Remove ".igc" files after converting them'
+	)
+	parser.add_argument(
+		"-z",
+		"--nozip",
+		action = "store_true",
+		help = 'Do not compress output files as ".zip.acmi", instead output ".txt.acmi", useful for debugging'
+	)
+	parser.add_argument(
+		"-d",
+		"--debug",
+		action = "store_true",
+		help = "Do not zip crashdumps, instead output error messages normally, useful for debugging"
+	)
+	parser.add_argument(
+		"-o",
+		"--output",
+		type = str,
+		default = "",
+		help = "Output path for acmi file(s)"
+	)
+
+	return parser
+
+def parseArgPath(path, pathName="Argument"): # parses paths received from cmd arguments, checks if they are valid and properly formats them
+	p = pl.Path(path).resolve()
+	if p.is_file():
+		return p.parent.as_posix()+"/"
+	elif p.is_dir():
+		return p.as_posix()+"/"
+	else:
+		# if path is not valid, set it to script's location and notify user
+		c.warn(pathName+" path is not valid, defaulting to current program directory")
+		return pl.Path(__name__).resolve().parent.as_posix()+"/"

common/console.py

@@ -0,0 +1,12 @@
+import colorama as cr
+
+cr.init()
+
+def info(msg):
+	print(cr.Fore.BLUE+"[INFO] "+cr.Style.RESET_ALL+msg)
+
+def warn(msg):
+	print(cr.Fore.YELLOW+"[WARN] "+cr.Style.RESET_ALL+msg)
+
+def err(msg):
+	print(cr.Fore.RED+"[ERROR] "+cr.Style.RESET_ALL+msg)

common/crashdumping.py

@@ -0,0 +1,26 @@
+import traceback as tb
+import datetime as dt, zipfile as zf, pprint as pp
+
+# write a program crashdump
+def progDump(args, prog, path=""):
+	arcName = dt.datetime.now().strftime(prog+"_Crashdump_%Y-%m-%d_%H-%M-%S")
+	body = "ERROR LOG "+dt.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+	body += "\n\nARGS:\n"+pp.pformat(args)
+	body += "\n\nERROR TRACEBACK:\n"+tb.format_exc()
+
+	with zf.ZipFile(path+arcName+".zip", "a", zf.ZIP_DEFLATED) as zlog:
+		zlog.writestr(arcName+".log", body)
+
+	return arcName+".zip"
+
+# write the crashdump for an exception occured while handling a specific file
+def fileDump(file, prog, path=""):
+	arcName = dt.datetime.now().strftime(prog+"_Crashdump_%Y-%m-%d_%H-%M-%S")
+	fname = file+".igc_Exception.log"
+	body = "Handled File Name:\n"+file+".igc\n\n"
+	body += "Error:\n"+tb.format_exc()
+
+	with zf.ZipFile(path+arcName+".zip", "a", zf.ZIP_DEFLATED) as zlog:
+		zlog.writestr(fname, body)
+
+	return arcName+".zip"