Analyze command - Improvements in error handling and logging (#34)

* Modest improvement in error reporting and Archive handling

Check for Archives rather than trying to mount every file just in case it turns out to be an archive.
Unity Archives have a distinctive signature string at the start so we can pre-triage which files are actually Unity Archives.
The expected values were confirmed by studying the Unity parsing code and debugging the load of some files in the editor.

Output improvement - when erasing progress message move back to the start of the line so that errors are not printed at a random indent.

This is only a modest improvement because we still don't have some mechanism to predict if a file is a Unity Serialized File without attempting to load it and then failing.

* Analyze - summarize number of successful, ignored and failed files

Previously it was hard to tell if errors were shown whether anything actually had been analyzed.
Partial imports are normal until we find a way to reliably detect serialized files.

* Document more expected errors from Analyze

Mention errors that migth be visible (based on testing a folder with AssetBundles from different builds all being mixed together)

Author: SkowronskiAndrew

Analyzer/AnalyzerTool.cs

@@ -41,28 +41,35 @@ public int Analyze(
             searchPattern,
             noRecursion ? SearchOption.TopDirectoryOnly : SearchOption.AllDirectories);
 
+        int countFailures = 0;
+        int countSuccess = 0;
+        int countIgnored = 0;
         int i = 1;
         foreach (var file in files)
         {
             if (ShouldIgnoreFile(file))
             {
-                var relativePath = Path.GetRelativePath(path, file);
-
                 if (m_Verbose)
                 {
+                    var relativePath = Path.GetRelativePath(path, file);
                     Console.WriteLine();
                     Console.WriteLine($"Ignoring {relativePath}");
                 }
-                ++i;
-                continue;
+                countIgnored++;
+            }
+            else if (!ProcessFile(file, path, writer, i, files.Length))
+            {
+                countFailures++;
+            }
+            else
+            {
+                countSuccess++;
             }
-
-            ProcessFile(file, path, writer, i, files.Length);
             ++i;
         }
 
         Console.WriteLine();
-        Console.WriteLine("Finalizing database...");
+        Console.WriteLine($"Finalizing database. Successfully processed files: {countSuccess}, Failed files: {countFailures}, Ignored files: {countIgnored}");
 
         writer.End();
 
@@ -98,69 +105,83 @@ bool ShouldIgnoreFile(string file)
 
     private static readonly HashSet<string> IgnoredExtensions = new()
     {
-        ".txt", ".resS", ".resource", ".json", ".dll", ".pdb", ".exe", ".manifest", ".entities", ".entityheader"
+        ".txt", ".resS", ".resource", ".json", ".dll", ".pdb", ".exe", ".manifest", ".entities", ".entityheader",
+        ".ini", ".config"
     };
 
-    void ProcessFile(string file, string rootDirectory, SQLiteWriter writer, int fileIndex, int cntFiles)
+    bool ProcessFile(string file, string rootDirectory, SQLiteWriter writer, int fileIndex, int cntFiles)
     {
+        bool successful = true;
         try
         {
-            UnityArchive archive = null;
-
-            try
+            if (IsUnityArchive(file))
             {
-                archive = UnityFileSystem.MountArchive(file, "archive:" + Path.DirectorySeparatorChar);
-            }
-            catch (NotSupportedException)
-            {
-                // It wasn't an AssetBundle, try to open the file as a SerializedFile.
-
-                var relativePath = Path.GetRelativePath(rootDirectory, file);
-                writer.WriteSerializedFile(relativePath, file, Path.GetDirectoryName(file));
-
-                ReportProgress(relativePath, fileIndex, cntFiles);
-            }
-
-            if (archive != null)
-            {
-                try
+                using (UnityArchive archive = UnityFileSystem.MountArchive(file, "archive:" + Path.DirectorySeparatorChar))
                 {
-                    var assetBundleName = Path.GetRelativePath(rootDirectory, file);
-
-                    writer.BeginAssetBundle(assetBundleName, new FileInfo(file).Length);
-                    ReportProgress(assetBundleName, fileIndex, cntFiles);
+                    if (archive == null)
+                        throw new FileLoadException($"Failed to mount archive: {file}");
 
-                    foreach (var node in archive.Nodes)
+                    try
                     {
-                        if (node.Flags.HasFlag(ArchiveNodeFlags.SerializedFile))
+                        var assetBundleName = Path.GetRelativePath(rootDirectory, file);
+
+                        writer.BeginAssetBundle(assetBundleName, new FileInfo(file).Length);
+                        ReportProgress(assetBundleName, fileIndex, cntFiles);
+
+                        foreach (var node in archive.Nodes)
                         {
-                            try
+                            if (node.Flags.HasFlag(ArchiveNodeFlags.SerializedFile))
                             {
-                                writer.WriteSerializedFile(node.Path, "archive:/" + node.Path, Path.GetDirectoryName(file));
-                            }
-                            catch (Exception e)
-                            {
-                                EraseProgressLine();
-                                Console.Error.WriteLine($"Error processing {node.Path} in archive {file}");
-                                Console.Error.WriteLine(e);
-                                Console.WriteLine();
+                                try
+                                {
+                                    writer.WriteSerializedFile(node.Path, "archive:/" + node.Path, Path.GetDirectoryName(file));
+                                }
+                                catch (Exception e)
+                                {
+                                    // the most likely exception here is Microsoft.Data.Sqlite.SqliteException,
+                                    // for example 'UNIQUE constraint failed: serialized_files.id'.
+                                    // or 'UNIQUE constraint failed: objects.id' which can happen
+                                    // if AssetBundles from different builds are being processed by a single call to Analyze
+                                    // or if there is a Unity Data Tool bug.
+                                    EraseProgressLine();
+                                    Console.Error.WriteLine($"Error processing {node.Path} in archive {file}");
+                                    Console.Error.WriteLine(e.Message);
+                                    Console.WriteLine();
+
+                                    // It is possible some files inside an archive will pass and others will fail, to have a partial analyze.
+                                    // Overall that is reported as a failure
+                                    successful = false;
+                                }
                             }
                         }
                     }
-                }
-                finally
-                {
-                    writer.EndAssetBundle();
-                    archive.Dispose();
+                    finally
+                    {
+                        writer.EndAssetBundle();
+                    }
                 }
             }
+            else
+            {
+                // This isn't a Unity Archive file.  Try to open it as a SerializedFile.
+                // Unfortunately there is no standard file extension, or clear signature at the start of the file,
+                // to test if it truly is a SerializedFile.  So this will process files that are clearly not unity build files,
+                // and there is a chance for crashes and freezes if the parser misinterprets the file content.
+                var relativePath = Path.GetRelativePath(rootDirectory, file);
+                writer.WriteSerializedFile(relativePath, file, Path.GetDirectoryName(file));
+
+                ReportProgress(relativePath, fileIndex, cntFiles);
+            }
+
             EraseProgressLine();
         }
         catch (NotSupportedException)
         {
             EraseProgressLine();
             Console.Error.WriteLine();
             //A "failed to load" error will already be logged by the UnityFileSystem library
+
+            successful = false;
         }
         catch (Exception e)
         {
@@ -170,9 +191,47 @@ void ProcessFile(string file, string rootDirectory, SQLiteWriter writer, int fil
             Console.WriteLine($"{e.GetType()}: {e.Message}");
             if (m_Verbose)
                 Console.WriteLine(e.StackTrace);
+
+            successful = false;
+        }
+
+        return successful;
+    }
+
+    private static bool IsUnityArchive(string filePath)
+    {
+        // Check whether a file is a Unity Archive (AssetBundle) by looking for known signatures at the start of the file.
+        // "UnifyFS" is the current signature, but some older formats of the file are still supported
+        string[] signatures = { "UnityFS", "UnityWeb", "UnityRaw", "UnityArchive" };
+        int maxLen = 12; // "UnityArchive".Length
+        byte[] buffer = new byte[maxLen];
+
+        using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
+        {
+            int read = fs.Read(buffer, 0, buffer.Length);
+            foreach (var sig in signatures)
+            {
+                if (read >= sig.Length)
+                {
+                    bool match = true;
+                    for (int i = 0; i < sig.Length; ++i)
+                    {
+                        if (buffer[i] != sig[i])
+                        {
+                            match = false;
+                            break;
+                        }
+                    }
+                    if (match)
+                        return true;
+                }
+            }
+            return false;
         }
     }
 
+
+
     int m_LastProgressMessageLength = 0;
 
     void ReportProgress(string relativePath, int fileIndex, int cntFiles)
@@ -195,7 +254,7 @@ void ReportProgress(string relativePath, int fileIndex, int cntFiles)
     void EraseProgressLine()
     {
         if (!m_Verbose)
-            Console.Write($"\r{new string(' ', m_LastProgressMessageLength)}");
+            Console.Write($"\r{new string(' ', m_LastProgressMessageLength)}\r");
         else
             Console.WriteLine();
     }

UnityDataTool/README.md

@@ -62,14 +62,6 @@ Example: `UnityDataTool analyze /path/to/asset/bundles -o my_database.db -p "*.b
 **Refer to this [documentation](../Analyzer/README.md#How-to-use-the-database) for more information
 about the output database structure.**
 
-Note: If a SerializedFile is built without TypeTrees, then the command will not be able to extract information about the contained objects.  It will print an error similar to this example, then skip to the next file:
-
-```
-Error processing file: C:\Src\TestProject\Build\Player\TestProject_Data\level0
-System.ArgumentException: Invalid object id.
-```
-
-See [this topic](../Documentation/unity-content-format.md) for more information about TypeTrees.
 
 ### Example Input to the Analyze command
 
@@ -104,6 +96,44 @@ For Player builds there is no single -p option that can catch all SerializedFile
 
 The `--no-recurse` option can reduce the volume of these warnings.
 
+
+### Errors when TypeTrees are missing
+
+If a SerializedFile is built without TypeTrees, then the Analyze command will not be able to extract information about the contained objects.  It will print an error similar to this example, then skip to the next file:
+
+```
+Error processing file: C:\Src\TestProject\Build\Player\TestProject_Data\level0
+System.ArgumentException: Invalid object id.
+```
+
+See [this topic](../Documentation/unity-content-format.md) for more information about TypeTrees.
+
+### SQL Errors
+
+The following SQL errors may occur when running the analyze command:
+
+```
+SQLite Error 19: 'UNIQUE constraint failed: objects.id'
+```
+
+or
+
+```
+SQLite Error 19: 'UNIQUE constraint failed: serialized_files.id'.
+```
+
+The likely cause of these errors is the same serialized file name appearing in more than Player or AssetBundle file that is processed by Analyze.
+
+This may occur:
+
+* If you analyze files from more than one version of the same build (e.g. if you run it on a directory that contains two different builds of the same project in separate sub-directories).
+* If two scenes with the same filename (but different paths) are included in a build.
+* In a build that used AssetBundle variants.
+
+The conflicting name makes it impossible to uniquely identify the serialized file and its object in the database, and makes it ambiguous how to interpret dependencies from one file to another.
+
+The [comparing builds](../Documentation/comparing-builds.md) topic gives some ideas about how to run Analyze more than once if you want to compare two different versions of the same build.
+
 ## dump
 
 This command dumps the contents of a SerializedFile into a file of the selected format. It currently