Custom folder patterns for library scan (grok-js)

[sandreas]

Hello @advplyr ,

as you may know, I'm the author of m4b-tool and tone and I really love audiobookshelf.

The recommended directory structure implies having a directory for each audio book, e.g.:

# series
/<author>/<series>/Vol <part-number> - <title> {<narrator>}/

# single titles
/<author>/<title>/

Currently, I'm struggling getting the series to be recognized with my existing directory structure, which is the following:

# series
/<author>/<series>/<part-number> - <title>.m4b

# single titles
/<author>/<title>/<title>.m4b

This results in audiobookshelf stacking my parts of a series into a single audio book with multiple files. It is mainly because the current metadata extraction does not support movement-name (series) and movement (series-part) for m4b files (ffmpeg really sucks for mp4 metadata extraction). Sure I could re-organize my whole library to match audiobookshelf recommendation, but unfortunately that would result in a ton of work. Instead, I had an idea: How about providing a SETTING for how the audiobook directory structure is.

In m4b-tool I use --batch-pattern, in tone it is --path-pattern (which I would prefer as name), so here is my feature description:

• Add a setting path-pattern for the Library, where you can put multiple path-patterns, that use grok-js like syntax to match part of the path with metadata fields
• Make the scanner match the path in order, so that you can prioritize, which path-pattern is gonna match first
• If no pattern matches, fallback to either metadata or default directory structure detection

The grok-js library is pretty old and has not been updated for a long time, but I think there are other implementations... it should also not be too hard to implement (here is the C# implementation i use for tone: https://github.com/Marusyk/grok.net)

What do you think?

[advplyr]

Hey, this is something I've wanted to do for a while but haven't come up with a good solution yet. It has been brought up a few times and there is a request for it #528.

I like the idea of using grok syntax. There are a lot of edge cases to cover and for that reason the current implementation is split up into multiple functions with regular expressions (See https://github.com/advplyr/audiobookshelf/blob/master/server/utils/scandir.js#L209).

For example, the series sequence can be taken from the title folder but only if there is also a series folder.

You could specify a publish year or a series sequence # as the first part of a book title folder name.
1984 - Book Title would use 1984 as the publish year
19 - Book Title would use 19 as the series sequence (if there is a series folder) or 0.5 - Book Title, 198 - Book Title

If server setting parse subtitles is enabled then the last part of the book title folder if separated by - would be used as the subtitle.
i.e. 1984 - Book Title - Some subtitle
and 1984 - Book Title would not detect a subtitle

Audio file names only try to parse out a track and disc number but also support being separated into CD folders i.e. /BookTitle/CD01/audiofile.mp3

I think it is possible to break this down into a list of grok strings and I think it would clean things up a lot.
Another thing to keep in mind is that podcasts or any future media types will be parsed using different rules.

[sandreas]

I like the idea of using grok syntax.

Great :-)

For example, the series sequence can be taken from the title folder but only if there is also a series folder.

Of course... audio books are named in so many different ways, that it is nearly impossible, to catch all edge cases with a simple regex based system. Some edge cases that I came across:

• A book, that has a year as its title (German: 1984 from George Orwell, https://www.audible.de/pd/1984-Hoerbuch/3837154270)
• A series beginning with 0 (German: Julia Schwarz - Die Autopsie, https://www.audible.de/pd/Die-Autopsie-Hoerbuch/B09VPXBCMX)
• A series using roman numbers (The Dark Tower IV, https://www.audible.de/pd/The-Dark-Tower-IV-Hoerbuch/B019WTIGL4)
• A series using decimal numbering (German: Patrick Rothfuss, Kingkiller Chronicles, https://www.audible.de/pd/Die-Furcht-des-Weisen-1-Hoerbuch/B00AHG8U2C)
• A series without ANY numbering (German: Der kleine Rabe Socke, https://www.audible.de/series/Der-kleine-Rabe-Socke-Das-Hoerspiel-Hoerbuecher/B00TFCYZ96)

I think it is possible to break this down into a list of grok strings and I think it would clean things up a lot.

Yeah I also think that. I also added shorthands (%s) and custom patterns for PARTNUMBER and NOTDIRSEP in tone, see:

• https://github.com/sandreas/tone/blob/a0a99eb61bf4c44a07d2f8b44513e0721f240ff2/tone/Services/GrokPatternService.cs#L108
• https://github.com/sandreas/tone/blob/7671919ba50a3fb3f756c697d2667e91374866d5/tone/Program.cs#L59

That way you can specify multiple path patterns very easy, e.g. my personal structure:

--path-pattern="audiobooks/%g/%a/%s/%p - %n.m4b" --path-pattern="audiobooks/%g/%a/%z/%n.m4b"

BTW: I would LOVE to see support for movement and movement-index for series in tag extraction, that would make things a lot easiers.

[advplyr]

1984 works for me because I don't have a subtitle there. If my 1984 book folder were written like 1984 - Some other info then it would pull "Some other info" as the title and 1984 as the publish year. The requirement for publish year is that it is first and separated by -.

Roman numerals are one of the reasons that series sequence accepts a string input, so if you have a roman numeral set in the audio file metadata then it would pull in correctly. It is auto-detecting it from the folder name that is the difficult part.

Decimals are currently supported in the folder naming for series sequence. i.e. 0.5 Book Title.

Series without sequences will still add the series and just not give the books a sequence.

These are all good edge cases and a lot of why making single regex's is so difficult.

We can definitely add support for movement and movement-index. I didn't even know about these. The scanner does support custom meta tags so you don't have to use the available ID3 tags for mp4. The new embed meta tag feature is embedding the same set of metadata into mp3 and mp4.
I'm not sure why this would be a bad thing except for that certain software may not support custom meta tags, but they should!

[sandreas]

Oh, I forgot - since I don't exactly know, how you extract the metadata from files, I assume you use ffmpeg / ffprobe?

If you are willing to add an optional static 20MB dependency with tone to your docker image to extract metadata more accurately, I would offer you to write an audiobookshelf specific metadata serializer (tone dump --format="audiobookshelf" audiobook.m4b), where you can define the format you would expect and I dump this format to console... e.g. a specific json as you wish...

You could go for:

// pseudocode

const existCode = runShellCommand("tone --help");
if(exitCode === 0) {
    useToneAsMetadataExtractor();
} else {
   useFffmpegAsMetadataExtractor();
}

I could provide the following fields (and more):

    
    public TimeSpan TotalDuration { get; }
    public string? Album { get; set; }
    public string? AlbumArtist { get; set; }
    public string? Artist { get; set; }
    public int? Bpm { get; set; }
    public string? ChaptersTableDescription { get; set; }
    public string? Composer { get; set; }
    public string? Comment { get; set; }
    public string? Conductor { get; set; }
    public string? Copyright { get; set; }
    public string? Description { get; set; }
    public int? DiscNumber { get; set; }
    public int? DiscTotal { get; set; }
    public string? EncodedBy { get; set; }
    public string? EncoderSettings { get; set; }
    public string? EncodingTool { get; set; }
    public string? Genre { get; set; }
    public string? Group { get; set; }
    public ItunesCompilation? ItunesCompilation { get; set; }
    public ItunesMediaType? ItunesMediaType { get; set; }
    public ItunesPlayGap? ItunesPlayGap { get; set; }
    public string? LongDescription { get; set; }
    public LyricsInfo? Lyrics { get; set; }
    public string? Part { get; set; }
    public string? Movement { get; set; }
    public string? MovementName { get; set; }
    public string? Narrator { get; set; }
    public string? OriginalAlbum { get; set; }
    public string? OriginalArtist { get; set; }
    public float? Popularity { get; set; }
    public string? Publisher { get; set; }
    public DateTime? PublishingDate { get; set; }
    public DateTime? PurchaseDate { get; set; }
    public DateTime? RecordingDate { get; set; }
    public string? SortTitle { get; set; }
    public string? SortAlbum { get; set; }
    public string? SortArtist { get; set; }
    public string? SortAlbumArtist { get; set; }
    public string? SortComposer { get; set; }
    public string? Subtitle { get; set; }
    public string? Title { get; set; }
    public int? TrackNumber { get; set; }
    public int? TrackTotal { get; set; }

    public IList<ChapterInfo> Chapters { get; } // with start, length, end, title, subtitle and picture
    public IList<PictureInfo> EmbeddedPictures { get; } // with format as base64 or binary - also export as file if you wish
    public IDictionary<string, string> AdditionalFields { get; } // either CUSTOM fields or specific native fields that are not mapped 

    // also size, streams, codec, channels etc... 

Where the AdditionalFields dictionary is very flexible and extensive... it reads nearly any defined tag.
Let me know if you would like to have a talk about this...

[advplyr]

Ffprobe for extracting metadata and Ffmpeg for writing metadata.

Does Tone support custom meta tags? If I want to write the metadata ISBN or ASIN?

Does Tone support embedding cover images?

One issue with using Ffmpeg for writing metadata is we can't just modify the existing audio file. We have to write the metadata to a new audio file then move and overwrite the existing audio file.

I wonder if we could bundle the ffmpeg binaries with the Tone binaries so that we only need to download one package. For the debian package and for the upcoming windows installer we don't get the luxury of Docker to download so we have to download those in the installer.

[sandreas]

Does Tone support custom meta tags? If I want to write the metadata ISBN or ASIN?

Yes... very powerful and easy. Although I must say that I only tested it myself and there may be some issues atm... but nothing that is hard to fix.

tone tag --meta-additional-field="ISBN=978..." --meta-additional-field="ASIN=0815..." my-audiobook.m4b

Does Tone support embedding cover images?

Yes... also very powerful. You can embed multiple images in different formats (e.g. jpg and png) and there is also support for Image-Types (Generic, Front, Back, etc...)

# import fixed cover file
tone tag --meta-cover-file="/tmp/a-cover.jpg" my-audiobook.m4b

# import covers in same folder automatically (e.g. my-audiobook.cover.jpg, my-audiobook.front.jpg, etc.)
tone tag --auto-import=covers my-audiobook.m4b

One issue with using Ffmpeg for writing metadata is we can't just modify the existing audio file. We have to write the metadata to a new audio file then move and overwrite the existing audio file.

Yes, that is one reason why I developed tone... others are, that ffmpeg does not read track length accurately, cannot embed covers into m4b files correctly and multiple minor issues with chapters. Although ffmpeg is fantastic in general, these minor issues are really annoying especially when tagging audio books.

tone v.0.0.4 also is scriptable (you can write your own JavaScript taggers, that read custom files or even load information from the internet). Since you are using JavaScript, it should feel pretty natural, see https://github.com/sandreas/tone#custom-scripted-taggers-experimental.

I wonder if we could bundle the ffmpeg binaries with the Tone
tone does not need an installer... it is ONE binary. Download it and it is ready to use. No libs, no dependencies...

You could also use exiftool, maybe sox or kid3... I don't want to push you to use tone :-) Its in an early state of development, maybe you wanna think that over, but it is the only tool I know that:

• is cross platform
• easy to setup
• scriptable
• supports multiple formats (mp3, m4b, flac, etc.)
• supports the vast majority of available tags / metadata for every format (audio books, music, custom tags, chapters)
• can be extended to fit exactly your personal needs ;-) (I would work with you to fix all your issues as fast as possible, since I am interested in getting audiobookshelf to work for myself)

[advplyr]

Can you show an example of how to use this inside abs?

[sandreas]

Can you show an example of how to use this inside abs?

I'm not sure, what you mean by this :-) What would you like to see?

BTW: With tone 0.0.6 there is a new metadata format tone.json which dumps and accepts nearly every possible metadata field in form of json:

$ tone dump --format=json christmasmiscellany2018_01_various_64kb.mp3
{
  "meta": {
    "album": "A Christmas Miscellany 2018",
    "albumArtist": "",
    "artist": "Lucy Maud Montgomery",
    "chaptersTableDescription": "",
    "composer": "",
    "comment": "https://archive.org/details/a_christmas_miscellany_2018_1807_librivox",
    "conductor": "",
    "copyright": "",
    "description": "",
    "discNumber": 0,
    "discTotal": 0,
    "genre": "speech",
    "lyrics": null,
    "originalAlbum": "",
    "originalArtist": "",
    "popularity": 0.0,
    "publisher": "",
    "publishingDate": "0001-01-01T00:00:00",
    "recordingDate": "0001-01-01T00:00:00",
    "title": "01 - A Christmas Of Long Ago (1906)",
    "trackNumber": 1,
    "trackTotal": 0,
    "chapters": [],
    "embeddedPictures": [],
    "additionalFields": {
      "tlen": "437.63"
    }
  }
}

As you see, there are some rough edges regarding the default values, which should be prohibited (e.g. trackTotal=0 does not make sense for a dump, same for chapters=[], etc.). I'm working on that.

You can also query specific property values by JSONPath:

$ tone dump --format=json christmasmiscellany2018_01_various_64kb.mp3 --query='$.meta.album'
A Christmas Miscellany 2018

And it is also possible to IMPORT metadata in this format:

tone tag --meta-tone-json-file="tone.json" my-audio-file.m4b

It is also planned to provide more data on the upper level, e.g.:

$ tone dump --format=json christmasmiscellany2018_01_various_64kb.mp3
{
  "meta": {
  },
  "audio": {
    "codec": "...",
    "duration": 23456
  },
  "file": {
    "size": 234666,
    "name": "my-audiobook.m4b",
    "created": "2022-02-12T12:34:56Z,
  }
}

But this is not fully implemented yet.

[advplyr]

I'm not sure how to use Tone in abs yet. I can see how to use Tone using the CLI on my local machine.

[advplyr]

I could write a node wrapper for Tone similar to this node wrapper for ffprobe
https://github.com/ListenerApproved/node-ffprobe/blob/master/lib/ffprobe.js

[sandreas]

I could write a node wrapper for Tone similar to this node wrapper for ffprobe

That would be the way to go. You could take a look at m4b-tools wrapper to get inspired. It creates a temporary file in tone.json format, runs an import and removes the file.

Oh and be aware, that there is a bug in the tag library for description and recordingDate, see Zeugma440/atldotnet#155

For docker you can use a simple COPY call referencing the multiarch image:

FROM sandreas/tone:v0.0.6 as tone
# ...

COPY --from=tone /usr/local/bin/tone /usr/local/bin/

Example: https://github.com/sandreas/dockerhub-builds/blob/main/m4b-tool/latest/Dockerfile

[advplyr]

I've been playing around with Tone today and see you have predefined meta tags. To save me some time going through the code do you have a list somewhere of how these map to different audio formats?

I'm trying to learn more about the different versions of ID3 and the supported tags for each format but can't seem to find a source of truth. I would like to finally standardize the ID3 metadata used in Abs for the various audio file formats.

[sandreas]

I've been playing around with Tone today and see you have predefined meta tags. To save me some time going through the code do you have a list somewhere of how these map to different audio formats?

I use atldotnet. Unfortunately there is no such mapping table and I'm not really sure I understand what you are asking for... There is no such thing as a standardized mapping of metadata to different formats or specifications, just best practises - I also thought about publishing a best practise guide, but this would be a lot of work. The best sources for mapping I found were these:

• MP3TAG reference (as stated above): https://docs.mp3tag.de/mapping/
• Hydrogen Audio: https://wiki.hydrogenaud.io/index.php?title=Tag_Mapping
• Kodi: https://kodi.wiki/view/Video_file_tagging

Some other resources I like to use as reference:

• Plex Audiobook Guide: https://github.com/seanap/Plex-Audiobook-Guide#tags-that-are-being-set
• Quicktime tags: https://exiftool.org/TagNames/QuickTime.html

Based on these mapping tables and information, I created my personal mapping table of tags that are not supported natively in atldotnet (although you can always use AdditionalFields to still read and write them). See here for details: https://github.com/sandreas/DotnetLibAudioMetadata/blob/2aa4523dda20d71bd9ec72a00bec153f01dbe5c2/Sandreas.AudioMetadata/AudioMetadata/MetadataTrack.cs#L52

Of special interest is the field ----:com.pilabor.tone:PART for part of series. MovementName can be used for the name of the series, while Movement is an integer field and may be used for the part of the series, if it is an integer. If it is not (some series have sub-parts like 0.5 or 2.1), there is no standardized way of storing this information. So I introduced a custom string field for my usage in tone.

BTW: ffmpeg really is NO source of truth here. There are several very annoying things when you use ffmpeg to tag your files. The ffmetadata format does not specify a lot of tags required for audiobooks (e.g. Movement, MovementName, PurcaseDate, just to name a few)

I'm trying to learn more about the different versions of ID3 and the supported tags for each format but can't seem to find a source of truth.

Well, this is easy. For ID3 visit the official spec: https://id3.org/id3v2.3.0. I recommend using 2.3.0 wherever possible, but there are also links to v2.4.0 (latest) and v2.2.0 (the one before). I think that previous versions are no longer relevant - even if ID3 V1, which uses a fixed 128 byte size TRAILER (end of the file) often is seen as "fallback" and still present, it is not really useful in most cases.

I would like to finally standardize the ID3 metadata used in Abs for the various audio file formats.

I'm surprised that ID3 is your preferred format :-) I would never use mp3 for audiobooks again since I found m4b.

Please note that latest tone v0.0.9 fixed a lot of issues since v0.0.6. I think it is now pretty stable and supports most of the relevant tags for audiobooks. You can now also use tone.json format to export or import ALL known metadata fields including lyrics and embedded pictures (covers). Another thing, that might be interesting is, that tone now is able to dump audio stream details:

tone dump "Santa Claus/Santa Claus.m4b" --format=json --query=$.audio
{
  "bitrate": 64,
  "format": "MPEG-4 Part 14",
  "formatShort": "MPEG-4",
  "sampleRate": 22050.0,
  "duration": 1289730.0,
  "vbr": true,
  "channels": {
    "count": 2,
    "description": "Stereo (2/0.0)"
  },
  "frames": {
    "offset": 36,
    "length": 10317840
  },
  "metaFormat": [
    "mp4"
  ]
}

The frames information might be of special interest - this shows up the window of raw binary data (with metadata fields). So this is the core part of the audio file. Building a hash of these offset would never change, even if metadata changes. This is interesting for finding duplicates or keeping the metadata when files get moved around. Just in case you need this somewhere.

[advplyr]

Thanks for this detailed response.

There is no such thing as a standardized mapping of metadata to different formats or specifications, just best practises

I meant to say that I would like to create a standard for audiobookshelf because the best practices are inconsistent.

BTW: ffmpeg really is NO source of truth here. There are several very annoying things when you use ffmpeg to tag your files. The ffmetadata format does not specify a lot of tags required for audiobooks (e.g. Movement, MovementName, PurcaseDate, just to name a few)

Right, Ffmpeg is not suitable for writing meta tags which is unfortunate because we need to use Ffmpeg for transcoding regardless. This is why the ideal solution for audiobookshelf would be to bundle Ffmpeg with Tone so we only need to do one download.

Handling this in docker is the easiest, but we also build a debian package that installs Ffmpeg and eventually a Windows installer that will install Ffmpeg. Installing them separately is fine, I will have to host the binaries for Tone somewhere. Windows binary, linux amd64, arm64 and arm/v7.

I'm surprised that ID3 is your preferred format :-) I would never use mp3 for audiobooks again since I found m4b.

It's not my preferred format but audiobookshelf needs to support writing mp3 meta tags and mp4. I think it would be better to write ID3 tags to mp3s and write iTunes tags to mp4s. I'm not very familiar with any of it though and for now would prefer a tool that handles this.

[sandreas]

I will have to host the binaries for Tone somewhere. Windows binary, linux amd64, arm64 and arm/v7.

tone provides these in the github releases page and it is one single file to download (no installer required). What about macOS (Intel, arm64)?

I think it would be better to write ID3 tags to mp3s and write iTunes tags to mp4s. I'm not very familiar with any of it though and for now would prefer a tool that handles this.

tone does exactly this... by default it writes native metadata, which means mp4 on all mp4 files, id3v2.3 on mp3 files, if NO metadata is present but keeps the ID3 version, if there already IS metadata (v2.2, v2.3 or v2.4). So I'm pretty sure it does would you would expect it to do. But you have to verify that yourself.

[sandreas]

Yeah I'm not removing the code that pulls the ffprobe metadata but I don't want to run both for each audio file. Running ffprobe for each audio file is expensive and this is a bottleneck for very large libraries.

Funny, thats pretty much why I wrote tone. And that ffmpeg does not play well with millisecond timestamps, mp3 chapters and special tags like movement-name... Why are there no other cross platform cross format metadata reader/writer out there supporting ALL the possible tags? Strange...

I can help with tone also if necessary down the road, I used to write a lot of C# many years ago.

Yeah sure, feel free to open discussions or PRs whenever you like. Currently I'm in discussion with the spectre.console team (my command line library) - that is a big part of my known issues section. I think they have a problem in their command line arguments processing but maybe I'm totally wrong with that. Let's hope this will be the next fix, although you won't need these kind of command line arguments, if you use tone.json format.

[advplyr]

Tone throws an error when reading ID3 metadata in m4b files. The error messages are coming through stdout so node-tone is unable to parse the json response.
Is tone going to support reading ID3 tags from mp4 files?
Either way it would be preferable that errors get written to stderr.

[advplyr]

Hey I put more details here: sandreas/tone#26
Hope that makes sense

[sandreas]

Thanks. Updated the issue, need a test file to reproduce... my ffmpeg refuses to create a mp4 file with id3 metadata :-)

[sandreas]

@advplyr Just to inform you again: I unfortunately need your feedback to 100% fix this problem... I already did some code changes but without your files, I failed to reproduce the problem.

[sandreas]

Just to mention something: https://github.com/Borewit/music-metadata

I don't know if you are aware of this... :-)

[advplyr]

Thanks, I did see that one and it is only for reading metadata and not writing.