Ordering of higlights

[Gurrlin]

Hi,

Fist of all, thanks for making this straight forward tool. The readwise kobo implementation was working fine but stopped syncing my highlights after 91 entries, not sure if there's a limit to the number of higlights synced or perhaps the % of the book.

As a compliment to readwise October works great, but I was wondering if it's possible for the highlights to be sorted by order in the book? Currently the import I got is sort of a jumbled mess which is detrimental to the way I approach and review my higlights.

[marcus-crane]

Hi there,

There are no limits on the number of highlights that can be associated with a book so if you can file an issue to track it, we can start doing some investigation.

If you're able to perform a sync and then forward your application logs to [email protected] when you hit an error, that'll help me get a better understanding of what might be going wrong.

As for the question around highlight ordering, there is an open issue which I haven't gotten back to but this did prompt me to sit down and have a closer look at the issue.

At present, October does something kind of dumb which is that it fetches all of your bookmarks ordered by book title and chapter percentage. The dumb part is that this chapter percentage is not taking into account the chapter itself so a highlight at the end of Chapter 1 would be sorted near the end of the light compared to a highlight at the start of Chapter 9 so that's clearly a bug I'll fix very shortly.

As for the following, you can feel free to skip it if you don't care about the technical details but for anyone who might be interested in learning a bit more, I'll walk through some of the possible snags in truly ordering highlights. It also helps me think through the issue as well.

---

Here is an example of a highlight pulled from my own Kobo's Bookmark table where highlights and notes are stored:

VolumeID: "file:///mnt/onboard/Gow, Ian D. & Kells, Stuart/Big Four_ The Curious Past and Perilous Future of the Global Accounting Monopoly, The - Ian D. Gow & Stuart Kells.kepub.epub"
ContentID: "/mnt/onboard/Gow, Ian D. & Kells, Stuart/Big Four_ The Curious Past and Perilous Future of the Global Accounting Monopoly, The - Ian D. Gow & Stuart Kells.kepub.epub!!OEBPS/chap01.html"
ChapterProgress: 0.678571428571429
StartContainerPath: span#kobo\.53\.1
StartOffset: 0

The identifiers here roughly capture the following information:

• VolumeID: The file path of the book
• ContentID: The file path of the chapter of the book. Here we have chap01.html which is nicely sortable due to containing a number and a fairly common convention (chap01, part01, text01 etc) but there are no hard guarantees.
• ChapterProgress: A number reflecting the progress through the current chapter. This highlight was taken 67% of the way through chap01.html
• StartContainerPath: Technically epub books are HTML files so this contains the HTML element of the highlighted content. In this case, this file is a kepub so it has some fairly standard ordering but wackier path formats such as OEBPS/30_rm_draft-3-8.xhtml#point(/1/4/2/26/1:132) do exist. In generally, these appear to be sortable alphanumerically though.
• StartOffset: I'd have to double check but I believe this reflects the number of chapters indents a highlight is so for example, if my highlight was "This is a sentence", the offset would be 10 for the current sentence

Anyway, there is plenty here that can be sorted easily enough. The ideal query would probably sort by VolumeID, ContentID, ChapterProgress, StartContainerPath and StartOffset in that order.

It is technically possible to have the exact same ChapterProgress with slightly different StartContainerPath values such as if you make two highlights of two lines that are right next to each other so a very high level of granularity helps.

Now, the tricky part here is that Readwise's API accepts either two types of ordering: You can provide an integer (a number) or you can have them ordered by the order that they are sent.

This raises a bit of an issue. At the moment, we just submit highlights in the order they are sorted. This is fine but if you were to submit new highlights from the same book, they would appear at the end of the list.

We also could explicitly provide a number to each highlight but we would need to know the number of every highlight that exists for the book in order to calculate the proper placement. If the first item in the list is Highlight 1 and I submit 4 different batches of highlights for the same book over time, I'd end up with the first 4 entries being somewhere in the opening pages of the book but not in any strict order.

If ebooks had a clear representation of the current page stored in the DB, it would be a bit easier to deal with but there isn't really a concept of a page with a digital book since you can "shrink" or "grow" the number of pages by adjusting the font size and so on making the idea of a page number meaningless compared to what it means for the physical copy of the book.

Ideally, it'd be possible to take all of the values above and smush them all together into an identifier which is uniquely sortable no matter where in the book you are, assuming your digital copy of the book doesn't change such that different copies have different chapter naming schemes (ie; chap0001.html vs part0001.html etc) but the Readwise API doesn't support that at present.

I did notice that the newly released official Kobo integration supports numbering although it's not entirely clear to me what the layout of a Kobo Store purchased book looks like compared to any random book pulled from the internet so they may have a much more consistent ordering system.

Hopefully I'll be able to ask them how the new integration works and maybe that'll shed some light on how the above might be solved in the most ideal way.

It does also raise the question too that you'd probably have to resync your existing books in order to benefit from page numbering assuming you still have the highlights in question available.

[aidankinzett]

Hi, have you had the chance to look at/fix this issue? My imports seem to be in the order that they appear in the chapter instead of the book. Thanks for your work on this software

[marcus-crane]

Hi Aidan,

Nothing much to show. It's less of a minor fix and more like a rewrite of the core of October. The only way to "know" where highlights take place in a book is basically to parse the actual book (or the table of contents specifically) as that information doesn't exist on Kobo devices.

Usually around this time of year when everything is quiet, I aspire to sit down and work on October a bit but I don't promise anything much as admittedly, I don't use it myself much these days. That doesn't necessarily mean it's abandoned, just that it doesn't always get prioritised as far as side projects.

I'll post here when I have something to share though!

[Gurrlin]

Thanks for the reply and the update!
To clarify, the October and Readwise implementation works as intended. What I meant to say that syncing my Kobo account on the Readwise webpage is what seemingly stopped working for the book I mentioned.

Also intresting to get some technical knowledge on how it looks behind the scenes.

[marcus-crane]

Ahhh, gotcha. The V2 Readwise Kobo integration, which is only for Kobo Store purchased books, is entirely handled by the Readwise team and not something I have any insight into 🙂

The split is basically that you can use October for sideloaded content or you can use the official integration for store purchased content since the former is a bit too niche and requires accessing the actual device unlike store-synced highlights.

I didn't actually realise the official integration was back until yesterday as the first version was fairly quickly shut down by Kobo while it seems V2 has their official blessing.

As far as ordering, I've started looking into what's involved in parsing the epubs (which are technically just a zip file containing HTML files) and it seems pretty promising.

I'll post any updates in the linked Github issue

[Gurrlin]

I see, I'll try sending ReadWise another e-mail then as I never got a response with that issue.

I also noticed that Oktober doesn't keep formatting of the higlighted text. Most notably is that there's no line breaks. At least not for the book I used which was bought from the Kobo store.