Byte Offsets in the XREF table

[MyMilkedEek]

Hi all,

A few weeks back, I implemented a rudimentary byte offset in the XREF table and I wanted to collect some feedback on this. Especially on how Object streams are handled.

In the current implementation, if an object resides in an object stream then the byte offset is the offset to the object stream that contains the object. I feel that this is the preferred way of dealing with these, but I wanted to hear other opinions on whether or not it should show it.

[yulian-gaponenko]

I'd consider one of the two options:

1. Just print the object stream number, because the offset to the object stream can be easily seen by examining this object line.
2. Print the object stream number and the offset to the object inside the object stream.

Btw, how does this work together with the PDF editing capabilities of RUPS? I.e. when something is changed in the document, what would/should happen with these offsets? Should we update them? Should we hide this info? Should we only show this info in read-only mode?

[MatthiasValvekens]

Print the object stream number and the offset to the object inside the object stream.

This sounds pretty reasonable. Granted, RUPS currently doesn't offer introspection of object streams, so that offset isn't immediately actionable, but it's indeed more useful than just printing the offset of the stream itself, IMHO.

I think the most tricky aspect is making it obvious in the UI what the second number refers to. There are at least three ways to read that number (without the context of this conversation):

• The byte offset of the object stream in the file (as in Michaël's screenshot)
• The byte offset of the object inside the object stream (as in Yulian's suggestion)
• The index of the object inside the object stream (which IIRC is what the xref stream would contain, but I don't have the spec in front of me).

All of these are positive integers so they look the same. Perhaps a tooltip or a prefix would help :)

[MyMilkedEek]

Yeah, the offset within the stream might be more useful, but not immediately I would say. At least not until RUPS offers an insight into that stream.

That being said, I'm thinking long the lines of:

ObjStm 12 (Internal Offset: 1234)

Which may prove to be too long, so perhaps:

ObjStm 12 with a tooltip Internal Offset: 1234

Or do our own shorthand:

12 0 R: 1234 with a tooltip explaining the meaning behind this: This object can be found in Object Stream ID 12 with a byte offset of 1234 within that stream.

As for the updating question. As of now, the changes you make before saving are not reflected in the XREF table, simply because the objects haven't been written yet and as such have no XREF entry. Saving does reload the document and as such, the XREF table as well.

[mkl-public]

Indeed it's the index within the object stream.

Thus, probably Object n in Stream m

[MyMilkedEek]

Would you prefer index or offset within the stream?

[yulian-gaponenko]

I guess index would be more useful, especially since it's indeed how objects are referred in the PDF syntax itself. And if it'd be possible to introspect the contents of the object stream, indexes will be much more intuitively useful rather than offsets.

[MatthiasValvekens]

Yeah, I also like the object n in stream m wording that that would enable.

[MyMilkedEek]

I'm looking into writing object n in stream m which is indeed the better option, but I feel RUPS needs to show these streams first for it to have any kind of use. Just getting them included in the PDF Tree View is what I'm working on right now:

Not completely sold on the placement of this, but this allows me to reuse the main view.

[petervwyatt]

If I could have a wish list (and I appreciate that development complexity is a separate issue) then it would include:

• more visibility of and into each cross-reference stream. By "into" I mean that the decompressed xref stream data for each xref stream is presented in a tabular human-readable form based on Table 18 in ISO 32000-2:2020.

• more visibility of and into each object stream. By "into" I mean that the first line in the decompressed object stream (that contains N pairs of integers) is human readable - ideally in tabular form but otherwise as a line of text

I think this also means that the RUPS view of the trailer needs to show that it is really an XRefStream and not a traditional (simplified) trailer. Maybe this is as simple as showing the /Type /XRef and /W entries as well as any Prev entry.

Separately:

• it appears that RUPS is not showing any custom keys from trailers
• when I click on the object number of the XRef stream in the XREF tab, the console tab shows multiple "WARN - Error while parsing PdfStream" errors. I assume this is because RUPS doesn't know this is an XRef stream

[michaeldemey]

Hi Peter,

• More visibility into XREF Streams is definitely an addition we'd be willing to look into. Not going to lump it in with this fix because it might not ever make it to /develop ;)
• I'm actually looking into this already. As of yet, the visualization reads not-formatted for now:
  

As for RUPS not showing the custom keys in the trailer: I already looked into the issue and this is actually a bug in iText's behaviour. iText RUPS uses PdfDocument, but in stamping mode, to manipulate PDF files. This mode unfortunately took a shortcut by reusing the "new document" mode for the /Trailer dict and re-instantiates the object. This meant that it's ignoring custom entries. I've added a fix and I'm going through the merging process. RUPS did no extra analysis/validation/cleaning in this case, so it should be fixed moving forward.

Where stampedTrailer is a /Trailer with a custom entry being read.

And yeah, RUPS is complaining about that as well for ObjStm but it does read them perfectly, make sure to double check the Stream tab, while we work to get rid of that message.

[MatthiasValvekens]

I assume this is because RUPS doesn't know this is an XRef stream

If memory serves, RUPS will try parsing any stream as a content stream by default, and log errors if that fails, which isn't particularly elegant. You'll see similar behaviour on metadata streams. There's separate handling for image data, though, so perhaps it's not too difficult to add "nice" handling for other commonly used stream types. Examples I can think of right now: XMP metadata, object streams, xref streams, embedded file streams.

All of these should be identifiable as such without requiring any extra context if the stream extent dictionary contains all required keys, so it sounds feasible... It's been a while since I looked at that area of the source, though :).

[petervwyatt]

Thanks Michaël and Matthias!

Separate question: RUPS displays JPEG (DCTDecode) images fine and logs console messages for not being able to do the same for JPEG 2000. That's all good. However, for JBIG2 images, I get "Image can't be loaded" white text on a red background in the Stream tab, but no console message. Does this mean the JBIG2 image data in question is potentially corrupt or is the support missing as per JPEG 2000 but just not logged?
See object 612 in this PDF: https://s3.documentcloud.org/documents/23108357/redacted-version-of-exhibits-a-j-to-letter-to-the-honorable-kathaleen-st-j-mccormick-from-edward-b.pdf

[michaeldemey]

RUPS delegates the rendering to the Java APIs, it might just be that Java doesn't support it properly.

[michaeldemey]

FYI: #29