Inline widgets in Flutter

[matthew-carroll]

Super Editor needs to support various inline non-text visuals. Generically speaking, this means small widgets that sit within a block of text.

Three major questions that we need to answer include:

1. Can Flutter render the desired content?
2. How should super_editor structure non-text content within text blocks?
3. How can super_editor extend the concept of selection, caret movement, and other interactions to arbitrary inline non-text content?

Rendering inline content with Flutter

Flutter's rich text layout is based on a tree of InlineSpans. The root span for a block of text is a TextSpan. A TextSpan can display generic Widgets inline by configuring one or more WidgetSpans as the children for the TextSpan.

Example:

  // Renders "We asked @SuprDeclarative and he said we're good." with "@SuprDeclarative" replaced with a widget.
  RichText(
      textAlign: TextAlign.start,
      text: TextSpan(
        text: "We asked ",
        style: const TextStyle(
          color: Colors.black,
        ),
        children: [
          // This WidgetSpan displays a yellow pill with the text "@SuprDeclarative" inside it.
          WidgetSpan(
            alignment: PlaceholderAlignment.middle,
            child: Container(
              padding: const EdgeInsets.only(left: 4, top: 4, bottom: 4, right: 8),
              decoration: const ShapeDecoration(
                shape: StadiumBorder(),
                color: Colors.yellow,
              ),
              child: Row(
                mainAxisSize: MainAxisSize.min,
                crossAxisAlignment: CrossAxisAlignment.center,
                children: const [
                  Icon(
                    Icons.account_circle,
                    size: 14,
                    color: Colors.black,
                  ),
                  SizedBox(width: 4),
                  Text(
                    "@SuprDeclarative",
                    style: TextStyle(
                      color: Colors.black,
                      fontSize: 12,
                    ),
                  ),
                ],
              ),
            ),
          ),
          const TextSpan(text: " and he said we're good."),
        ],
      ),
    );

Based on a coding spike, it looks like Flutter is capable of rendering the nominal case for the most common use-cases:

Possible problems

• How do we render inline code samples that need to wrap lines? This doesn't look like it's possible with Flutter right now.

Structuring non-text content

Super Editor's representation of text with styles and metadata is called AttributedText. AttributedText allows for any number of overlapping attributions that span any number of characters of text.

AttributedText(
  text: 'I am bold',
  spans: AttributedSpans(
    attributions: [
      const SpanMarker(attribution: bold, offset: 5, markerType: SpanMarkerType.start),
      const SpanMarker(attribution: bold, offset: 8, markerType: SpanMarkerType.end),
    ],
  ),
);

Currently, AttributedText has no concept of non-text content.

It's possible that inline widgets could be supported without changing AttributedText. The following is a proposal that I received:

Paragraph(
  children: [
    AttributedText(...),
    SomeInlineWidget(),
    AttributedText(...),
  ],
);

In practice, splitting a single text block into multiple AttributedText segments is probably impractical. User selections and attributions might cross the inline widget and impact multiple AttributedText instances. This proposal would likely require individual app developers to coordinate content and attribution changes across multiple AttributedText instances. That would be very cumbersome and developers might avoid Super Editor due to that difficulty.

A more useful solution would find a way to utilize AttributedText to reference non-text content such that the non-text content can be rendered and selected without making AttributedText aware of non-text content.

To get a feel for what this might look like, consider the following use-cases. Keep in mind that this is just a proposal for what the content structure might look like; this structure doesn't solve rendering or user interaction.

Inline Icon

One might display an inline icon in the following ways.

Replace an arbitrary character with an icon bitmap:

final fireIconAttribution = InlineIconAttribution(asset: "images/fireIcon.png");

AttributedText(
  text: 'Super Editor is ▯',
  spans: AttributedSpans(
    attributions: [
      const SpanMarker(attribution: fireIconAttribution, offset: 16, markerType: SpanMarkerType.start),
      const SpanMarker(attribution: fireIconAttribution, offset: 16, markerType: SpanMarkerType.end),
    ],
  ),
);

Replace a substring with an icon bitmap:

final fireIconAttribution = InlineIconAttribution(asset: "images/fireIcon.png");

AttributedText(
  text: 'Super Editor is :fire:',
  spans: AttributedSpans(
    attributions: [
      const SpanMarker(attribution: fireIconAttribution, offset: 16, markerType: SpanMarkerType.start),
      const SpanMarker(attribution: fireIconAttribution, offset: 21, markerType: SpanMarkerType.end),
    ],
  ),
);

Inline avatar

An inline avatar could be attributed in a similar fashion to an inline icon:

final avatarAttribution = InlineAvatarAttribution(avatar: "https://myavatar.com", name: "@SuprDeclarative");

AttributedText(
  text: 'I am @SuprDeclarative',
  spans: AttributedSpans(
    attributions: [
      const SpanMarker(attribution: avatarAttribution, offset: 5, markerType: SpanMarkerType.start),
      const SpanMarker(attribution: avatarAttribution, offset: 20, markerType: SpanMarkerType.end),
    ],
  ),
);

Inline code

Inline code could be attributed in a similar fashion to an inline icon:

final codeAttribution = InlineCodeAttribution("super_editor");

AttributedText(
  text: 'To get started, use a super_editor widget.',
  spans: AttributedSpans(
    attributions: [
      const SpanMarker(attribution: codeAttribution, offset: 22, markerType: SpanMarkerType.start),
      const SpanMarker(attribution: codeAttribution, offset: 33, markerType: SpanMarkerType.end),
    ],
  ),
);

A downside to this approach is that the app developer needs to keep the code in InlineCodeAttribution in sync with the content of AttributedText. For example, consider what happens when the user deletes a character within the inline code attribution.

An adjustment to this approach might be to give the attribution renderer more insight into the content that it's rendering so that the renderer can inspect the AttributedText, find the inline code text, and then pass that to the inline widget.

Rendering AttributedText with non-text content

Rendering is where things start to get tricky.

As a reminder, the process for rendering AttributedText involves chopping up the AttributedText at every attribution marker, passing a set of attributions into a function, and then that function returns a TextSpan. The TextSpans are accumulated into a list and then passed to a RichText widget. Obviously, we'll need to generalize the return type of the render function to InlineSpan, but that's a minor change.

The simplest case

Looking at displaying inline widgets, the simplest case is one in which the inline content has no other overlapping attributions:

"I am |@SuprDeclarative|"

In this case, the render function would return an appropriate WidgetSpan for the InlineAvatarAttribution that surrounds "@SuprDeclarative". That should solve the simplest case for a non-selectable text replacement. This same approach would work the same way for an icon replacement.

Overlapping attributions

In the case of something like inline code, we have the possibility that an attribution, like bold, partially overlaps the inline widget attribution:

// [ ] - represents bold
// | | - represents inline code
"I am [bold |This is] some code|"

In this scenario, the render function is going to receive text that is chopped up into the following groups:

1. "I am " - no attributions
2. "bold " - bold
3. "This is" - inline code + bold
4. " some code" - inline code

Notice that the above segmentation prevents the render function from combining "This is" and " some code" into a single widget. For this reason, the render function might need some new look-ahead capabilities so that it can collect additional groups of text.

Another option is to prohibit any attributions from overlapping an inline widget attribution. I'm hesitant to go down that route because we have no idea what kind of metadata developers might want to inject into their text, including inline widget content.

Inline widget interaction

The most complicated part of inline widget content is working that content into existing editor interactions.

Consider the following:

• The user taps to place a caret
• The user drags a caret position on mobile
• The user drags an expanded selection across text
• The user moves forward, backward, up, or down using arrow keys
• The user presses backspace at the trailing end of an inline widget (especially inline code)
• The user inserts a character within inline code
• The user pastes clipboard content into inline code

Super Editor currently has two levels of user interaction. At the higher level, the user can interact with the document layout and document selection. At the lower level, the user can interact with a text layout and text selection.

The introduction of selectable, editable inline arbitrary content introduces a 3rd level of layout and selection that sits beneath that of the text layout. Super Editor now needs to support:

1. Document layout
2. Text layout
3. Arbitrary 3rd party layout in text

The total impact on the existing Super Editor implementation is unclear, however a couple details seem likely.

First, Super Editor will need to replace its use of TextSelection with a broader concept that represents a mixture of selected text and selections of arbitrary inline content. This change is kind of like taking the complexity of DocumentSelection and applying it to every TextNode.

Second, Super Editor will need to introduce a layout interface for every inline widget so that whenever the user taps or drags over the inline widget, Super Editor can query for the meaning of that position within the inline widget. For something like an icon, those answers are simple - any area over the icon indicates an icon selection. However, for something like inline code, the inline code needs to translate the given offset to the corresponding character of text within the inline code widget.

Conclusion

At this moment, I think we can say:

• Flutter can render nominal use-cases. It can't render multi-line inline widgets, such as inline code.
• Attributions within AttributedText can already be used to structure inline content.
• A few adjustments to the AttributedText rendering system should allow for inline widget rendering.
• Treating all inline widgets as an all-or-nothing selectable unit can probably be implemented with a moderate amount of work.
• Extending a complete layout and interaction model to inline widgets will require significant work, and it will require reworking some of Super Editor's core text selection mechanisms.

[miguelcmedeiros]

How do we render inline code samples that need to wrap lines? This doesn't look like it's possible with Flutter right now.

@matthew-carroll, using the demo from #393 and resizing the window it seems that flutter handles the multi-line scenario:

Or did I understood wrongly what is the limitation?

[matthew-carroll]

@miguelcmedeiros I meant when the inline code itself needs to wrap. Try replacing "super_editor" with a longer line of inline code.

[justinmc]

How about something like a table? I guess a table usually wouldn't be inline, but that would reasonably need to support rich text formatting within the table widget.

I think it would be worthwhile for Flutter to invest in supporting multiline inline widgets. People probably already expect that to work in RichText. Any idea if there is an issue for that or should I create one?

I wonder if @chunhtai has any opinion on attributing rich text having worked on Flutter's AttributedString.

[chunhtai]

AttributedString is an engine/framework interface to send information to the engine side, but it is basically the same thing a TextSpan with overlapping style/attribute and with range. I think it will be easier if TextSpan can have overlapping style/attribute.

supporting multiline inline widgets.

I am curious how we can do this

[matthew-carroll]

I think a table is inherently a block level construct, but you might be right about content within cells.

I think we can make a few statements about inline widgets that break by their own preferences:

• Such widgets should still be able to work as they do today. So if an inline widget really does want to take up more space than remains in a line, and thereby force itself onto the next line, that should be possible.
• In the case where an inline widget breaks by its own preferences, we'll need the possibility of creating multiple RenderBoxs. Each break in the inline widget needs its own box. Flutter can't, for example, cut the pixels at the boundary because something like inline code wants to display beveled corners around every discrete box. Odds are, if we want multiple RenderBoxs, we probably want developers to provide a Widget for each RenderBox because almost no one knows how to use RenderBoxs.

I have no idea what the layout protocol and API should look like, but I think we can imagine a high level solution.

1. The RenderParagraph lays out text up to the point of an inline widget.
2. The RenderParagraph enters a kind of layout loop.
   1. The RenderParagraph passes appropriate geometry information to the inline widget
   2. The inline widget returns either a Widget or a RenderBox that fits the geometry
   3. The RenderParagraph positions the RenderBox as desired
   4. The RenderParagraph updates the geometry for the remaining space available in the line
   5. Repeat until the inline widget no longer returns content

I'm referring to the thing we're laying out as an "inline widget" because that's how it's discussed elsewhere. But with this change, that "inline widget" is closer to some kind of builder. Or, more accurately, it's a very narrow, specialized layout implementation. A layout that understands concepts of line geometry.

Depending on where this investigation leads, we might find that this inline widget should implement a new type of RenderObject. A RenderBox supports BoxConstraints, and a RenderSliver supports SliverConstraints. Perhaps what we're discovering here is the need for a RenderLines, which might support LineConstraints and produce LineSegments. That might be overkill, but it's a thought. One thing that I think we're discovering more and more is that the convenient idea of some arbitrary visual widget in text is overly simplistic and it's missing multiple dimensions of control.

What do you guys think?

[matthew-carroll]

@justinmc @chunhtai I thought about this a bit more and I'm wondering if it would make sense to refactor how all the text is flowed, based on the above RenderLines concept.

Flutter has established this strong boundary between the idea of text layout and inline widget layout, as if the two are very different goals. But inline code is both text AND an inline widget. Clearly these categories are not distinct.

Imagine that we introduce the machinery for inline widgets to lay themselves out along lines, as described previously. At that point, I think it begs the question as to why regular text layout wouldn't use the exact same machinery. Why wouldn't text layout, itself, go line by line and fill the available space, with a reported line height?

If text layout uses a generic RenderLines mechanic then I think that would open the door for text to avoid nearby objects, like images, with minimal effort. A developer could define a new RenderBox, for example, that supports text with "floated" content by laying out text and inline widgets, line by line, constricting each line based on the size and location of floated items throughout the content.

All of this is to say that I think this might be an opportunity to treat text and inline widgets the exact same way from a layout perspective, which I think would prove to be a big win as text-based layout complexity grows.

[matthew-carroll]

Here are some slides that expand on the goals of a new text layout algorithm: https://docs.google.com/presentation/d/1Ymjz87vxp41Y8l4sonJGlnkXri_oeAsGlO5hZ-8617I/edit?usp=sharing

[chunhtai]

I think RenderLines idea may probably the only way we can achieve with wraping inline widget, but the API may be too complicated from what i imagine it would be. It looks like most of the examples that need wrapping are usually text based, why couldn't they be a TextSpan with required style?

On the other hand, there is an example in inline widget with special layout

I wonder whether we can do what html does, create something like ColumnSpan and RowSpan for these?