Version 0.2 Roadmap

[jugglinmike]

Hey everyone! I know it might not look it, but @iros and I have been hard at
work designing the next iteration of the library. We wanted to share our plans
with you and solicit feedback--let use know what you think!

Abstracted Data Attributes

Almost immediately after the release of d3.chart, developers identified an
aspect of reusable visualizations that we missed completely: the shape of the
input data. You can see #22 and #12 for discussion, but it boils down to this:
truely reusable charts shouldn't force users to define the input data in any
particular way.

The solution has been frought with gotchas, but I believe we're at a point
where we can support abstract data attributes. Chart authors would be required
to "reserve" the data that their chart relies on:

d3.chart("AwesomeChart", {
    dataAttrs: ["time", "space"],
    initialize: function() {
        // d3 code as normal, with references to `d.time` and `d.space`
    }
});

...and chart consumers could then (optionally) specify a "mapping" of the
chart's data shape to the shape of their own data:

var data = [{ tiempo: 230120, espacio: 930 }];
var miGraphico = d3.select("svg").chart("AwesomeChart", {
    dataMapping: {
        time: function(d) { return d.tiempo; },
        space: function(d) { return d.espacio; }
    }
});

d3.chart would preform this transformation in real-time to avoid the memory
overhead of copying potentially-large datasets wherever possible. To keep
d3.chart more focused, we're implementing this functionality in a standalone
JavaScript module: DataMap.

Named Mixins and Data Demuxing

Currently, the Chart#mixin method kind of does a lot. It creates a new chart
with the specified options and ties its draw function to that of the parent
chart. We'd like to simplify the method to accept a name and a chart. This
means you will only have one canonical method for creating charts (d3.chart),
and Chart#mixin simply creates associations between two charts' draw
methods.

d3.chart("Demo", {
    initialize: function() {
-       this.mixin("ChartName", this.base.append("g"), {
-           someOption: 23
-       });
+       var mixinChart = d3.chart(this.base.append("g"), {
+           someOptions: 23
+       });
+       this.mixin("mixinChart", mixinChart);
    }
});

"Great," you're thinking, "a more verbose API." I know, I know, but stick with
me--it's worth it!

By separating responsibilities like this, we can make it less confusing to wire
up charts in different ways.

Some charts don't need to be redrawn every time new data arrives (@iros and I
refer to these as "decorator" charts--a chart that defines a tooltip would be
one example). In this case, you would just make a chart, and not "mix it in".

Some charts explicitly want to share draw; often this is the case when you
are combining two charts to visualize two datasets together. In that case, we
might have a "container" chart that mixes together a bar chart and a line
chart:

d3.chart("TwoSets", {
    dataAttrs: ["series1", "series2"],
    initialize: function() {
        var bar = d3.chart("BarChart", this.base.append("g"), {});
        var line = d3.chart("LineChart", this.base.append("g"), {});

        // This is awkward and unintiutive
        bar.transform = function(data) {
            return d3.chart("BarChart").prototype.transform(data.series1);
        };
        line.transform = function(data) {
            return d3.chart("LineChart").prototype.transofrm(data.series2);
        };

        this.mixin("bar", bar);
        this.mixin("line", line);
    }
});

As you're accostumed to, the TwoSets chart's draw method will invoke the
draw method of both BarChart and LineChart. The problem is that, without
the hacky transform trick (which we're currently using in one of our example
charts,
unfortunately) both charts will be given the same data. I say, "unfortunately"
because this is not what's desired, normally. With named mixins, we can
concisely "demux" input data to more intuitively support this use case:

d3.chart("TwoSets", {
    dataAttrs: ["series1", "series2"],
    initialize: function() {
        var bar = d3.chart("BarChart", this.base.append("g"), {});
        var line = d3.chart("LineChart", this.base.append("g"), {});

-       // This is awkward and unintiutive
-       bar.transform = function(data) {
-           return d3.chart("BarChart").prototype.transform(data.series1);
-       };
-       bar.transform = function(data) {
-           return d3.chart("LineChart").prototype.transofrm(data.series2);
-       };

        this.mixin("bar", bar);
        this.mixin("line", line);
+   },
+   demux: function(mixinName, data) {
+       if (mixinName === 'bar') {
+           return data.series1;
+       } else {
+           return data.seris2;
+       }
    }
});

This lets you mix independent charts together more concisely.

Committing to Constructors

From the start, we've tried to implement a "d3-like" API to promote more
consistent code style for people accustomed to d3.js. This meant, among other
things, hiding away constructor functions and supporting more declarative forms
for instantiated charts ("look ma, no new!).

Unfortunately, this approach makes it difficult to use module systems like AMD
or CommonJS (see #41). All along, you've been able to access chart
constructors, but their usage hasn't been documented or backed by tests. In
version 0.2, we'd like to formally define the behavior of the constructors
created by d3.chart and Chart#extend. This won't be an API change but a
formal endorsement of an alternative way of handling Charts.

Simplified Internals

We want to make the internals a little more approachable. To do this, we plan
to remove the variadicNew function and instantiate Charts directly with the
new operator. This will limit the API a bit, though: you won't be able to
write initialize methods with arbitrary arity--all charts will be limited to
a single "options" parameter.

---

So what do you think? It's a lot of change, but we think (given a proper
migration guide), it will be worth it.

[samselikoff]

I haven't had as much time to play with d3.chart lately, but these changes look promising and seem to address needs that arose from real-world use. I particularly like the abstracted data attributes, and I think I'll need to get my hands dirty with the other features to really understand them.

Great job guys monsters, and thanks so much for all your hard work!

[tbranyen]

Hey @jugglinmike this looks really good. One question I have is if the concept of a remove method will make it into 0.2. Something along the lines of what was discussed here: #63

[jugglinmike]

@tbranyen I think that should be minor enough to add in.

[jugglinmike]

Alright, this issue is now a pull request containing a work-in-progress for version 0.2 . Please note that this branch is intended to facilitate discussion--not for general development. Consider it unstable as we will likely rebase and force-push before finally merging.

[iros]

Are we implicitly deciding to manage external packages with npm here? I'm ok with that, but wanted to check that this makes sense going forward too.

[jugglinmike]

This feels kind of gross to me, too, but it seems like the simplest solution to the problem. We already rely on NPM during development--using another package manager would be introducing new development dependencies. I'm open to other approaches, but since this is completely an internal concern (it does not effect consumers of the library in any way), we can change it whenever we want.

[iros]

I really don't have a more elegant solution. I'm ok with this as well tbh, especially since it gets built in. If that weren't the case, maybe we'd have to think about it harder.

[jugglinmike]

It's important that we don't think too hard about anything in this project.

[jeanlauliac]

Did you guys investigate browserify? I know you don't want to lock the project into a module and/or package system, but going the npm/browserify way simplifies things a lot -- and d3.js supports it out-of-the-box. As far as I know this wouldn't lock the users as browserify can generate UMD files (--standalone). The article of Kyle R. is relevant as well.

[jugglinmike]

Thanks, @jeanlauliac! It seems like browserify has been gaining a lot of traction lately. I've used it, but only for a tiny library. It might be just the thing that we need.

[jugglinmike]

True to my word, I've rebased this branch. Here are the latest additions:

• More in-line documentation (updated to also include the d3.selection.prototype.chart method, the dataAttrs option for Chart.extend, and the dataMap option for the Chart constructor)
• An extra assertion to catch when developers attempt to bind to invalid lifecycle events
• A new entry for version 0.2 in the project's changelog.md file
• A new migration.md file to document how to update code from using v0.1.x to v0.2.x

That migration guide includes an abbreviated example of the changes that need to happen in consumer code, so it should help you if you're trying to get a sense of what's going on in v0.2. Remember that it doesn't define the new functionality in the release, though.

[iros]

Wouldn't it be "upward"? From the base up the chain? Just being a total jerk here.

[jugglinmike]

Honestly, I don't know which term is technically correct here. My guess is that this is entirely subjective because the prototype chain has no physical analog. Here's my rationale for this phrasing: in my experience, the phrase "up the prototype chain," is used fairly often when people describe JavaScript's property access mechanism. That would make "down" the direction of base-to-instance.

I'm not satisfied using anecdotal evidence to decide on this, though. I want the documentation to make the most sense to the most people. Is there some more concrete reason to use the opposite wording?

[iros]

Sent in a small pull req here #68.
The rest looks great. Can't wait to start using this!

I'm rewriting the examples, but we'll merge those in a different pull req.
I need to update the misoproject website for 0.2, and then we are good to go. Will leave a note here when that's ready.

[knuton]

This brings some great improvements!

Can you give an estimate for the release date? If I understand the above correctly, you both consider the API changes to be final and only need to work on the website and docs. So it's reasonable to rely on API stability at this point?

[iammerrick]

Lets get this out and about, I want to adopt it! :-D

[jugglinmike]

Hey everyone: thanks for your interest and encouragement! Irene and I just wrapped up a conference, so we finally have the time to release this version and support the transition process. We're shooting for early next week. In the mean time, feel free to experiment with version 0.2 today!

[jugglinmike]

Hi all,

Version 0.2 is nigh, but I wanted to give you a heads-up regarding a
last-minute change: we've removed the abstract data dereferencing API from the
release.

The main reason is this: the limits of the functionality are too subjective.
DataMap was never intended to be a complete data transformation solution. It
is an interesting trick for minor translation operations, but after extended
experimentation, we identified shortcomings/edge cases that would likely cause
confusion for many developers.

It would be a flaw for an API to solve only a subset of a problem. The API
would likely need to grow (or at least change in incompatable ways), and we'd
be spending a lot of time struggling with the same problems that libraries like
Miso DataSet and CrossFilter already address. This tends to violate the Unix philosophy which we regard as one of the more important tenants of software development.

There are always dangers when expanding an existing API, but in light of the
above, we were less inclined to accept these as a compromise:

• It made existing charts more difficult to use. Charts authored for v0.1 do
  not register dataAttrs and could not be used without opting out of
  dataMapping entirely.
• It made the library harder to understand. Encapsulation in a standalone
  module helped a bit, but there's no
  way around the conceptual complexity introduced by the functionality. This
  seems especially significant in the context of a release aimed at
  simplification.
• It introduced a new surface area for bugs.
• It was a commitment to a potentially-extraneous API (see above). If we only
  decided later that it really was out-of-scope, then we would alienate a lot
  of users in removing it.

Not All Doom and Gloom

The feature request was initially motivated by the general problem with data
manipulation. Clearly, we're inclined to leave that to a better tool. But the
reason why we entertained this idea was for mixins. In v0.1, if you mix two
charts (say, the example Bar Chart and the example Chord Diagram)
into a "parent" chart, both "children" receive the same data. If they each
expect a different shape, then you're out of luck; no amount of CrossFiltering
or DataSetting will fix that.

This is the problem that d3.chart itself introduces, and it's the problem
that the library is uniquely positioned to solve.

The new Chart#demux method allows you to preform any operation to data on a
per-attachment basis. Because we implemented DataAttrs first, we only thought
to use Chart#demux for "splitting" input data that comes in the form of:

{
  barData: [],
  chordData: []
}

...but it would be just as easy to include any arbitrary call to a data
manipulation library inside that same hook.

In other words, we've addressed the important problem without DataMap! This
means removing the feature isn't simply watering down a bold release--it's
streamlining it.

But I loved that Feature

We try to be as open as possible about the development of this library. An
unfortunate side effect is that even changing a pre-release version can catch
people off guard. As such, we don't want to abruptly shut the door on any
feature under development.

At the same time, this release has plenty of other bug fixes and new features,
and we don't want to delay their publication any further.

To address these competing needs, we've extracted the DataMap integration
into a standalone commit and opened an issue to facilitate further discussion
(if necessary). You'll find the discussion and commit in issue #82.

Version 0.2 is stronger than ever, and we're gearing up to release it by the
end of this week!