Solving D3 Graph Interaction In StatusWolf

Solving D3 Graph Interaction In StatusWolf

Summary

StatusWolf graphs need to support interactive features, this post details how those features exist in the current version of StatusWolf and how they've been updated for the next release to implement something I've not seen done before with d3 - graphs that highlight points on all lines based on the x-axis position of the mouse combined with the ability to click and drag directly on the graph to zoom in.

A Brief Overview

There are many things to consider when building an analytics dashboarding tool, dealing with data sources and the mountains of  information they contain, parsing that information into something understandable and presenting it to the user. Getting that last step right is the key to the whole process. In developing StatusWolf I'm constantly working to refine the graph presentation and interaction, and small changes can make big differences. The current version of StatusWolf includes a widget for searching OpenTSDB data and building line graphs of the metrics. OpenTSDB stores time series metrics, and line graphs are a well understood and appropriate choice for this type of data.

Current StatusWolf Line Chart Format

StatusWolf Graph 1 Graphs in the current release version of StatusWolf (0.8.11) allow for different types of interactions:

  • Highlighting lines by hovering on the legend

StatusWolf Graph 2

  • Hiding/showing lines by clicking on the legend

StatusWolf Graph 3

  • Highlight a point on the line by hovering over it

StatusWolf Graph 4

  • Click and drag to zoom

tech_blog_graph5 The point highlighting in particular is problematic. On graphs with more than a few lines, overlap interferes with the ability to highlight any given point, and creating hover targets that were large enough to hit easily while still remaining small enough to be visually congruent is an issue. In addition, each point is created as a transparent object when the data is loaded and made visible on a mouseover event. These points can quickly become a performance issue on larger datasets as the browser struggles with the number of objects that have now been added to the DOM.

  • Adding the dots was done in a function that created each element and attached the mouseover/mouseout events to toggle visibility and to highlight the entry in the legend:
Code
var dots = widget.svg.g.selectAll('.dots') .data(widget.graph.data) .enter().append('g') .attr('class', 'dots') .attr('data-name', function(d) { return widget.graph.legend_map[d.name]; }); dots.each(function(d, i) { d3.select(this).selectAll('.dot') .data(d.values) .enter().append('circle') .classed('dot', 1) .classed('transparent', 1) .classed('info-tooltip-top', 1) .attr('r', 5) .attr('cx', function(d) { return widget.graph.x(d.date); }) .attr('cy', function(d) { return widget.graph.y(+d.value); }) .attr('title', function(d) { return widget.graph.tooltip_format(d.date) + ' - ' + d.value; }) .style('fill', 'rgba(0, 0, 0, 0.0)') .style('stroke-width', '3px') .style('stroke', function() { return (widget.graph.color(widget.graph.legend_map[d.name])) }) .on('mouseover', function() { var e = d3.event; d3.select(this).classed('transparent', 0); var legend_item = $("span[title='" + $(this).parent().attr('data-name') + "']"); var legend_box = legend_item.parent(); legend_item.detach(); legend_box.prepend(legend_item); legend_item.css('font-weight', 'bold'); }) .on('mouseout', function() { d3.select(this).classed('transparent', 1); $("span[title='" + $(this).parent().attr('data-name') + "']").css('font-weight', 'normal'); }); });

New StatusWolf Line Charts

StatusWolf Graph 6 The next version of StatusWolf is nearly ready and will include usability updates to the graph interaction. Chief among the changes is that point highlighting is now done based on the x-axis position of the mouse pointer within the graph widget and the matching point on each line is highlighted. Solving this while still maintaining the interactions that were working well before took a bit of thought. There were several requirements for the change.

  • Highlighting had to be responsive and work across all lines of the graph
  • Each point must have its attached value visible when highlighted
  • The ability to show and hide lines by clicking on the legend must be maintained
  • The click and drag to zoom function must be maintained

Previously the stack of elements in the graph was roughly like this, in DOM order (items added later appear lower on the list but will be stacked on top of previous elements in the browser):

Code
<svg> <g> <x-axis> <y-axis> <brush container> <dots container for line 1> <dots container for line 2> ... <graph line 1> <graph line 2> ... </g> </svg>

The x-axis and y-axis layers are the grid lines and tick values, the brush container is a transparent rectangle that captures the click and drag events, and then there are the layers for the lines and the dots for each line. When an entry in the legend receives a mouseover event it triggers the corresponding line in the graph to increase in line weight and to move to the front of the stack in the browser (meaning it jumps to the bottom of this structure). Because the lines and points are above the brush container in the browser they actually interfere with the click and drag operation by intercepting the pointer event, but this wasn't a blocking problem because their relatively small target size meant that the brush container still received the majority of the events. I have seen other examples of multi-line highlighting based on x-axis position, so I did some research on solutions for D3 and didn't find a lot out there. Mike Bostock has an nice example of X-Value Mouseover, but it is for a single line only. I saw some questions elsewhere but no answers until I found a blog post by Heap Analytics (Getting the details right in an interactive line graph) with their solution for this very issue. StatusWolf is of course different enough that neither post offered anything like a cut-and-paste solution, but between the two of them they put me on the right track. The first step was to change from adding circle elements for every data point to adding a single circle element that mapped to the first point of each line.

Code
widget.graph.dots = widget.svg.g.append('g') .attr('class', 'dot-container') .selectAll('g') .data(dot_data) .enter().append('g') .attr('opacity', 0) .attr('data-name', function(d) { return widget.graph.legend_map[d.name]; }); widget.graph.dots .append('circle') .attr('cx', widget.graph.x(dot_data[0].values[0].date)) .attr('cy', function(d) { return widget.graph.y(d.values[0].value); }) .attr('r', 4) .style('fill', 'none') .style('stroke-width', '1.5px') .style('stroke', function(d) { return (widget.graph.color(widget.graph.legend_map[d.name])); }) .classed('dot', 1);

Initially I tried to maintain the same tooltip from the previous version for the data point value, but attaching them to the elements and getting them to move along with the highlighted points, while possible, proved problematic so those were scrapped in favor of adding a text element to the same g element that held the circle.

Code
widget.graph.dots.append('text') .attr('x', 0) .attr('y', 0) .style('fill', function(d) { return (widget.graph.color(widget.graph.legend_map[d.name])); }) .style('font-size', '.71em') .style('text-shadow', '2px 2px 2px #000');

Which gave me a structure like:

Code
<g class="dot-container"> <g opacity="0"> <circle cx="0" cy="20" r="4" class="dot" style="fill: none; stroke-width: 1.5px; stroke: rgb(98, 226, 0);"></circle> <text x="0" y="0" style="fill: rgb(98, 226, 0); font-size: 0.71em; text-shadow: rgb(0, 0, 0) 2px 2px 2px"></text> </g> </svg>

The observant will notice that the text element has no actual text contained in it, and that its position is at 0,0 and not near the actual point - those are implemented in the mousemove event. Next I added a transparent rectangle layer at the top of the stack to grab the mousover/mousout/mousemove events so they can be translated into a graph location.

Code
widget.svg.g.append('rect') .attr('height', widget.graph.height) .attr('width', widget.graph.width) .on('mouseover', function() { widget.graph.dots.attr('opacity', 1); widget.svg.g.selectAll('.metric path').attr('opacity', .5); }) .on('mouseout', function() { widget.graph.dots.attr('opacity', 0); widget.svg.g.selectAll('.metric path').attr('opacity', 1); }) .on('mousemove', function() { var x_position = widget.graph.x.invert(d3.mouse(this)[0]), x_position_index = bisect(dot_data[0].values, x_position), closest_timestamp = dot_data[0].values[x_position_index].date;

bisect() is a wrapper for d3.bisector():

Code
bisect = d3.bisector(function(d) { return d.date; }).left;

which finds the index of the closest data point in the values array, and that index is used to get the value of the date variable at that location. The remainder of the callback function in mousemove maps the mouse's x-axis position to coordinates on each line and sets the position parameters for the circle and text elements to move them into place. It also grabs the value of the data point and puts that in the text element.

Code
widget.graph.dots.select('circle') .attr('cx', widget.graph.x(closest_timestamp)) .attr('cy', function(d) { return d.values[x_position_index].value ? widget.graph.y(d.values[x_position_index].value) : widget.graph.y(0); } }); widget.graph.dots.select('text') .text(function(d) { return point_format(d.values[x_position_index].value); }) .attr('x', function(d) { if ((x_position_index / d.values.length) > .85) { return widget.graph.x(closest_timestamp) - 5; } else { return widget.graph.x(closest_timestamp) + 5; } }) .attr('text-anchor', function(d) { if ((x_position_index / d.values.length) > .85) { return 'end'; } else { return 'start'; } }) .attr('y', function(d) { return widget.graph.y(d.values[x_position_index].value) - 2.5; } });

The position of the text element is offset a bit up and to the right of the circle it's related to. There's also logic so that at the far right side of the graph the position of the text element is reverse to be to the left of the circle so that the value always remains visible. StatusWolf Graph 7 Once this was in place it worked like a charm, but created a new issue of its own. Because the rect element to track mouse position was at the top of the stack, it blocked the pointer events from reaching the brush and broke click and drag to zoom. The brush is a g container and has it's own transparent rect, and it uses a brushed event to track the click and drag, so the solution was to remove the dedicated rect for the point highlighting and attach the mouseover/mouseout/mousemove events to the background rect of the brush.

Code
widget.graph.zoombox.call(widget.graph.brush) .selectAll('rect.background') .attr('height', widget.graph.height) .attr('width', widget.graph.width) .on('mouseover', function() { widget.graph.dots.attr('opacity', 1); widget.svg.g.selectAll('.metric path').attr('opacity', .5); }) .on('mouseout', function() { widget.graph.dots.attr('opacity', 0); widget.svg.g.selectAll('.metric path').attr('opacity', 1); }) .on('mousemove', function() { var x_position = widget.graph.x.invert(d3.mouse(this)[0]), x_position_index = bisect(dot_data[0].values, x_position), closest_timestamp = dot_data[0].values[x_position_index].date; widget.graph.dots.select('circle') .attr('cx', widget.graph.x(closest_timestamp)) .attr('cy', function(d) { return d.values[x_position_index].value ? widget.graph.y(d.values[x_position_index].value) : widget.graph.y(0); } }); widget.graph.dots.select('text') .text(function(d) { return point_format(d.values[x_position_index].value); }) .attr('x', function(d) { if ((x_position_index / d.values.length) > .85) { return widget.graph.x(closest_timestamp) - 5; } else { return widget.graph.x(closest_timestamp) + 5; } }) .attr('text-anchor', function(d) { if ((x_position_index / d.values.length) > .85) { return 'end'; } else { return 'start'; } }) .attr('y', function(d) { return widget.graph.y(d.values[x_position_index].value) - 2.5; }); });

StatusWolf uses the brush functionality of D3 in a slightly non-standard way. Typically it's used either with a second, smaller representation of the graph where you can make your selection by clicking and dragging or perhaps with pre-set handles on the selection brush to change the size, or it's used to select and highlight points on a graph. There is a second rect associated with the brush that tracks the size of the selection, typically this is kept visible, allowing you to move the selection or to grab one of its sides and resize it. The extent can be cleared, resetting it and hiding the box and selection handles, but to implement the zoom behavior in this way the action is tied to the brushed event. Calling d3.svg.brush.clear() there nullifies everything and breaks the zoom. The workaround for that is to simply reset it manually.

Code
widget.graph.zoombox.select('rect.extent') .attr('x', 0) .attr('y', 0) .attr('height', 0) .attr('width', 0) .style('pointer-events', 'none'); widget.graph.zoombox.selectAll('g') .style('display', 'none');

The function for highlighting lines when hovering over legend items was also modified to use D3's insert method instead of append so that they will come forward but remain behind the dots and the brush container, fixing the issue with them interfering with pointer events and making sure they don't obscure any of the highlight points.

Code
var new_metric = d3.select('#' + widget.element.attr('id') + ' svg>g').insert('g', 'g.dot-container');

The full code for this new graph interaction is available at the StatusWolf Github repository. The new version will be released soon, in the meantime you can find it in branch v0.9_beta.

Summary

StatusWolf graphs need to support interactive features, this post details how those features exist in the current version of StatusWolf and how they've been updated for the next release to implement something I've not seen done before with d3 - graphs that highlight points on all lines based on the x-axis position of the mouse combined with the ability to click and drag directly on the graph to zoom in.

A Brief Overview

There are many things to consider when building an analytics dashboarding tool, dealing with data sources and the mountains of  information they contain, parsing that information into something understandable and presenting it to the user. Getting that last step right is the key to the whole process. In developing StatusWolf I'm constantly working to refine the graph presentation and interaction, and small changes can make big differences. The current version of StatusWolf includes a widget for searching OpenTSDB data and building line graphs of the metrics. OpenTSDB stores time series metrics, and line graphs are a well understood and appropriate choice for this type of data.

Current StatusWolf Line Chart Format

StatusWolf Graph 1 Graphs in the current release version of StatusWolf (0.8.11) allow for different types of interactions:

  • Highlighting lines by hovering on the legend

StatusWolf Graph 2

  • Hiding/showing lines by clicking on the legend

StatusWolf Graph 3

  • Highlight a point on the line by hovering over it

StatusWolf Graph 4

  • Click and drag to zoom

tech_blog_graph5 The point highlighting in particular is problematic. On graphs with more than a few lines, overlap interferes with the ability to highlight any given point, and creating hover targets that were large enough to hit easily while still remaining small enough to be visually congruent is an issue. In addition, each point is created as a transparent object when the data is loaded and made visible on a mouseover event. These points can quickly become a performance issue on larger datasets as the browser struggles with the number of objects that have now been added to the DOM.

  • Adding the dots was done in a function that created each element and attached the mouseover/mouseout events to toggle visibility and to highlight the entry in the legend:

[code language="javascript"] var dots = widget.svg.g.selectAll('.dots') .data(widget.graph.data) .enter().append('g') .attr('class', 'dots') .attr('data-name', function(d) { return widget.graph.legend_map[d.name]; }); dots.each(function(d, i) { d3.select(this).selectAll('.dot') .data(d.values) .enter().append('circle') .classed('dot', 1) .classed('transparent', 1) .classed('info-tooltip-top', 1) .attr('r', 5) .attr('cx', function(d) { return widget.graph.x(d.date); }) .attr('cy', function(d) { return widget.graph.y(+d.value); }) .attr('title', function(d) { return widget.graph.tooltip_format(d.date) + ' - ' + d.value; }) .style('fill', 'rgba(0, 0, 0, 0.0)') .style('stroke-width', '3px') .style('stroke', function() { return (widget.graph.color(widget.graph.legend_map[d.name])) }) .on('mouseover', function() { var e = d3.event; d3.select(this).classed('transparent', 0); var legend_item = $("span[title='" + $(this).parent().attr('data-name') + "']"); var legend_box = legend_item.parent(); legend_item.detach(); legend_box.prepend(legend_item); legend_item.css('font-weight', 'bold'); }) .on('mouseout', function() { d3.select(this).classed('transparent', 1); $("span[title='" + $(this).parent().attr('data-name') + "']").css('font-weight', 'normal'); }); }); [/code]

New StatusWolf Line Charts

StatusWolf Graph 6 The next version of StatusWolf is nearly ready and will include usability updates to the graph interaction. Chief among the changes is that point highlighting is now done based on the x-axis position of the mouse pointer within the graph widget and the matching point on each line is highlighted. Solving this while still maintaining the interactions that were working well before took a bit of thought. There were several requirements for the change.

  • Highlighting had to be responsive and work across all lines of the graph
  • Each point must have its attached value visible when highlighted
  • The ability to show and hide lines by clicking on the legend must be maintained
  • The click and drag to zoom function must be maintained

Previously the stack of elements in the graph was roughly like this, in DOM order (items added later appear lower on the list but will be stacked on top of previous elements in the browser): [code language="html"] <svg> <g> <x-axis> <y-axis> <brush container> <dots container for line 1> <dots container for line 2> ... <graph line 1> <graph line 2> ... </g> </svg> [/code] The x-axis and y-axis layers are the grid lines and tick values, the brush container is a transparent rectangle that captures the click and drag events, and then there are the layers for the lines and the dots for each line. When an entry in the legend receives a mouseover event it triggers the corresponding line in the graph to increase in line weight and to move to the front of the stack in the browser (meaning it jumps to the bottom of this structure). Because the lines and points are above the brush container in the browser they actually interfere with the click and drag operation by intercepting the pointer event, but this wasn't a blocking problem because their relatively small target size meant that the brush container still received the majority of the events. I have seen other examples of multi-line highlighting based on x-axis position, so I did some research on solutions for D3 and didn't find a lot out there. Mike Bostock has an nice example of X-Value Mouseover, but it is for a single line only. I saw some questions elsewhere but no answers until I found a blog post by Heap Analytics (Getting the details right in an interactive line graph) with their solution for this very issue. StatusWolf is of course different enough that neither post offered anything like a cut-and-paste solution, but between the two of them they put me on the right track. The first step was to change from adding circle elements for every data point to adding a single circle element that mapped to the first point of each line. [code language="javascript"] widget.graph.dots = widget.svg.g.append('g') .attr('class', 'dot-container') .selectAll('g') .data(dot_data) .enter().append('g') .attr('opacity', 0) .attr('data-name', function(d) { return widget.graph.legend_map[d.name]; }); widget.graph.dots .append('circle') .attr('cx', widget.graph.x(dot_data[0].values[0].date)) .attr('cy', function(d) { return widget.graph.y(d.values[0].value); }) .attr('r', 4) .style('fill', 'none') .style('stroke-width', '1.5px') .style('stroke', function(d) { return (widget.graph.color(widget.graph.legend_map[d.name])); }) .classed('dot', 1); [/code] Initially I tried to maintain the same tooltip from the previous version for the data point value, but attaching them to the elements and getting them to move along with the highlighted points, while possible, proved problematic so those were scrapped in favor of adding a text element to the same g element that held the circle. [code language="javascript"] widget.graph.dots.append('text') .attr('x', 0) .attr('y', 0) .style('fill', function(d) { return (widget.graph.color(widget.graph.legend_map[d.name])); }) .style('font-size', '.71em') .style('text-shadow', '2px 2px 2px #000'); [/code] Which gave me a structure like: [code language="html"] <g class="dot-container"> <g opacity="0"> <circle cx="0" cy="20" r="4" class="dot" style="fill: none; stroke-width: 1.5px; stroke: rgb(98, 226, 0);"></circle> <text x="0" y="0" style="fill: rgb(98, 226, 0); font-size: 0.71em; text-shadow: rgb(0, 0, 0) 2px 2px 2px"></text> </g> </svg> [/code] The observant will notice that the text element has no actual text contained in it, and that its position is at 0,0 and not near the actual point - those are implemented in the mousemove event. Next I added a transparent rectangle layer at the top of the stack to grab the mousover/mousout/mousemove events so they can be translated into a graph location. [code language="javascript"] widget.svg.g.append('rect') .attr('height', widget.graph.height) .attr('width', widget.graph.width) .on('mouseover', function() { widget.graph.dots.attr('opacity', 1); widget.svg.g.selectAll('.metric path').attr('opacity', .5); }) .on('mouseout', function() { widget.graph.dots.attr('opacity', 0); widget.svg.g.selectAll('.metric path').attr('opacity', 1); }) .on('mousemove', function() { var x_position = widget.graph.x.invert(d3.mouse(this)[0]), x_position_index = bisect(dot_data[0].values, x_position), closest_timestamp = dot_data[0].values[x_position_index].date; [/code] bisect() is a wrapper for d3.bisector(): [code language="javascript"] var bisect = d3.bisector(function(d) { return d.date; }).left; [/code] which finds the index of the closest data point in the values array, and that index is used to get the value of the date variable at that location. The remainder of the callback function in mousemove maps the mouse's x-axis position to coordinates on each line and sets the position parameters for the circle and text elements to move them into place. It also grabs the value of the data point and puts that in the text element. [code language="javascript"] widget.graph.dots.select('circle') .attr('cx', widget.graph.x(closest_timestamp)) .attr('cy', function(d) { return d.values[x_position_index].value ? widget.graph.y(d.values[x_position_index].value) : widget.graph.y(0); } }); widget.graph.dots.select('text') .text(function(d) { return point_format(d.values[x_position_index].value); }) .attr('x', function(d) { if ((x_position_index / d.values.length) > .85) { return widget.graph.x(closest_timestamp) - 5; } else { return widget.graph.x(closest_timestamp) + 5; } }) .attr('text-anchor', function(d) { if ((x_position_index / d.values.length) > .85) { return 'end'; } else { return 'start'; } }) .attr('y', function(d) { return widget.graph.y(d.values[x_position_index].value) - 2.5; } }); [/code] The position of the text element is offset a bit up and to the right of the circle it's related to. There's also logic so that at the far right side of the graph the position of the text element is reverse to be to the left of the circle so that the value always remains visible. StatusWolf Graph 7 Once this was in place it worked like a charm, but created a new issue of its own. Because the rect element to track mouse position was at the top of the stack, it blocked the pointer events from reaching the brush and broke click and drag to zoom. The brush is a g container and has it's own transparent rect, and it uses a brushed event to track the click and drag, so the solution was to remove the dedicated rect for the point highlighting and attach the mouseover/mouseout/mousemove events to the background rect of the brush. [code language="javascript"] widget.graph.zoombox.call(widget.graph.brush) .selectAll('rect.background') .attr('height', widget.graph.height) .attr('width', widget.graph.width) .on('mouseover', function() { widget.graph.dots.attr('opacity', 1); widget.svg.g.selectAll('.metric path').attr('opacity', .5); }) .on('mouseout', function() { widget.graph.dots.attr('opacity', 0); widget.svg.g.selectAll('.metric path').attr('opacity', 1); }) .on('mousemove', function() { var x_position = widget.graph.x.invert(d3.mouse(this)[0]), x_position_index = bisect(dot_data[0].values, x_position), closest_timestamp = dot_data[0].values[x_position_index].date; widget.graph.dots.select('circle') .attr('cx', widget.graph.x(closest_timestamp)) .attr('cy', function(d) { return d.values[x_position_index].value ? widget.graph.y(d.values[x_position_index].value) : widget.graph.y(0); } }); widget.graph.dots.select('text') .text(function(d) { return point_format(d.values[x_position_index].value); }) .attr('x', function(d) { if ((x_position_index / d.values.length) > .85) { return widget.graph.x(closest_timestamp) - 5; } else { return widget.graph.x(closest_timestamp) + 5; } }) .attr('text-anchor', function(d) { if ((x_position_index / d.values.length) > .85) { return 'end'; } else { return 'start'; } }) .attr('y', function(d) { return widget.graph.y(d.values[x_position_index].value) - 2.5; }); }); [/code] StatusWolf uses the brush functionality of D3 in a slightly non-standard way. Typically it's used either with a second, smaller representation of the graph where you can make your selection by clicking and dragging or perhaps with pre-set handles on the selection brush to change the size, or it's used to select and highlight points on a graph. There is a second rect associated with the brush that tracks the size of the selection, typically this is kept visible, allowing you to move the selection or to grab one of its sides and resize it. The extent can be cleared, resetting it and hiding the box and selection handles, but to implement the zoom behavior in this way the action is tied to the brushed event. Calling d3.svg.brush.clear() there nullifies everything and breaks the zoom. The workaround for that is to simply reset it manually. [code language="javascript"] widget.graph.zoombox.select('rect.extent') .attr('x', 0) .attr('y', 0) .attr('height', 0) .attr('width', 0) .style('pointer-events', 'none'); widget.graph.zoombox.selectAll('g') .style('display', 'none'); [/code] The function for highlighting lines when hovering over legend items was also modified to use D3's insert method instead of append so that they will come forward but remain behind the dots and the brush container, fixing the issue with them interfering with pointer events and making sure they don't obscure any of the highlight points. [code language="javascript"] var new_metric = d3.select('#' + widget.element.attr('id') + ' svg>g').insert('g', 'g.dot-container'); [/code] The full code for this new graph interaction is available at the StatusWolf Github repository. The new version will be released soon, in the meantime you can find it in branch v0.9_beta.