Hi, Bryan!
I'm sorry I didn't give as much context, but I'll try to do that both here, and in a much more detailed GitHub issue.

Axe-core is an accessibility engine for web UI testing. It's a tool that runs the HTML (for the most part) of a website, and then identifies accessibility errors and the pertinent/responsible nodes or elements for those errors. It does this via the axe API.

For the question itself, it wasn't really about CI. It was just a matter of leaving axe-core in the PR or not. It has its usefulness up until a point, and I asked the question because I wasn't sure whether or not we would ever get beyond that point. I'll illustrate even further.

The nature of bokeh is to render data visualizations as <canvas> elements. Axe-core itself can't do anything with <canvas> elements, but there are certain bokeh data visualizations that have some content outside the canvas elements. With those, it can be very helpful.

However, there aren't so many unique problems (I only count about 7 for now) that would necessarily warrant using axe-core right now, but there could be in the future. It's not a certainty, but it's a possibility.

Of course, it can still be removed now and iteratively added later on, but I'll write a detailed report and also attach it to this discussion so that the issues it can currently identify would be easy to reference.

Yes, I agree, but I want to sharpen the question here. If there were zero maintenance cost, then adding these tests wouldn't even be in question. But there's never zero maintenance, and the question for me is about whether the net cast by Axe-core is good enough in this case to justify the maintenance burden of adding it.

I personally lean towards thinking that the maintenance to benefit ratio here is too high, so I would be in favor of documenting the issues found but not integrating Axe-core into the future accessibility testing stack.

This may not have been made clear, but Axe-core is a conservative set of checks that can be run against web pages to find accessibility violations. It can be thought of loosely like a basic spell checker, but for accessibility. As Chiemezuo has documented, its utility for Bokeh is limited (basically because of <canvas>).

How can we help the team make an informed decision re: burden to benefit ratio?

What kinds of ongoing maintenance is expected if the tools are checked in? Can you give some specific examples?

Hi @bryevdv, thanks for that question, and thank you so much for engaging on this subject. I'm going to offer some thoughts, but please know that my position is weakly held, and I will defer to your better judgement concerning Bokeh's maintenance. :)

To me, all code is a burden. For example, if somebody were to change the way the docs are built, they might break the Axe-core checks, and then they would have to figure out how to fix them, even though they were working on docs, not accessibility tests. Dependencies also have to be updated every once in a while.

To be clear, I'm not saying that the maintenance burden is large. I'm emphasizing that the reward is small.

Let's recall two things:

1. Axe-core captures only a fraction of accessibility errors
2. The visual representation of the data that Bokeh provides is opaque to Axe-core because it's rendered with <canvas>

This means that Axe-core can only capture a fraction of a fraction of Bokeh accessibility issues.

I also don't get the sense that this fraction of what is being capturing is of great importance when it comes to actual accessibility barriers to using Bokeh. Let's take the slider, for example. Axe-core is complaining because there isn't anything in the markup that unambiguously associates the text above the slider (e.g., Amplitude) with the slider control. In other words, it wants the slider to have an aria-label or aria-labelledby attribute. While this is undoubtedly a good idea, I don't think it really poses much of an accessibility barrier. This is because a screen reader user navigating the page will likely encounter the title before the slider, so they will know what the slider is for. The more important thing is that the slider is keyboard accessible and communicates its current value to assistive tech, which Bokeh's slider already does (whoever implemented it already was already accessibility-minded, which is awesome!). I'm not saying that it's not important to label the slider. One reason that comes immediately to mind is that screen reader software provides a way to bring up all of the form controls on the page, but if they are not labelled in an accessible way, then this form control quick nav is pretty useless. And I'm sure there are other reasons. But I don't believe it's a showstopper, more like a polish task.

We can take another example, the lack of an accessible name for the Bokeh link icon in the toolbar. In my mind, it's really a minuscule difference that the link reads out "https://bokeh.org" instead of "Bokeh homepage" or something.

And then some things are understated. Like the date picker. The real problem with the date picker is not that it doesn't have an accessible name; it's that it's impossible to use with just a keyboard, and it's challenging to use with a screen reader.

In other words, I feel like this tool, Axe-core, is a bit of a distraction. I think it was worthwhile to investigate its potential. Along the way we discovered and documented its limitations, all of which is valuable.

What I am not sure of is that it is valuable to add it to the codebase. There is the maintenance question, yes, and there's also the fact that this is not quite ready to ship, so there's still some amount of work to do before this is ready for merge. For example, we need to make it clear that the approach depends on the docs being built without actually doing the docs-build step in the test.

If more of Bokeh was testable by Axe-core, I would be in favor of adding it. Also I want to be clear that this is not about Playwright. I still want to add Playwright to the repo. I'm just not sure about this idea of build the docs and pass each example from the docs through Axe-core.

Oh, I should add, for balance and fairness, that the I do think the contrast issues discovered are worthwhile. Perhaps for that reason alone, it might be worth adding.