-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test Coverage API #123713
Comments
Would it be possible to load the coverage data directly from a file on disk (or in the remote workspace, whatever we want to call it) rather than as a result of the output of the test run? It seems like it'd be pretty limiting to only have coverage if it were provided by a test runner extension, if I'm understanding your current API draft correctly. In my codebase for work, the test suite takes well over 90 minutes to run in full. So it pretty much only gets run in CI (parallelized, obviously), which means that the coverage is only available as a CI artifact. It never exists locally (at least, not from a full run). The more likely case where this might be useful would be if you ran your test suite locally (because it was fast enough), but not using a VS Code Test Explorer extension. So then you just had the |
Yes, the APIs are designed for this 🙂 You can call |
also: I'm pretty interested in getting the CI story right, so I'm glad to have someone who has an existing use case and need for it! Please let me know if this or any other parts are confusing or seem to present roadblocks for you. |
I'd make sure to document the coverage-from-file use-case specifically (I know this is an API draft and actual docs are a while away, but I think it's important 😄). Regarding CI, probably a crazy idea, but: It'd be nice to point VS Code at a URL where it can get the coverage file (maybe it passes a param with the current branch name as well) so I wouldn't have to download the coverage.xml myself manually. But that'd have to be custom-built to redirect to the latest CI artifacts somehow since there's no stable URL that CircleCI (or any other CI system) provides for something like this, as far as I know. And maybe that should be a capability handled by an extension rather than as something in core VS Code. 🤷♂️ |
Yep, that's a good idea for an extension |
I have a question about the FileCoverage API. If I am looking at a coverage report, it looks like I only know the file URI. So, if someone starts a coverage test run (which may take a while to run) then makes any changes to the file the statements may not line up correctly with the current editor. In the TextDocument API there is a Should we track FileCoverage to the specific version of the file the test run was started with? Maybe as an optional param? |
While we could store the version, there's not a lot of useful things we can do with it (and it gets reset if the file is opened and closed). The most reliable way to do this would be for the extension to use |
The field For example, a method or a line might have 4 branches, but only 2 of them are executed. Take the In editor each line can also have this metadata Updateforget about it, I just realized, if the |
For reference, there is an updated API with migration guide that was posted: #122208 (comment) |
I implement the coverage API and data flow for it internally. There's no UI for it, but it's there to try out. At some point in the next few weeks UI will magically show up ✨ |
Hi, just to update folks, we bumped this API to target finalization next iteration, rather than the this one (release next week.) This is a sizable API and there were several other exciting APIs entering proposed and finalization that took up our team's bandwidth. |
@connor4312 appreciate the hard work regardless, thanks! |
I think I ran into a bug, trying to use the coverage API to integrate coverage data loaded from a file. I am trying to follow the instructions which were previously provided up-thread ScenarioI am creating two test runs. Those test runs have no test cases attached, they only contain a name, some output ( Minimal reproimport * as vscode from "vscode";
class BazelCoverageProvider implements vscode.TestCoverageProvider {
provideFileCoverage(
token: vscode.CancellationToken,
): vscode.ProviderResult<vscode.FileCoverage[]> {
const detailedCoverage = [
new vscode.DeclarationCoverage(
"test_func",
12,
new vscode.Position(1, 12),
),
new vscode.DeclarationCoverage(
"test_func2",
0,
new vscode.Position(10, 0),
),
new vscode.StatementCoverage(true, new vscode.Position(15, 3)),
new vscode.StatementCoverage(false, new vscode.Position(16, 0)),
];
return [
vscode.FileCoverage.fromDetails(
vscode.Uri.file("/Users/avogelsgesang/hyper/hyper-db/hyper/ir/IR.cpp"),
detailedCoverage,
),
];
}
// Already fully resolved. Noting to resolve
resolveFileCoverage?(
coverage: vscode.FileCoverage,
token: vscode.CancellationToken,
): vscode.ProviderResult<vscode.FileCoverage> {
return coverage;
}
}
export async function activate(context: vscode.ExtensionContext) {
// Create the test controller
const testController = vscode.tests.createTestController(
"bazel-coverage",
"Bazel Coverage",
);
context.subscriptions.push(testController);
// Provide coverage info 1
const run = testController.createTestRun(
new vscode.TestRunRequest(),
"/my/coverage.lcov",
false,
);
run.appendOutput("Information about where we loaded the coverage from...");
run.coverageProvider = new BazelCoverageProvider();
run.end();
// Provide coverage info 2
const run2 = testController.createTestRun(
new vscode.TestRunRequest(),
"/my/second_coverage.lcov",
false,
);
run2.appendOutput("Some other information...");
run2.coverageProvider = new BazelCoverageProvider();
run2.end();
} Observed behaviorI see two coverage entries in the test result list, and I can switch between them. Also, I see the output from the 2nd test run. However, I do not see the test run names "/my/coverage.lcov" and "/my/second_coverage.lcov" anywhere. Also, there seems to be no way to see inspect the Expected behaviorI guess the "Close test coverage"/"View test coverage" should be grouped under a test run? Even if there are no tests and only coverage data inside this test run? Also, it would be good to be able to set the overall test run into a "success"/"failed" state, even without listing explicit sub-tests |
@connor4312 Will the API change #207512 affect the coverage API stable release ETA? Do we still target on March release now? |
We're still targeting the March release, this API will be finalized in the next few hours. @vogelsgesang I'll check that out, I've opened a new issue for tracking. |
It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one. This is not supported from this API, as it mostly brings total coverage across all tests. |
Tracked in #212196 |
📣 2024-03 update: test coverage support has been finalized. You can find the finished, usable version of the API in
vscode.d.ts
and a guide in our docs.Original discussion has been preserved below.
Notes researching of existing formats:
Clover
<line num="24" count="5" type="stmt"/>
<line num="12" count="0" type="cond" truecount="0" falsecount="3"/>
. Either truecount/falsecount being 0 indicates incomplete branch coverage.gcov
Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php
DA:<line number>,<execution count>[,<checksum>]
BRDA:<line number>,<block number>,<branch number>,<taken>
, declared multiple times for line, one for each branchcobertura
<line number="21" hits="58" branch="true" condition-coverage="50% (1/2)"/>
<line number="20" hits="5" branch="false"/>
This results in the following API. The
TestCoverageProvider
is given as an optional provider on the managed TestRun object. This is similar to theSourceControl
interface in vscode.d.ts. The provider is then pretty standard; it'll only be examined when the run finishes, so all its data is static.It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.
I refer to the atomic unit of coverage as "statement coverage". The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it's called statement coverage, but it can be used as line coverage.
The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don't like the method names on the TestCoverageProvider, but have not thought of better ones yet.
The text was updated successfully, but these errors were encountered: