Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ESD-2010] Fix auto generated tests #955

Merged
merged 2 commits into from
Apr 23, 2021
Merged

Conversation

woodfell
Copy link
Contributor

@woodfell woodfell commented Apr 22, 2021

Fix autogenerated C tests

The tests as currently written are not really testing very much. Each test case is passed in a byte array from the spec which represents the encoded frame. This data is then passed in to sbp_send_message as the payload rather than just the payload being extracted and passed in. This means that the payload as encoded by sbp_send_message is not the actual payload which relates to the message under test but a complete frame which contains the payload:

If the test case for message X is passed a particular set of test data it should be encoded as

PREAMBLE | SENDER | TYPE | LEN | PAYLOAD | CRC

but it's actually encoded as

PREAMBLE | SENDER | TYPE | LEN | PREAMBLE | SENDER | TYPE | LEN | PAYLOAD | CRC | CRC

The encoded frame is then passed back in to libsbp and the payload extracted via a logging callback - the payload which is actually an entire frame. The "payload" is then compared against the encoded frame data defined in the test spec. The first part of each generated test case is pretty much just testing memcpy

The second part of the test case uses a series of expected fields and compares them against the encoded data. However it checks against the data which was passed in from the spec. This part is therefore checking that the test spec is correctly written and not checking anything in libsbp itself.

These things combined mean that libsbp is not being tested at all. This is demonstrated by a test case for MSG_VEL_NED_COV which has an incomplete set of expected fields and should be failing but isn't.

This PR reworks the auto generated C test so that the procedure for each case is:

1 - Create an instance of the message struct and fill out with the fields from the test spec
2 - Encode the test message using sbp_send_message
3 - Check the encoded data from sbp_send_message against the expected encoded frame data from the test spec
4 - Check that the SBP message callback was called properly
5 - Check that the SBP frame callback was called properly
6 - Cast the decoded payload back to the correct message struct and verify the fields.

By doing this it checks that
1 - The test spec is well written
2 - libsbp is behaving properly

Variable length arrays need special handling and complicated some of the macros in the test suite. The length of the encoded message is tracked while the test message is being built up since sizeof for messages containing variable length arrays is meaningless.

Strings are similarly badly defined in SBP with some having no termination, single null termination, double null termination, or null delimination. These are now all handled with memcpy/memcmp rather than strcpy/strstr

For reviewers:
The only files in this PR which have been modified by hand are
generator/sbpg/targets/resources/sbp_c_test.c.j2
generator/sbp/targets/test_c.py
spec/tests/yaml/swiftnav/sbp/navigation/test_MsgVelNEDCOV.yaml

Everything else is auto generated

@woodfell woodfell changed the title Fix auto generated tests [ESD-2009] Fix auto generated tests Apr 22, 2021
@woodfell woodfell changed the title [ESD-2009] Fix auto generated tests [ESD-2010] Fix auto generated tests Apr 22, 2021
Copy link
Contributor

@RReichert RReichert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, good spotting this issue.

@woodfell woodfell merged commit c463fa9 into master Apr 23, 2021
@woodfell woodfell deleted the woodfell/fix_autocheck branch April 23, 2021 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants