Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ct: update auto skip stats when error in suite handled #7487

Merged
merged 1 commit into from
Aug 17, 2023
Merged

ct: update auto skip stats when error in suite handled #7487

merged 1 commit into from
Aug 17, 2023

Conversation

hizel
Copy link

@hizel hizel commented Jul 10, 2023

Hi,

in test case with non-existent group in all() , ct prints Error detected and FAILED, but set exit code set 0 like there was no error here. If ct_run call in ci(like gitlab ci, etc) error can be overlooked.

-module(example_SUITE).
-compile(nowarn_export_all).
-compile(export_all).

all() ->
  [
    {group, exist},
    {group, unexist}
  ].

groups() ->
  [
   {exist, [], [test1]}
  ].

test1(_) ->
  ok.

example ct start

$ ct_run -noshell -dir .  -suite example_SUITE || echo $?


Common Test v1.24 starting (cwd is /home/hizel/tmp/err)


Common Test: Running make in test directories...
Recompile: example_SUITE

CWD set to: "/home/hizel/tmp/err/ct_run.ct@core-hizel.2023-07-10_14.34.38"

TEST INFO: 1 test(s), 0 case(s) in 0 suite(s)

Testing tmp.err.example_SUITE: Starting test, 0 test cases

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Error detected: 'Invalid reference to group unexist in example_SUITE:all/0'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Testing tmp.err.example_SUITE: *** FAILED {ct_framework,error_in_suite} ***
Testing tmp.err.example_SUITE: TEST COMPLETE, 0 ok, 0 failed of 0 test cases

Updating /home/hizel/tmp/err/index.html ... done
Updating /home/hizel/tmp/err/all_runs.html ... done

@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2023 •
edited
Loading

CT Test Results

    2 files    58 suites   1h 26m 52s ⏱️
449 tests 436 ✔️ 13 💤 0 ❌
484 runs  468 ✔️ 16 💤 0 ❌

Results for commit 8f880ec.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@u3s u3s added the team:PS Assigned to OTP team PS label Jul 11, 2023
@u3s u3s self-assigned this Jul 11, 2023
@u3s
Copy link
Contributor

u3s commented Jul 14, 2023

I agree current behavior is not optimal.

Can you fix failing tests? I've spotted problems in:

  • ct_auto_compile_SUITE
  • ct_test_server_if_1_SUITE

@hizel hizel force-pushed the ct-error-in-suite branch 5 times, most recently from 8153ae0 to 29e29dc Compare August 8, 2023 11:31
@hizel
Copy link
Author

hizel commented Aug 8, 2023

Hi, i rewrite code and add test

@u3s u3s added the testing currently being tested, tag is used by OTP internal CI label Aug 16, 2023
@u3s u3s merged commit 4697f9c into erlang:master Aug 17, 2023
15 checks passed
@hizel
Copy link
Author

hizel commented Sep 22, 2023

Hi, apparently i created pr to the wrong branch master, because this patch not included in the latest release 26.1. How can I fix it?

@u3s
Copy link
Contributor

u3s commented Sep 22, 2023

Hi, apparently i created pr to the wrong branch master, because this patch not included in the latest release 26.1. How can I fix it?

Ahh. I've missed that as well. Please create new PR targeted to maint and we will try to sort it out.

@hizel hizel mentioned this pull request Sep 22, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@u3s u3s

Labels
team:PS Assigned to OTP team PS testing currently being tested, tag is used by OTP internal CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hizel @u3s