Move to 1.1.0 TES spec compliance

[vsmalladi]

• Add in TES compliance suite to 1.1 and 1.0
• Updated TES compliance suite to update the service info testing for email PR: Update schema to match pattern of contact form or email. Closes #69. elixir-cloud-aai/openapi-test-runner#70 . This now allows our service info to pass for 1.0 and 1.1 compliance.

fixes #254
fixes #479

[vsmalladi]

Fixes #479

[yuliadub]

@vsmalladi according to ADO, all compliance tests failed --- @MattMcL4475 the error seems like the TES api was not reachable, did something change about how we connect to it?

error: Exception Name - OS Error. Type - TestRunnerException. Message - Connection error to POST http://tes:***@tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com/v1/tasks. Details - HTTPSConnectionPool(host='tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com', port=443): Max retries exceeded with url: /v1/tasks (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1135)')))

[BMurri]

error: Exception Name - OS Error. Type - TestRunnerException. Message - Connection error to POST http://tes:***@tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com/v1/tasks. Details - HTTPSConnectionPool(host='tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com', port=443): Max retries exceeded with url: /v1/tasks (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1135)')))

That means that the Certificate Manager in Kubernetes hasn't yet installed the LetsEncrypt certificate, which is definitely an environmental issue. However, by the time the compliance tests run, that cert should have been installed already. We get around that for the deployer's smoke test by using kubectl's port forwarding feature. Maybe we need to do that for the compliance test too.

OTOH, not having a working certificate will prevent the deployment from being used by the end user, so the cert manager issue needs to be dealt with somehow.

[yuliadub]

error: Exception Name - OS Error. Type - TestRunnerException. Message - Connection error to POST http://tes:***@tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com/v1/tasks. Details - HTTPSConnectionPool(host='tesinttest-53111467-5761-05994.southcentralus.cloudapp.azure.com', port=443): Max retries exceeded with url: /v1/tasks (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1135)')))

That means that the Certificate Manager in Kubernetes hasn't yet installed the LetsEncrypt certificate, which is definitely an environmental issue. However, by the time the compliance tests run, that cert should have been installed already. We get around that for the deployer's smoke test by using kubectl's port forwarding feature. Maybe we need to do that for the compliance test too.

OTOH, not having a working certificate will prevent the deployment from being used by the end user, so the cert manager issue needs to be dealt with somehow.

@BMurri thanks for the context. wonder if the failure was a random thing (something with the set up just didnt go right) - will try to rerun and see if that is still happening.

[vsmalladi]

@yuliadub any luck to see if this is fixed on a rerun?

[yuliadub]

@vsmalladi did you try to push an update to this PR to kick off another run? If not, I suggest you do.

from looking at the test running on other branches, it is working: https://malibu00.visualstudio.com/Emmons/_releaseProgress?_a=release-environment-logs&releaseId=6229&environmentId=7183

[vsmalladi]

@yuliadub i have merged in main and tests are running now.

Below are 1.0.0 and 1.1.0 outputs

2024-05-17T17:32:13.0475620Z �[0m
2024-05-17T17:32:13.0476013Z �[33m############################################   Compliance Testing Summary   ############################################�[0m
2024-05-17T17:32:13.0476409Z �[33m#                                                                                                                      #�[0m
2024-05-17T17:32:13.0476765Z �[33m#                                                   Total Tests - 23                                                   #�[0m
2024-05-17T17:32:13.0477195Z �[33m#                        Passed - 16 (1, 2, 4, 5, 6, 7, 9, 10, 11, 12, 13, 15, 17, 20, 21, 22)                         #�[0m
2024-05-17T17:32:13.0477905Z �[33m#                                                    Failed - 0 ()                                                     #�[0m
2024-05-17T17:32:13.0478306Z �[33m#                                        Skipped - 7 (3, 8, 14, 16, 18, 19, 23)                                        #�[0m
2024-05-17T17:32:13.0478690Z �[33m#                                                                                                                      #�[0m
2024-05-17T17:32:13.0479096Z �[33m########################################################################################################################�[0m
2024-05-17T17:32:13.0479309Z �[33m

2024-05-17T17:36:18.8249703Z �[33m############################################   Compliance Testing Summary   ############################################�[0m
2024-05-17T17:36:18.8265191Z �[33m#                                                                                                                      #�[0m
2024-05-17T17:36:18.8265600Z �[33m#                                                   Total Tests - 23                                                   #�[0m
2024-05-17T17:36:18.8266022Z �[33m#                 Passed - 20 (1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 18, 19, 20, 21, 22)                 #�[0m
2024-05-17T17:36:18.8266407Z �[33m#                                                Failed - 3 (8, 17, 23)                                                #�[0m
2024-05-17T17:36:18.8266743Z �[33m#                                                    Skipped - 0 ()                                                    #�[0m
2024-05-17T17:36:18.8267064Z �[33m#                                                                                                                      #�[0m
2024-05-17T17:36:18.8267412Z �[33m########################################################################################################################�[0m

Tests failing now:

• create_task_ignore_error.yml
• list_tasks_full.yml
• filter_task_by_tag.yml: [failed test]:(https://github.com/elixir-cloud-aai/openapi-test-runner/blob/17d44d7874d41cef30b0bef546b32f20596d499a/tests/filter_task_by_tag.yml#L55)

The other two fail due to a response return. That should be solved by PR #664

[yuliadub]

this PR claims to fix #479 but all it does is add the output from the tests, so I am a bit confused -- are there separate PRs that actually edit TES code to fix tests?

for adding support for 1.1 things looks good to me.

[vsmalladi]

@yuliadub yes I had to update the compliance tests with this PR elixir-cloud-aai/openapi-test-runner#70

THat is now merged in so now Service info passes as part of the tests.

[yuliadub]

@vsmalladi I see - so the error was on the compliance test side, not us - I am sorry I totally missed that part. if that is the case, looks good to me. (a brief summary of all that in this PR could have helped clear that up vs just links to the issues, although now I do see the link to the elixir PR in one of the issue comments that I missed earlier)

[vsmalladi]

@yuliadub thanks updated the summary.