Why use Portman, in your opinion?

[pongohaja]

I have used Portman for some time to convert an OpenAPI document to a Postman collection (PC), upload the PC to the Postman app, and run the PC with Newman in a CI/CD pipeline.
Assume that our backend source code is updated with a changed API. Then it is compiled and started, the OpenAPI Specification (OAS) is downloaded and a PC is created.
Since the PC is based on the new API, I still wonder what kind of errors are found compared to not using Portman.
My experience so far is that I find deficiencies in the code when adapting the Portman Config file to the new API, but the CI pipeline does not find so much.
What are your experiences?
What kind of faults have you found, or are expected to be found?
What is the main benefit of testing with Portman?

I hope you share your insights with me and others.

[thim81]

hi @pongohaja

With Portman, we tried to provide a tool that automates the maintenance of API tests and assists with the repetitive code you would have to write.

What typically happens:

1. You write/change the code of your API
2. Update the unit tests & API tests
   Rinse & repeat

Next to that, you would also have to maintain your documentation and other documents that your API users would need to be able to use your API.

With a "Spec driven" approach, you put the OpenAPI document at the center and use all the information available in the OAS document to automate as much as possible. You could even go so far as to generate your API code based on the OpenAPI.

This is a diagram of how I typically use OpenAPI as the source for various outputs that are being generated.

You want to offer your API users the best possible DX (developer experience) by making sure that the documentation shows the latest changes, that the changes are all tested, and that what they are experiencing when using your API is the same as what has been described in your documentation. Keeping everything in sync is a time-consuming and repetitive task.
In the diagram, I can focus on maintaining the OpenAPI document and a bit of Portman configuration. I don't have to write any tests or manually update Postman collections or public documentation, these are all generated.

Portman helps you to guard the API contracts so that they match your documentation, which reflects your code (or vice versa).
Portman does this by:

• converting your OpenAPI document into a working Postman collection
• injecting Postman contracts tests with just simple configurations that cover: HTTP response codes, the expected content-type and schema validation
• easily adding "content" checks, to verify if the values returned match with what is expected
• allowing generating "fuzzing" tests. Fuzzing changes the requests (body, query params, ... ) to unexpected values in an effort to cause unexpected behavior and errors in the API response.
• options to change the Postman collection without manually having to alter the Postman every time your API changes, but do it in an automated way

I wrote a blog post about the advantages of "spec-driven" development, and in my experience, using OpenAPI in combination with Portman helps with the automation of such flows and reduces manual maintenance.

https://blog.apideck.com/spec-driven-development-part-1

I also cover the usage and benefits of using Portman in a Podcast with the K6 team:
https://www.youtube.com/watch?v=Be66Db4wHLA

[pongohaja]

Thank you, @thim81, for your descriptive reply.
I see the benefits of a spec-driven approach even though we haven't used it so far.
Our environment creates an OAS from the SW. Then it can be fetched by a call to endpoint "/openapi".
However, after some thinking about what you wrote, we understood what we had done wrong.
Before, when we had modified the SW, we immediately fetched a new OAS and run Portman to generate a new Postman Collection (PC). Consequently, an API not changed by purpose would not be discovered.
Now, we have changed to use the latest working OAS and PC to verify the new SW. In this way, the old contract is used to verify the new SW. In case of a failed run with Newman, we need to find out whether the API has been changed by purpose or not.
Eventually, we fetch a new OAS and generate a new PC and run Newman again.