General
Can each developer run parallelly their own test using their own cluster?
HyperTest tests every repo/service individually. When set up HyperTest creates a master dashboard that controls/creates/destroys these service dashboards from a central UI. So devs would set up HyperTest separately across individual micro-services and access regression and performance reports for them through dedicated dashboards
Do you support multiple services or dashboards?
HyperTest tests each repo/service but creates a master dashboard that controls all individual service dashboards. This master dashboard is created by default on any of your node groups inside your k8s cluster during HyperTest set-up. Once created, all individual service dashboards can be launched from the master dashboard UI.
How will traffic mirroring work if I opt for k8s instead of VM setup in the cloud?
HyperTest supports various methods of mirroring traffic from your k8s node groups;
It can mirror traffic from your service meshes on istio, refer this guide
Can set up as a side-car inside your cluster across node groups, refer this guide
If incoming traffic terminates outside your k8s cluster, say a load balancer or proxy then mirroring is rather straightforward, refer this guide
How are Test cases/ User Sessions recorded?
HyperTest automatically generates test cases (user sessions) from the application’s network traffic. When set up with your load balancer or reverse proxy, it asynchronously logs all incoming requests. The requests that denote a unique flow are then grouped together as a test case using custom session differentiator values like IP add, session cookies or anything else
How does HyperTest group various requests as a single-user session/test case?
HyperTest differentiates sessions using unique differentiators. Some of the commonly used differentiators are:
The IP address of the user
Session cookie Mod-headers as chrome or android plugins.
Set up in the user browser or app when testing.
What are incidents?
Incidents are differences between the expected output and real output of any API. HyperTest generates the expected output of the API from its master branch and the real output of the same API from the test branch. These outputs are compared in real-time. The most common type of incidents HyperTest reports are;
Status_code_change
Key_removed
Key_added
Data_type_change
Value_modified
Array_order_change
Header_added
Header_removed
Header_modified
Go to settings and click on Incident type to get the complete list of all incidents.
What are API regressions?
Integration tests of HyperTest catch breaking changes as API regressions. These breaking changes have the potential to damage the application irreparably leading to patches or hotfixes in production, and even rollbacks. When dev is making a new change across any of their repos, as a side-effect API regression can either;
That effect a lot of APIs together like 500 status code error or,
Cause more than one incident in a single API at the same time
View the complete list of different regressions that HyperTest reports
What is deduplication?
HyperTest can generate hundreds and thousands of test cases (use sessions) from any application’s network traffic. But these user sessions would be duplicacy of 2 kinds;
Duplicate requests (similar schema, data, parameters) inside a single session
The duplicate session with exactly similar requests (schema, data, parameters)
HyperTest intelligently de-duplicates non-unique requests in a single session (test case) and then non-unique sessions. Hence, only runs unique requests in sessions and only unique sessions. This ends up saving a median of 90% in execution time by completing these tests in minutes vs hours.
What is a time window?
HyperTest logs incoming traffic and stores request data as test data. Since HyperTest monitors your application 24/7 it keeps receiving newer test data for the same requests.
Take the example of login. HyperTest will keep receiving newer, more fresh login request data. To avoid creating a very large and unmanageable test data store HyperTest keeps retiring older request data but only when it receives new data for the same request. Hence it asks teams to set a time period which is roughly the longest time your application would need to send newer data for the same requests. It is by default set to 90 days but can be changed from the service configuration.
What are saved sessions (test cases)?
HyperTest generates hundreds of sessions (test cases) from your application’s network. Each session is a defined sequence of API calls, called exactly in that order when run as a test case. Each API request carries the data to execute that API call. HyperTest keeps retiring older request data (check test window) because it keeps receiving newer request data from your application.
Take the case of login. When testing that flow, HyperTest would retire older login data when it receives new login data from your app. This automatic replenishment of test data eliminates the problem of test data maintenance. But there could be some specific flows in your app which are executed rarely from your app and HyperTest would likely not receive new request data that can retire the old.
In such cases, there is an option to save certain flows permanently. Saved sessions (test cases) freeze the request data in time giving you the ability to re-run such cases with the same data set again and again.
What are bookmarks?
When HyperTest executes a test, it could possibly run several hundred test cases (user sessions) together. But there would be times when you would want to run only a few test cases (sessions) at a time and not the entire suite.
If that need arises, you can bookmark a session. HyperTest locally saves this session on your browser. Next time you start a test, you can choose to run only these sessions (test cases) from your entire suite which are called bookmarked sessions (test cases) Please make a note that these sessions are different from saved sessions.
What happens to requests older than the time window?
Requests older than the time window are retired automatically, but only when a newer request with fresh data is not received.
What are permanent rules?
HyperTest gives teams a simple way to customise their regression report to only include those regressions which are of interest to them. To view the complete list of API regressions HyperTest reports please check error types from settings.
Despite this, there could be a possibility that teams would like to build custom rules to silence a certain manifestation of a regression. Consider the following examples:
Data_Type_Change: Report a regression only if the key value changes from a value (expected) to null (real) from original to new. A permanent rule can be made to this effect
Response code error: In the majority of cases, when there is a new API available the original version (not present) will give a 404 and the new version would return a 200. A permanent rule can suppress reporting this as a regression.
Last updated