Hypertest Docs
HyperTestKnowledge Base
  • Quick Set-up
    • Install HyperTest
    • Deploy your application
    • Mirror Traffic
      • Using Kubernetes
      • Using Amazon ECS
      • Using Docker
      • Using Nginx
      • Using Goreplay
        • ElasticBeanStalk Script for Goreplay
      • Using Apache
      • Using IIS
      • Using Istio
      • Using Kong
      • Using HAProxy
      • Others
  • HyperTest Overview
    • Introduction
    • Architecture
  • Detailed Setup Guide
    • Detailed Setup Guide
      • Installation
        • Linux VM
        • Kubernetes Cluster
          • Hardware Requirements
          • Cluster Setup
            • EKS
              • Existing Application Load Balancer
              • Calculate Setup Cost for EKS Cluster
            • GCP
            • AKS
            • Self Managed
              • Microk8s with EKS-D
          • HyperTest Installation
      • Mirror Traffic
      • Configure HyperTest
      • Automate Starting a Test
        • CI Integration
          • GitHub Checks Integration
            • Mandatory checks
          • Gitlab Integration
          • Bitbucket Integration
          • Jenkins Pipeline
            • Jenkins Plugins for PR events
  • Upgrade HyperTest
    • Upgrade HyperTest
      • Linux VM
      • Kubernetes Cluster
  • User Guides
    • Usage Guide
      • Install and Configure HyperTest
        • Install HyperTest
        • Configure Base DNS
        • Add New Service
      • Tests Runs and Analysis
        • View Test Cases
        • Start New Test Run
        • Understand Your Test Run Analysis
        • Prioritize Your Error Types
        • Track Bugs
        • Ignore Errors/Desired Changes
        • View/Download Report
        • View Consolidated Dashboard Reports
        • Sign-off Reports
        • Reduce Execution Time
        • Deduplication Rules
      • Troubleshooting and FAQs
    • Best Practices Guide
      • Cost Optimization
    • Dashboard Tour
      • Dashboard
      • All Sessions
      • Regression Report
      • Notifications
      • First Test Run
      • Interactions
      • Custom Analysis
      • Configuration
    • User Management
      • Create Admin User
      • Roles, Groups & Users
      • Enabling User Signup
    • Other Guides
      • Basic Nginx Auth for Linux HT
  • Middleware
  • Advanced Features
    • Import test cases from Postman
  • Change Log
  • Troubleshooting
  • FAQs
    • Setup
    • General
    • Regression Report
  • Glossary
    • Session
Powered by GitBook
On this page
  • Can each developer run parallelly their own test using their own cluster?
  • Do you support multiple services or dashboards?
  • How will traffic mirroring work if I opt for k8s instead of VM setup in the cloud?
  • How are Test cases/ User Sessions recorded?
  • How does HyperTest group various requests as a single-user session/test case?
  • What are incidents?
  • What are API regressions?
  • What is deduplication?
  • What is a time window?
  • What are saved sessions (test cases)?
  • What are bookmarks?
  • What happens to requests older than the time window?
  • What are permanent rules?
  1. FAQs

General

PreviousSetupNextRegression Report

Last updated 1 year ago

Can each developer run parallelly their own test using their own cluster?

HyperTest tests every repo/service individually. When set up HyperTest creates a master dashboard that controls/creates/destroys these service dashboards from a central UI. So devs would set up HyperTest separately across individual micro-services and access regression and performance reports for them through dedicated dashboards

Do you support multiple services or dashboards?

HyperTest tests each repo/service but creates a master dashboard that controls all individual service dashboards. This master dashboard is created by default on any of your node groups inside your k8s cluster during HyperTest set-up. Once created, all individual service dashboards can be launched from the master dashboard UI.

How will traffic mirroring work if I opt for k8s instead of VM setup in the cloud?

HyperTest supports various methods of mirroring traffic from your k8s node groups;

It can mirror traffic from your service meshes on istio, refer guide

Can set up as a side-car inside your cluster across node groups, refer guide

If incoming traffic terminates outside your k8s cluster, say a load balancer or proxy then mirroring is rather straightforward, refer guide

How are Test cases/ User Sessions recorded?

HyperTest automatically generates test cases (user sessions) from the application’s network traffic. When set up with your load balancer or reverse proxy, it asynchronously logs all incoming requests. The requests that denote a unique flow are then grouped together as a test case using custom session differentiator values like IP add, session cookies or anything else

How does HyperTest group various requests as a single-user session/test case?

HyperTest differentiates sessions using unique differentiators. Some of the commonly used differentiators are:

  1. The IP address of the user

  2. Session cookie Mod-headers as chrome or android plugins.

  3. Set up in the user browser or app when testing.

What are incidents?

Incidents are differences between the expected output and real output of any API. HyperTest generates the expected output of the API from its master branch and the real output of the same API from the test branch. These outputs are compared in real-time. The most common type of incidents HyperTest reports are;

  1. Status_code_change

  2. Key_removed

  3. Key_added

  4. Data_type_change

  5. Value_modified

  6. Array_order_change

  7. Header_added

  8. Header_removed

  9. Header_modified

Go to settings and click on Incident type to get the complete list of all incidents.

What are API regressions?

Integration tests of HyperTest catch breaking changes as API regressions. These breaking changes have the potential to damage the application irreparably leading to patches or hotfixes in production, and even rollbacks. When dev is making a new change across any of their repos, as a side-effect API regression can either;

  1. That effect a lot of APIs together like 500 status code error or,

  2. Cause more than one incident in a single API at the same time

View the complete list of different regressions that HyperTest reports

What is deduplication?

HyperTest can generate hundreds and thousands of test cases (use sessions) from any application’s network traffic. But these user sessions would be duplicacy of 2 kinds;

  1. Duplicate requests (similar schema, data, parameters) inside a single session

  2. The duplicate session with exactly similar requests (schema, data, parameters)

HyperTest intelligently de-duplicates non-unique requests in a single session (test case) and then non-unique sessions. Hence, only runs unique requests in sessions and only unique sessions. This ends up saving a median of 90% in execution time by completing these tests in minutes vs hours.

What is a time window?

HyperTest logs incoming traffic and stores request data as test data. Since HyperTest monitors your application 24/7 it keeps receiving newer test data for the same requests.

Take the example of login. HyperTest will keep receiving newer, more fresh login request data. To avoid creating a very large and unmanageable test data store HyperTest keeps retiring older request data but only when it receives new data for the same request. Hence it asks teams to set a time period which is roughly the longest time your application would need to send newer data for the same requests. It is by default set to 90 days but can be changed from the service configuration.

What are saved sessions (test cases)?

HyperTest generates hundreds of sessions (test cases) from your application’s network. Each session is a defined sequence of API calls, called exactly in that order when run as a test case. Each API request carries the data to execute that API call. HyperTest keeps retiring older request data (check test window) because it keeps receiving newer request data from your application.

Take the case of login. When testing that flow, HyperTest would retire older login data when it receives new login data from your app. This automatic replenishment of test data eliminates the problem of test data maintenance. But there could be some specific flows in your app which are executed rarely from your app and HyperTest would likely not receive new request data that can retire the old.

In such cases, there is an option to save certain flows permanently. Saved sessions (test cases) freeze the request data in time giving you the ability to re-run such cases with the same data set again and again.

What are bookmarks?

When HyperTest executes a test, it could possibly run several hundred test cases (user sessions) together. But there would be times when you would want to run only a few test cases (sessions) at a time and not the entire suite.

If that need arises, you can bookmark a session. HyperTest locally saves this session on your browser. Next time you start a test, you can choose to run only these sessions (test cases) from your entire suite which are called bookmarked sessions (test cases) Please make a note that these sessions are different from saved sessions.

What happens to requests older than the time window?

Requests older than the time window are retired automatically, but only when a newer request with fresh data is not received.

What are permanent rules?

HyperTest gives teams a simple way to customise their regression report to only include those regressions which are of interest to them. To view the complete list of API regressions HyperTest reports please check error types from settings.

Despite this, there could be a possibility that teams would like to build custom rules to silence a certain manifestation of a regression. Consider the following examples:

  1. Data_Type_Change: Report a regression only if the key value changes from a value (expected) to null (real) from original to new. A permanent rule can be made to this effect

  2. Response code error: In the majority of cases, when there is a new API available the original version (not present) will give a 404 and the new version would return a 200. A permanent rule can suppress reporting this as a regression.

this
this
this