End-to-end testing and Generative AI

Transforming End-to-End Testing with Generative Agentic Workflows | Harness Blog

\ \ Sushil Kumar\ \ All this author’s posts](/content/authors/sushil-kumar/index.html)

Discover how Generative AI and agentic workflows revolutionize E2E testing, overcoming LLM-assisted coding bottlenecks. Learn about automated intent-based testing, simplified test creation, and visual testing advancements.

The world of software development is abuzz with excitement about how Generative AI is revolutionizing the industry. Thanks to large language models (LLMs), code generation productivity is skyrocketing, leading to an explosion in the volume of code generated. However, all this new code needs to be thoroughly tested before deployment. Testing comes in various forms, with code testing focused on validating whether the new piece of code works as intended. The real challenge lies in conducting end-to-end functional, workflow, and regression testing to ensure that new code not only works well in isolation but also integrates seamlessly with existing application workflows and does not break any existing functionalities. This process is largely manual today, posing a significant bottleneck that could potentially negate the tremendous gains from LLM-assisted coding. So, the pressing question is: how can Generative AI help eliminate these bottlenecks?

Can LLMs Generate Usable E2E Test Code?

The answer clearly lies in automation. But can LLMs help generate test automation code as easily as they generate code? Unfortunately, the situation is more complex. While LLMs are incredibly useful for generating unit test code, they face significant challenges when it comes to end-to-end (E2E) testing.

Why LLMs Excel at Generating Unit Test Code

Unit testing focuses on individual units or components of a software application, ensuring that each part functions correctly in isolation. LLMs are adept at unit testing because:

Well-Defined Inputs and Outputs: Clear inputs and outputs make it easy for LLMs to generate meaningful test cases.
Limited Scope: The narrow focus of unit tests allows LLMs to understand and generate relevant test cases.
Code Context: LLMs can analyze dependencies within a small codebase to create accurate unit tests.

The Complexity of E2E Testing

In contrast, E2E testing involves:

Complex Interactions: Replicating real user interactions like clicking buttons and navigating pages.
Dynamic Elements: Adapting to changing web elements based on user interactions or data states.
Iterative Discovery: Iteratively discovering the next actions based on the application’s current state, requiring adaptiveness and decision-making that LLMs struggle with.

Agentic Workflows: Solving the E2E Testing Problem

What Are Agentic Workflows?

Agentic workflows refer to automated processes that mimic human decision-making and interaction patterns. In the context of E2E testing, an agentic workflow can autonomously navigate through an application, making decisions and adapting to changes in real time, just like a human tester would. These workflows leverage advanced AI techniques to understand the application’s state, determine the next steps, and execute them iteratively until the entire workflow is completed.

Automating E2E Testing with Agentic Workflow

What Can the Agentic Test Automation Workflow Do for You?

Automated Intent-Based Testing

One of the most promising end goals of automation is the ability to exercise the same degree of flexibility and adaptiveness as manual testing. This can address many pains traditionally associated with automation, such as brittle tests that frequently break whenever the UI changes. Intent-based testing allows the system to understand and execute tasks based on the user’s intent, making the automation process more resilient and adaptable to changes.

Dramatically Simplify Test Creation Using Natural Language Prompts

While intent-based testing is a medium-to-long-term goal, Generative AI can significantly speed up and simplify test creation today through the use of natural language commands. Although there has been a surge of recorders that make recording simple steps like clicking on links or buttons easier, the real complexity arises in interactions involving business logic. For instance, on a travel booking site, selecting the flight with the lowest fare or booking a room with the lowest price usually requires writing complex scripts. Natural language commands can simplify these interactions by allowing users to specify their requirements in plain language, reducing the need for complex scripting. For a deeper dive into how natural language can simplify automated tests for dynamic pages, check out this blog post.

Additionally, natural language can help simplify assertions, which are crucial for verifying that the application behaves as expected. This simplification can make it easier to create comprehensive and accurate test cases

Writing Assertions is as easy as asking a question

Automate Visual Testing

UI test automation frameworks often struggle with testing visual elements. However, with the advent of vision models like GPT-V, Generative AI agents can perform visual testing for elements such as canvas bar charts and can even detect visual regressions automatically. This capability expands the boundaries of automation, allowing for more comprehensive testing that includes visual aspects, which are often critical for user experience.

Suggest Test Cases to Makes Your Testing More Comprehensive

Generative AI can automatically generate test cases, particularly around boundary conditions and negative testing. By exploring edge cases and potential failure points, AI-driven testing can ensure a more thorough examination of the software, catching issues that might otherwise go unnoticed. This comprehensiveness can lead to more robust and reliable applications, reducing the risk of post-deployment failures.

AI Generated E2E Test Cases

The Road Ahead

Integrating Generative AI and agentic workflows in testing can transform the software development lifecycle by automating complex testing and simplifying test creation. This technology overcomes bottlenecks in LLM-assisted coding, democratizes testing, and reflects real-world use cases.

As models and agents improve, they will revolutionize test creation and maintenance, acting as powerful users and testers. Harness is leading this transformation with the first Generative AI-powered test automation agent.

Harness' GenAI Test Automation Assistant

At Harness, we are at the forefront of making this transformation happen with the industry’s first Generative AI-powered test automation agent. Our innovative solution leverages the power of Generative AI to automate complex testing processes and simplify test creation, helping you overcome the bottlenecks currently hindering the full realization of LLM-assisted coding productivity.

Join us in revolutionizing the world of software testing and delivery.

‍