Our Test Management Journey: From Jira Xray to GitLab

Test management solutions are designed to help both quality and delivery teams effectively manage, organize, and track their testing activities. Dashlane’s quality team recently decided to implement Jira plugins. We settled on Xray for test management needs and GitLab for our source code management and continuous integration. We found that Xray worked well with Jira, but it could not integrate well with GitLab, our source code and CI tool. Using tooling outside the development ecosystem led to the perception that "quality" was consistently viewed as the responsibility of others rather than aligning with our principle that "quality is a shared responsibility for all."

Our objective was to enhance the sense of quality ownership among the delivery teams, concurrently fostering the adoption of a shift-left culture. Teams responsible for building our product features and experiences (“delivery teams”) were not in complete control over their quality lifecycle, often relying on different tools, processes, and people to tackle each stage.

In this pursuit, it became evident that Xray was no longer the tool fit for our current needs. Our discussions with Xray revealed that there were no plans/intentions to develop either a standalone solution or a Gitlab plugin akin to Jira. We sought a solution that could seamlessly interface with Gitlab, where teams were already managing their work.

If we were to persist with using Jira and Xray solely for testing, it could potentially evolve into an independent project. This scenario raised two primary concerns:

The team started looking at a centralized solution that would integrate with GitLab and facilitate the test and defect creation (bugs/issues/incidents), storage, documentation, execution, and—more importantly—efficient tracking of their progress. We looked at solutions that would integrate with GitLab and tell us more about the value of those tests. We also wanted to keep the source code of our automated tests closely connected to the test cases. 

Our selection process comprised of the following stages.

A year has passed since our successful transition from Xray and Jira to Gitlab. The statistics and comparisons provided are grounded in the benchmarking conducted in August 2022.

We looked into various test management solutions featuring strong integrations with GitLab. Our shortlist included Qase, Testmo, Kualitee, Practitest, TestRail, QA Touch, and Tuskr for further investigation.

We evaluated those solutions based on decision drivers and nice-to-have factors.

SSO support: Integration with SSO was a must for us. Due to a lack of support, we dropped QA Touch and Tuskr.

Bi-directional bug sync: We needed bi-directional bug sync between GitLab and test management tools. We dropped Practitest as it only provided uni-directional sync.

User support: Both Qase and Kualitee had excellent user support and were very responsive in getting back to our questions via chat, email, or video. At that time, Testmo only had email support, but they were responsive. 

Test case usability: Qase stood out as the most user-friendly and intuitive, with Testmo following closely in ease of use from our analysis. Kualitee was excluded from consideration as we felt we could get the most out of this solution if we used it for task and project management alongside test and defect management, which didn’t match our requirements. Additionally, Testrail's user experience was deemed less optimal, with the tool's navigation proving challenging based on the feedback from the team overseeing its research.

Documentation: Both Qase and Testmo had excellent documentation and were easy to follow comparatively.

Value to cost: Testmo was offering the best value proposition after we analyzed their price structure.

Integration with our testing from CI: This involves updating the test case with the latest test execution status obtained from the continuous integration tool. Most of the tools we explored have this ability to some extent, either with APIs or a dedicated integration.

Performance: We examined the response time of test case management tools to assess their performance. Notably, Testmo exhibited remarkable speed even under load, with the capability to import approximately 390 tests within a mere 3 seconds.

Exploratory testing: At Dashlane, as part of our shift-left strategy, we started diving more deeply into exploratory testing techniques. Testmo had a specific menu and tooling to support the explorer, which we didn’t come across in other tools.

On-prem deployment: Kualitee and Testrail had this possibility. On-prem deployment provides greater control, security, and compliance for sensitive data and critical applications.

Test plan as a Mindmap: Only QA Touch and Testrail had this option during our exploration, enhancing our ability to better visualize and organize our testing.

Ease of migration from Jira: We only evaluated a few tools for this factor. Qase was an outperformer as it had dedicated integration support from the Xray plugin for Jira.

We chose Testmo as the finalist because it was a simple and easy-to-use solution that had the best value proposition. It also offered better organization of tests and dedicated support for exploratory testing.

However, having a tool outside the GitLab ecosystem is still a challenge: It makes it harder to involve all stakeholders and track progress made on issues since they live outside GitLab Dashboards.

Our first obvious option was to go with GitLab Test Case, introduced in GitLab 13.6.

However, even though this feature was integrated, it was still too immature for our needs. The issue type has no assignees, test executions, priorities, linking-related issues, and so on.

We dropped it as it didn’t match our expectations. Nevertheless, it made us take a deeper look into the GitLab issue structure.

The way we organized our testing in GitLab helped us consider a different approach: We could use GitLab issues as our "Test Management Tool" since it could fit our needs. 

We used GitLab issue type with scoped labels to identify the test cases. To ease test creation and to ensure a consistent format across the organization, we built a test case template to guide the test creation. Preconditions were separate issue types in Xray, but we made it an integrated part of the test cases in GitLab. 

One challenge we encountered was associating the issues (tests) on GitLab with the execution results of tests on Continuous Integration, even though both exist on the same platform. Additionally, there was no option to associate GitLab’s test cases with its execution at the time of our assessment. This meant that we couldn’t know when tests were actually completed.

With Xray, Testmo, and other test case management solutions, we had the capability to directly link the test execution results from GitLab CI to the issue.

We found a hack for tracking test executions by updating a dedicated label, posted by Mek from GitLab. We used scoped labels to mark the test executions.

While this hack isn’t a better experience than using test management tools, it is a manageable solution for our needs.

We used GitLab issue types with scoped labels and bug templates to guide the bug creation process, similar to what we set up with test cases.

On top of that, we used a Slack bot to streamline reporting bugs.

We were using test plans as a separate issue type in Xray, and we decided not to have a dedicated issue for it in GitLab. Instead, we set up test planning to be part of user stories as acceptance criteria.

On top of the default project and group level analytics offered by GitLab, we can create custom analytics by creating yml configuration files as explained here.

For specific graphs that we can't extract from GitLab, including Automation Score, Manual vs Automation breakdown, and Feature/Platform wise breakdown, we need to rely on some visualization tool for charts by extracting data from GitLab. In our case, we use Tableau/Kibana to visualize. (You can see an example dashboard here.)

This has been one drawback of our transition away from Jira. Xray used to provide on-the-go dashboards on their platform like other test management solutions. But in GitLab, we had to push all the issue data into Tableau/Azure workbooks. The information is there, but additional effort needs to be made to put it in an accessible format.

Due to the seamless integration between Jira and Confluence, it is possible to showcase Jira issues directly on Confluence pages. This integration ensures that real-time updates regarding issue status, resolution, summary, execution status, automation status, and more are reflected. All someone has to do is paste the JQL (Jira Query language) in Confluence to get a comprehensive view, eliminating the need to navigate to Jira separately. Unfortunately, we don’t have something quite as integrated with GitLab. While the GitLab connector plugin offers the potential for achieving analogous functionalities, it is important to note that it does not replicate the exact same experience.

Ultimately, we decided to use GitLab issues for test management as it’s easier to involve all stakeholders, requires less context switching, is cost-effective, and we didn’t have to significantly compromise on any decision drivers.

Additionally, GitLab is investing more in test management, so we can anticipate many new capabilities being added soon. For example, test sessions are something we’re looking forward to.

Sagar Uday Kumar

With Dashlane since 2016, Sagar is a Senior Quality Engineering Manager with over a decade of experience in the software industry. He's particularly passionate about Quality, Automation, and DevOps, and he has a Master's of Science in Communication & Computer Security from Telecom ParisTech in France. Currently based in Paris, he hails from Bangalore, the Silicon Valley of India.

Read more