ALICJA SUSKA
SOURCEGRAPH

Code Insights:
Detect & Track

Introducing a new type of Code Insight based on a regular expression capture group that creates data series automatically to track versions of languages and patterns.

PROJECT DETAILS

Company
Year
2022
Platform
Web
My role
Product Designer
Team
Product Manager, Engineering Manager, 6 Developers

PRODUCT

Sourcegraph
Code Intelligence Platform that lets developers search and explore all of their organization's code, perform large-scale changes and gain unique insights about the codebase.
Code Insights
Code Insights reveals high-level information about the codebase, based on both how the code changes over time and its current state. Code Insights is based on Sourcegraph's universal code search, making it precise and configurable. Users can track anything that can be expressed with a Sourcegraph search query: migrations, package use, version adoption, code smells, codebase size and much more, across 1,000s of repositories.

PROBLEM SPACE

ADDING DATA SERIES MANUALLY

Currently, users need to create data series one-by-one in the creation UI. They need to define a name, search query and choose the colour. During the research, we've discovered that users would like the system to automatically detect versions based on the capture group within the search query.

Creation UI: Adding data series manually

Adding data series manually UI
Problem 1: Users can track only known patterns
  • A high-level issue with the current solution is that users can track only what they are able to define, hence what they are aware of;
  • The expectation is that the system will help them discover new information about the code base, not only measure what is already known.
Problem 2: Laborious creation flow
It's a common use case to track versions of languages or packages. In some cases, it means that users need to create tens of entries manually.
Problem 3: Insights require maintenance
  • Every time that new version of a language or patten appears, users need to remember to manually add it to the insight
  • Our goal for Code Insights is to be low-maintenance: set up once and come back to analyse the results as needed

USER FEEDBACK

JOBS TO BE DONE

Why solve these problems?

The customer's level of satisfaction with the current solution is low, both when it comes to setup (manual) and the outcomes (tracking only what they are aware of). It's the most common feature request that would unblock a lot of sales opportunities for big customers.

Customer value matrix

Diagram showing customer value

COMPANY GOALS

Company objective
Increase the MAU (Monthly Active Users)
Project goal
Reduce friction for creating insights that track versions of languages and patterns
Project goal
Increase the number of people regularly exploring insights

Our users

Consumers: IC Devs & Managers
People who are viewing insights are interested in accuracy of data and getting exhaustive results. They want to be able to trust the tool and use Insights to make decisions about the code base and company strategy.
Creators: IC Developers
Majority of insights is going to be created by individual contributors (possibly requested by their manager). Those developers care about the creation speed, seamless workflow and shallow learning curve.

Solution SPACE

CREATION UI

Main differences between the creation forms for existing and new insight types lay in defining data series - multiple query vs. one query and how regex capture groups are interpreted.

Insights types comparison table

Comparison of two types of insights

Creation UI: switching between defining data series manually and automatically

For the v.1 of design, I've decided to allow users switch between two ways of creating data series - manual and from capture groups - using a segmented control button.

Questions and considerations
๐Ÿค” Will users notice the segmented control and understand it correctly and see the difference between manual and capture group type
๐Ÿค” We need to disable 'all repos' for automatically generated insights. Is it disrupting the flow?
๐Ÿค” Going back and forth between two modes if users have already filled out some data may be confusing.

USABILITY TESTING ๐Ÿงช

I've tested the prototype of Creation UI with 7 users:

The results
โœ… 7/7 understood the difference between the manually and automatically added data series
โœ… 6/7 were able to create a query with a regex capture group for automatically generated data series
๐Ÿ‘Ž 3/7 noticed the mode switch and tried it by themselves. They found it confusing to switch after filling out the data
๐Ÿ‘Ž 5/7 didn't notice that 'all repos' was no longer available for capture group mode. When prompted, all of them found it confusing
๐Ÿ’ก 5/7 suggested that insight created from capture group should be a separate type from the manual one

INTRODUCING A NEW TYPE OF INSIGHT

Taking into consideration the results of usability testing and how insights with automatically generated data series are conceptually different from the regular insights (setup, type of query, behaviour, maintenance), we've decided to create a new insight type.

The main challenge is to explain all the differences on the selection screen and do not create a hesitation moment (or a blocker) for the users who had just begun the flow.

I came up with two types of visuals I wanted to test with users: icons and charts. During the usability testing, I wanted to compare reactions to both variants (with the same titles and descriptions to keep the results fair).

Insight type selection - cards with icons

Option 1: Cards with icons

Insight type selection - cards with charts

Option 2: Cards with charts

Questions and considerations
๐Ÿค” Are questions and descriptions clear enough for users to make a good choice in a reasonable time?
๐Ÿค” Do charts help users build an understanding or are they confusing?
๐Ÿค” Do cards look interactive and is it clear how to make a selection?

IMPROVED CREATION UI

Updated creation UI

Updated creation UI: detect & track insight

2nd USABILITY TEST ๐Ÿงช

After making improvements to the creation flow, I've performed 6 more hallway tests with a different group of Sourcegraph engineers (due to time constraints I couldn't test with customers this time).

The results
โœ… 6/6 separate flow for automatically generated data series was clear and helped users navigate the task
โœ… 6/6 successfully created the capture group insights I asked for (detect and track all Java versions)
๐Ÿ‘Ž Names and descriptions of insight types took a long time to analyse and 4/6 users said they were confused
๐Ÿ‘Ž 5/6 users preferred charts than icons although different layouts and number of data series of each chart created a lot of confusion

IMPROVED INSIGHT TYPE SELECTION

Updated insights choice UI

Updated creation flow: choosing the insight type

Improvements
  • Charts with consistent layout and the same number of data series - focus on the shape of lines over time
  • Behaviour-focused names: 'Track changes' and 'Detect & track changes'
  • Updated descriptions with bolded keywords
  • Added example use to better convey the application of each insight type
  • Added 'create' button since cards only didn't look interactive enough

MEASURING THE SUCCESS

KPIs & metrics

Impact on company goals

Adding this highly-requested type of insight should unblock some of the sales deals and make the expansion within existing accounts easier. We will be monitoring the overall insights impact on the bottom line as well as what role 'detect & track' insights play in the process.

Go to the next project
Back to top