Code Insights - Detect and Track (Sourcegraph)

By clicking “Accept All Cookies”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. View our Privacy Policy for more information.

Preferences Deny Accept

Privacy Preference Center

When you visit websites, they may store or retrieve data in your browser. This storage is often necessary for the basic functionality of the website. The storage may be used for marketing, analytics, and personalization of the site, such as storing your preferences. Privacy is important to us, so you have the option of disabling certain types of storage that may not be necessary for the basic functioning of the website. Blocking categories may impact your experience on the website.

Reject all cookies Allow all cookies

Manage Consent Preferences by Category

Essential

Always Active

These items are required to enable basic website functionality.

Marketing

Essential

These items are used to deliver advertising that is more relevant to you and your interests. They may also be used to limit the number of times you see an advertisement and measure the effectiveness of advertising campaigns. Advertising networks usually place them with the website operator’s permission.

Personalization

Essential

These items allow the website to remember choices you make (such as your user name, language, or the region you are in) and provide enhanced, more personal features. For example, a website may provide you with local weather reports or traffic news by storing data about your current location.

Analytics

Essential

These items help the website operator understand how its website performs, how visitors interact with the site, and whether there may be technical issues. This storage type usually doesn’t collect information that identifies a visitor.

Confirm my preferences and close

PRODUCT

Sourcegraph

Code Intelligence Platform that lets developers search and explore all of their organization's code, perform large-scale changes and gain unique insights about the codebase.

Code Insights

Code Insights reveals high-level information about the codebase, based on both how the code changes over time and its current state. Code Insights is based on Sourcegraph's universal code search, making it precise and configurable. Users can track anything that can be expressed with a Sourcegraph search query: migrations, package use, version adoption, code smells, codebase size and much more, across 1,000s of repositories.

ADDING DATA SERIES MANUALLY

Currently, users need to create data series one-by-one in the creation UI. They need to define a name, search query and choose the colour. During the research, we've discovered that users would like the system to automatically detect versions based on the capture group within the search query.

Creation UI: Adding data series manually

Problem 1: Users can track only known patterns

A high-level issue with the current solution is that users can track only what they are able to define, hence what they are aware of;
The expectation is that the system will help them discover new information about the code base, not only measure what is already known.

Problem 2: Laborious creation flow

It's a common use case to track versions of languages or packages. In some cases, it means that users need to create tens of entries manually.

Problem 3: Insights require maintenance

Every time that new version of a language or patten appears, users need to remember to manually add it to the insight
Our goal for Code Insights is to be low-maintenance: set up once and come back to analyse the results as needed

Why solve these problems?

The customer's level of satisfaction with the current solution is low, both when it comes to setup (manual) and the outcomes (tracking only what they are aware of). It's the most common feature request that would unblock a lot of sales opportunities for big customers.

Diagram showing customer value

Our users

Consumers: IC Devs & Managers

People who are viewing insights are interested in accuracy of data and getting exhaustive results. They want to be able to trust the tool and use Insights to make decisions about the code base and company strategy.

Creators: IC Developers

Majority of insights is going to be created by individual contributors (possibly requested by their manager). Those developers care about the creation speed, seamless workflow and shallow learning curve.

CREATION UI

Main differences between the creation forms for existing and new insight types lay in defining data series - multiple query vs. one query and how regex capture groups are interpreted.

Comparison of two types of insights

Creation UI: switching between defining data series manually and automatically

For the v.1 of design, I've decided to allow users switch between two ways of creating data series - manual and from capture groups - using a segmented control button.

Questions and considerations

🤔 Will users notice the segmented control and understand it correctly and see the difference between manual and capture group type

🤔 We need to disable 'all repos' for automatically generated insights. Is it disrupting the flow?

🤔 Going back and forth between two modes if users have already filled out some data may be confusing.

USABILITY TESTING 🧪

I've tested the prototype of Creation UI with 7 users:

4 hallway tests with 2 advanced Sourcegraph users and 2 new hires with limited knowledge about Code Insights
3 customers who were moderate Sourcegraph users (using 2-3 times a week) who had created 1-2 Code Insights in the past

The results

✅ 7/7 understood the difference between the manually and automatically added data series

✅ 6/7 were able to create a query with a regex capture group for automatically generated data series

👎 3/7 noticed the mode switch and tried it by themselves. They found it confusing to switch after filling out the data

👎 5/7 didn't notice that 'all repos' was no longer available for capture group mode. When prompted, all of them found it confusing

💡 5/7 suggested that insight created from capture group should be a separate type from the manual one

INTRODUCING A NEW TYPE OF INSIGHT

Taking into consideration the results of usability testing and how insights with automatically generated data series are conceptually different from the regular insights (setup, type of query, behaviour, maintenance), we've decided to create a new insight type.

The main challenge is to explain all the differences on the selection screen and do not create a hesitation moment (or a blocker) for the users who had just begun the flow.

I came up with two types of visuals I wanted to test with users: icons and charts. During the usability testing, I wanted to compare reactions to both variants (with the same titles and descriptions to keep the results fair).

Insight type selection - cards with icons

Option 1: Cards with icons

Insight type selection - cards with charts

Option 2: Cards with charts

Questions and considerations

🤔 Are questions and descriptions clear enough for users to make a good choice in a reasonable time?

🤔 Do charts help users build an understanding or are they confusing?

🤔 Do cards look interactive and is it clear how to make a selection?

2nd USABILITY TEST 🧪

After making improvements to the creation flow, I've performed 6 more hallway tests with a different group of Sourcegraph engineers (due to time constraints I couldn't test with customers this time).

The results

✅ 6/6 separate flow for automatically generated data series was clear and helped users navigate the task

✅ 6/6 successfully created the capture group insights I asked for (detect and track all Java versions)

👎 Names and descriptions of insight types took a long time to analyse and 4/6 users said they were confused

👎 5/6 users preferred charts than icons although different layouts and number of data series of each chart created a lot of confusion

IMPROVED INSIGHT TYPE SELECTION

Updated creation flow: choosing the insight type

Improvements

Charts with consistent layout and the same number of data series - focus on the shape of lines over time
Behaviour-focused names: 'Track changes' and 'Detect & track changes'
Updated descriptions with bolded keywords
Added example use to better convey the application of each insight type
Added 'create' button since cards only didn't look interactive enough

MEASURING THE SUCCESS

KPIs & metrics

'Detect & track' insights usage measured by the number of insights created
'Detect & track' insights usage measured by the number of hovers and clicks on the chart
Ease of use of 'Detect & track' insights measured by the number of support tickets and customer complaints

Impact on company goals

Adding this highly-requested type of insight should unblock some of the sales deals and make the expansion within existing accounts easier. We will be monitoring the overall insights impact on the bottom line as well as what role 'detect & track' insights play in the process.

Code Insights:
Detect & Track