ALICJA SUSKA
SOURCEGRAPH

Code Insights:
from Beta to GA

Creating a new product that surfaces information about how the codebase changes over time to help engineering leaders make better decisions.

PROJECT DETAILS

Company
Year
2021-22
Platform
Web
My role
Product Designer
Team
Product Manager, Engineering Manager, 5 Developers

PRODUCT

Sourcegraph
Code Intelligence Platform that lets developers search and explore all of their organization's code, perform large-scale changes and gain unique insights about the codebase.
Code Insights
Code Insights reveals high-level information about the codebase, based on both how the code changes over time and its current state. Code Insights is based on Sourcegraph's universal code search, making it precise and configurable. Users can track anything that can be expressed with a Sourcegraph search query: migrations, package use, version adoption, code smells, codebase size and much more, across 1,000s of repositories.

THE POC (2020)

THE ORIGINS

In early 2020, Code Insights were created as a POC by one of Sourcegraph's engineers.

He learned that some customers manually track search results count to understand their code better. He proposed to automate this for the users by creating a new sub-product that tracks search results for a given query over time and visualises it.

Sourcegraph has an extremely precise search engine so the company was uniquely positioned to provide users with accurate insights about their code base.

the 'why'

  • Expanding the target audience by serving new use cases
    So far, Sourcegraph has been heavily focusing on winning the IC devs - most of the features were the most appealing to individual engineers. Insights can be useful for engineering leaders: managers, tech leads, CTOs, etc. Those are the decision makers who ultimately buy Sourcegraph for their company.
  • Alignment with the product strategy of becoming a code intelligence platform
    The long-term vision for Sourcegraph is to surface high-level information about the code to the users. We want to help people make decisions and support them with the necessary context. In addition, Sourcegraph should become a code intelligence platform by combining code search with other features built on top of it.

USER EXPERIENCE (v.0)

I've differentiated three functional spaces that we could use for extensions: top horizontal, bottom horizontal and vertical. Each of those spaces has its pros and cons in relation to the needs of the extension action items.

Viewing Code Insights
  • Only one general dashboard view - no grouping
  • Chart datapoints are not interactive; they don't lead to search results
  • Issues with charts performance, accessibility, and lack of flexibility due to the chosen charting library
Proof of concept for code insights

The first version of the Code Insights general dashboard

Creating & Editing Insights
  • Configuring insights possible only via user settings editor. There is no dedicated creation UI
  • Users needed to add and fill in a special JSON property describing their new insight. There was no validation, autocomplete or hints available.
JSON file in user settings

User settings editor where users had to configure insights manually

BETA (09-2021)

IDEA VALIDATION

After creating the POC and getting feedback from some of our users, we ha enough market validation to start planning for Beta. Together with the PM and EM, we needed to perform wide product research to define what we want to build, why, for whom and how.

Exploratory research (scope)
  • Problems Insights can solve - define the most popular use cases to focus on first
  • Target audience - who should we focus on: ICs or managers
  • Business and user value - how does insight fit into our existing strategy
  • Market analysis - competitors, similar products, existing workarounds
  • *Implementation-related research (excluded from this case study)
Methodologies
  • Interviews with existing customers who expressed interest in Insights or fall into the company profile we're targeting
  • Interviews with sales team and prospects
  • Desk research - competitors, other solutions, similar tools, etc.
Validated hypothesis
  • Insights hold value for nearly every level of an engineering organisation
  • Use cases that resonated the most: migration, adoption, depreciation, versions (languages, packages, patterns, etc.), code reuse, code health, security
  • Sourcegraph is uniquely positioned to provide many of those insights because of its robust and accurate search
  • Insights are valuable enough to positively impact our sales process and the bottom line

CREATION UI

After talking to some of the Insight's early adopters among out customers, we quickly validated that creating insights via JSON in user settings is the biggest blocker. For the Beta release to be successful, we needed to provide users with a creation UI that would guide them through the process.

Step 1: type of insight
  • We've differentiated 3 types of insights that vary when it comes to the setup: based on custom search query, language insight and based on extensions
  • In the first step of the creation flow, users need to choose the path. I've also used it as a moment to showcase the graphs and what Insights can be can be used for.
Types of insights

Creation UI: Choosing the type of Insight

Step 2: Creation UI

After talking to some of the Insight's early adopters among out customers, we quickly validated that creating insights via JSON in user settings is the biggest blocker. For the Beta release to be successful, we needed to provide users with a creation UI that would guide them through the process.

Creation UI: Flow of creating an Insight

Repositories
  • Users can list up to 20 repositories
  • We support users with autocomplete and field validation
  • Users can select to run insight over all their repositories
Repositories UI

Creation UI: Targeted repositories

Data series
  • Users define each data series individually
  • Data series is based on Sourcegraph search query - everything that can be searched for, can be tracked as a Code Insight
  • Users can choose a name and a color for each data series
Data series UI

Creation UI: Data series

Live preview
  • For scope up to 20 repositories, we are able to show the insight preview as user is creating it
  • Live preview prevents creating 'in the dark'. We got a lot of positive feedback about this solution during usability tests
Insight live preview

Creation UI: Live preview

Chart settings
  • Users can adjust granularity of the chart and its name
  • Taking into consideration the variety of desired timelines that came out of our research, I've decided to keep the granularity as customisable as possible, without enforcing any default values
Chart settings UI

Creation UI: Chart settings

USABILITY TESTING

Creation UI went through two rounds of usability testing with internal and external testers. Key findings:
  • I moved naming an Insight to the last section of the form. Users signalised that they would like to create a chart first, and then calmly think about the name.
  • Testers were immediately asking about defining repositories using a search query. We should focus on removing the technical limitations and providing this experience as son as possible.
  • Insight preview is a must-have, all the users paid attention to it and welcomed it with enthusiasm
  • Initially, testers struggled to understand the granularity. Insight preview was very helpful in those cases.

CHARTS

We migrated to a different chart library that gives us better experience and more flexibility:

Code Insight: chart interactions

GA (03-2022)

BETA VALIDATION

We worked closely with a group of customers who decided to experiment with Code Insights in Beta. The overall reception of the feature was extremely positive but we've repeatedly heard feedback about improvements needed before Insights can be seen as a fully valuable feature.

Main issues to address before GA (General Availability)
  • Viewing Code Insights - one dashboard is not enough for big customers. They would like to group their insights by topic or group of interest;
  • Access - so far we have 2 visibility settings: public (visible for everyone on the instance) and private (visible only to a given user). Customers need more granularity so that they can control which groups of users (departments, teams, etc.) have access to insights)
  • Sharing Code Insights - users would like to share a link to an insight and make sure this person sees exactly the same data as them. Currently, Insights scope is adjusted for repositories a given user has visibility into. We need to either change the model or communicate it better to avoid confusion.

DASHBOARDS

Allowing users to group insights and control access to those groups became a clear next step in the process of becoming ready for the GA. We've decided to combine sharing, privacy and grouping in one solution - dashboards.

Code Insights dashboard

Code Insights Dashboard

Privacy & access
  • We've decided to move from insight-based to dashboard-based privacy settings
  • Insights inherit settings from dashboard(s) to which they are assigned. If an insight is assigned to dashboards with different privacy settings, it inherits the broader setting.
Adding a dashboard

When adding a dashboard, users can choose between 3 privacy settings: private (only this user), shared (one of the organisations) or global (visible for everyone on a given Sourcegraph instance). Those settings will be inherited by the insights included in the dashboard.

Add dashboard UI

Add new dashboard

Selecting a dashboard

Users can choose a dashboard from the dropdown. Dashboards are grouped by visibility and the list is searchable.

Select dashboard dropdown

Dashboard select (dropdown)

Assigning insights to a dashboard

Managing insights assigned to a dashboard is done via a popup. In addition, users can remove insights from a dashboard using the insights contextual menu.

Assign insights popup

Assigning insights to a dashboard

THE MARKETING LAUNCH

We've collaborated with our Marketing Team to prepare the Code Insights launch. In addition to providing input for the landing page, documentation, press release, and so on; I've created a promotional video that was used for social media posts, articles and on our website.

Promotional video for Code Insights launch

Post-ga

FUTURE IMPROVEMENTS

  • Filtering the insight - 'drill-down'
    Our users would like to narrow down the scope of already created insights and be able to compare results between repositories.
  • Automatically generated data series
    Another type of insight we're planning to introduce is 'detect and track patterns'. Data series will be generated dynamically for each unique value from the regular expression capture group included in the search query. Chart will be updated as new values appear in the code base.
  • Scoping repositories via search query
    Currently, we provide ability to scope insight to all repositories or a manually defined list of repositories. Our users would like to also define groups of repositories using dynamic search query.

MEASURING THE SUCCESS

KPIs & metrics

Impact on company goals

Go to the next project
Back to top