Self-service help for developers on Slack and Q&A forums

Introduction

Developers for Open Source Software (OSS) projects spend much of their time on GitHub for software development, Slack for interactive discussions, and community forums like StackOverflow for Q&A.

The top pain points that we observed are:

1. GitHub Issues are good for tracking bugs, feature requests, etc. but not for topical questions that are more support-oriented.
2. Slack is great for getting quick responses but can easily overwhelm maintainers. Users tend to repeatedly ask similar questions and searching for previous answers can be challenging.
3. Identification of friction points and consolidation of knowledge from Slack conversations is difficult due to the sheer volume and variety of messages.

To help address these pain points, Peritus.ai is working with open source communities to provide self-service help for developers on Slack and Q&A forums using our AI-enabled Peritus Assistant. Our goal is to consolidate knowledge from community conversations and documentation to quickly provide recommendations for best answers and experts for long-form technical questions.

We are offering Peritus Assistant for free for any non-profit Open Source Software project. Please drop us a note if you are interested in getting started.

Argo Project

Argo is an open-source project in the Cloud Native Computing Foundation (CNCF) ecosystem that provides a set of tools for Kubernetes to run workflows, manage clusters, and do GitOps. With the rapid growth of the project, the maintainers are becoming stretched for time.

To provide better support for the Argo Community, Peritus.ai has partnered with the project to provide developers self-service on the Argo Slack channels and provide actionable community intelligence, such as sentiment analysis, to the project maintainers.

Preparing the Peritus Assistant

Peritus ingests high-quality public content related to a project in order to prepare our recommendation engine. The content spans a wide range of sources covering technical content (documentation, blogs, etc.), Q&A forums (StackOverflow, Discourse, etc.), Messaging Platforms like Slack, and GitHub.

Next, we organize this information to train and assess the quality of the recommendations using previously answered questions as a reference. When the recommendation quality is above a threshold, the system is deemed ready to handle questions pertaining to the topic. The recommendations are made available via the Peritus Assistant for Slack and Chrome. The entire process can be completed within two weeks.

Analysis

An aggregate analysis of the conversations over several Argo Slack channels revealed some interesting insights. Over a four and a month period, approximately 57% of the questions received a response. For questions that receive a response, a majority of them get a response within an hour of posting. After an hour, there is a 71% chance that you may never receive a response. There could be several factors at play here:

1. As some of the channels receive scores of messages in a day, it’s likely that active contributors could miss these questions either due to the volume or due to the time of day (e.g.: time zones, weekends, etc.).
2. The Argo Slack channel, while very active, is still a best-effort channel to receive responses.
3. Sometimes, questions are very specific and may not lend themselves to getting quick answers.
4. Some of the questions are related to underlying technologies (e.g.: Kubernetes or Helm) and the user may be better served in approaching those respective communities.

Next Steps

Based on the ongoing feedback from the Argo maintainers, the aforementioned analysis, and usage patterns, the following features were prioritized:

• The maintainers expressed the need to get answers to questions about the project’s overall health on a few dimensions in order to get their job done and better serve the community. Some of these questions have factual answers (e.g.: number of GitHub stars, number of committers, etc.), time-based answers (e.g.: when was a specific version released), and others have answers that are productivity-oriented (e.g.: time to merge PRs).
• In order to improve the response rate and the response time, we are working on the ability to nudge users and experts to review unanswered questions. The former helps original posters to review Peritus recommendations in case they haven’t done so already, and the latter highlights questions that are well suited based on the expert’s preferences derived from past interactions.
• The Peritus recommendations are tagged to denote the associated technologies based on the question. With feedback from the Argo community, we are extending this to highlight where the user can get help from the communities of the related technologies (e.g.: Helm Slack channel, Kubernetes Discourse community).
• An aggregate view of the friction points in the Slack conversations to help community managers and maintainers quickly unblock stalled conversations, consolidate knowledge and prioritize bug fixes.

The overarching goal is to enable open source projects to better serve their communities and accelerate their growth by

1. Reducing friction for end-users to find help.
2. Making it easier for contributors to participate in areas of their expertise.
3. Equipping the maintainers and community managers with actionable insights derived from community activities across GitHub, Slack, and Q&A forums.

We are very thankful to Ed Lee and the Argo team, who have been very gracious with their time and support. Look forward to follow-up posts that provide an update on the progress we have made.