Forem: Stack Overflow

Experiment results: Testing close/reopen thresholds on Stack Overflow

Megan Risdal — Mon, 07 Oct 2019 18:46:52 +0000

At Stack Overflow, one of the main things we're working on is improving the ways our Q&A system facilitates feedback among users. Our user base plays an important role in not only answering questions asked by thousands of people per day, but also in helping to ensure that Stack Overflow is valuable as a resource which helps our future selves, too. This means that in many ways there are real people behind the keyboard using our software interacting with each other and influencing experiences people have (good and bad). Stack Overflow users deserve a system that sets everyone on all sides up for success.

Today, an essential part of how feedback is relayed among users on our site is our system for closing (and reopening) questions. While it was originally designed to help facilitate the creation and preservation of high quality question and answer artifacts, it's also a source of a lot of friction for our users. You ask a question on Stack Overflow because you think you can get an answer there. Getting your question closed today can be confusing, frustrating, and sometimes even unjustified.

So, what are we doing about it? We're taking a three-pronged approach:

Small changes to make the current experience feel less opaque or frustrating (e.g., redesigning our post notices including those for closed questions to be more helpful)
Experiments to understand the effects and emergent behavior of the system when we make changes
In the longer term, performing an audit of the current system and user research to inform a holistic overhaul

In the rest of this post, I'll describe a recent observational experiment we ran as an example of approach #2 including what we observed and our thoughts on next steps (which I'd love your feedback on!). Shog, a Community Manager at Stack Overflow, and I published a longform version of this write-up on meta. This is mostly Shog's write-up; I contributed the charts.

Close/reopen vote threshold experiment

In August, we ran an observational experiment on Stack Overflow in which we reduced the number of close/reopen votes required to close/reopen a question. Our main hypothesis was that this would make the system more efficient. We operationalized "efficiency" to mean, if a first close vote is cast, that question is more likely to ultimately get closed (same for reopening).

Why do we care about efficiency? For users whose questions are closed in the current system as it works today, we'd prefer that their questions get closed sooner before attracting downvotes/discouraging comments. And for users who volunteer to review questions ("curators"), if their vote does not result in closing or reopening, it should be because other members of the site reviewed it and decided that outcome was unwarranted; the vote should not be ignored.

As with all experiments (observational or experimental) and product changes we make on Stack Overflow, we also monitor other metrics. Some of the other factors we looked at:

Qualitative perceptions shared on meta (where we announced that we were doing the experiment) and in our site satisfaction survey
Volume of things like closed/reopened questions and participation in the respective review queues
"Close wars" (a close war is when a question is closed, reopened, then closed again)
Consensus (there are multiple reasons a close voter can choose from and reducing votes from 5 to 3 could impact consensus rates)

We ran the experiment for thirty days and compared the experimental period to pre- and post-periods.

The experiment results

Efficiency

Our hypothesis was confirmed. The close/reopen system was more efficient across the board.

Very importantly, the total quantity of questions nominated for closure stayed about the same, even as efficacy -- questions actually getting closed -- went up. This means that the experiment didn't trigger a wave of close voting.

Qualitative perceptions

We heard the most from our active, engaged users on meta who represent the curator side of this equation. From them we heard optimism and motivation to participate in the site plus a lot of thoughts on how the experimental threshold would impact the close system including users implicated in it.

We supplemented this feedback by looking at responses to our site satisfaction tracking survey which we recently started as a way to get more diverse feedback from Stack Overflow users. On the survey, we regularly see people mentioning closing as one of the most frustrating things about using Stack Overflow. During the experiment, we saw a slight uptick, but it wasn't anything significant. This is something we'll continue to monitor.

If you have thoughts, I'd love to hear them in the comments here!

Closing/reopening and participation

Right now, a very small group of people participate in closing/reopening and reviewing questions that have been voted to be closed/reopened. As long as we have the same system in place, we would prefer greater diversity among the people participating.

During the experiment, we saw more participation in reviewing questions. This was driven by an increase in reopen reviewers.

During the experiment, we saw an increase in the number of questions closed. Recall this was NOT the result of an increase in the number of questions voted to be closed, but rather an improvement in the efficiency of the system.

During the experiment, we saw an even larger relative increase in the total number of questions reopened.

Close wars

A "close war" is when a question is closed, reopened, then closed again. Did lowering the threshold required to close/reopen a question result in an increase in their occurrence?

In the 30 days prior to the experiment, 100 questions were closed at least twice.
During the 30-day experiment period, 188 questions were closed at least twice.

We did see an increase, but close wars are actually already relatively rare on Stack Overflow. We aren't too worried about this.

Consensus

During the experiment, it was much easier to close a question without consensus among close reasons.

In the 30 days prior to the experiment, 49 questions were closed without a consensus reason.
During the 30-day experiment period, 560 questions were closed without a consensus reason.

This is more concerning to us. It's an emergent behavior of the lower close threshold which we're not as happy to see.

Conclusion and call for feedback

From this experiment, we've concluded that a lower close/reopen vote threshold successfully increases the efficiency of the system. This has positive effects on our curators who volunteer to review questions and we hypothesize that it's a better experience for users who ask questions (though we'd love to hear more feedback from you!). It was especially encouraging that there was NOT an increase in the number of questions that were voted to be closed even while participation in the process increased.

Based on these results, we have learned that we would like to implement a consensus rule before lowering the close/reopen threshold to 3. That is, a question must receive n close votes have to agree before a question gets closed.

Our thinking is that a consensus provides more confidence that question should be indeed closed AND gives the question author more concrete feedback which they can learn from or act on.

What are your thoughts? What feedback do you want when you ask a question on Stack Overflow and how do you want to receive it? Do you think we should enforce consensus close reasons? What aspects about asking a question on Stack Overflow today frustrate you?

SELECT Post FROM Stack Overflow Questions WHERE Topic = "git" ORDER BY Votes DESC;

Megan Risdal — Fri, 30 Aug 2019 15:39:36 +0000

Recently, one of my friends, Jamie Hall, wrote a really great, practical tutorial titled "Get good at git". In it, he introduces the basic workflows you need to know to use the version control system everyone "loves to be hated by" (his joke, not mine). Not only is it a lovely introduction to git, this paragraph really resonated with me:

In practice, everyone [screws] up constantly. I’m talking about professional software engineers, with many years’ experience, totally [screwing] up their work through unforced errors and dumb mistakes.

Problems with git are so common that Katie Sylor-Miller created a website explaining how to get out of a specific git mess in plain English.

Knowing that the best place to find reassuring company with other developers totally screwing things up is Stack Overflow, I decided to have a look at the all-time top questions by votes to see how common git problems are. And it turns out that 5 of the top 10 most voted questions are about getting help with git.

If you've come across one of these questions (I know I have), you are very far from alone! I used DEV's new Stack Overflow liquid tag to embed the top git questions of all-time on Stack Overflow. Plus, on Stack Overflow you can now also click "share" and select "DEV" on any post.

#2 most upvoted question

May 29 '09 Comments: 12 Answers: 105

27198

I accidentally committed the wrong files to Git but haven't pushed the commit to the server yet.

How do I undo those commits from the local repository?

Open Full Question

#3 most upvoted question

Jan 5 '10 Comments: 3 Answers: 41

20360

Failed Attempts to Delete a Remote Branch:

$ git branch -d remotes/origin/bugfix
error: branch 'remotes/origin/bugfix' not found
$ git branch -d origin/bugfix
error: branch 'origin/bugfix' not found.

$ git branch -rd origin/bugfix
Deleted remote branch origin/bugfix (was 2a14ef7).

$ git push
Everything up-to-date

$ git pull
From github.com:gituser/gitproject

* [new

…

Open Full Question

#4 most upvoted question

Nov 15 '08 Comments: 8 Answers: 37

14043

What are the differences between git pull and git fetch?

Open Full Question

#8 most upvoted question

Dec 7 '08 Comments: 13 Answers: 39

11608

I mistakenly added files to Git using the command:

git add myfile.txt

I have not yet run git commit. How do I undo this so that these changes will not be included in the commit?

Open Full Question

#9 most upvoted question

Jul 6 '11 Comments: 1 Answers: 41

12018

How can I rename a local branch which has not yet been pushed to a remote repository?

Open Full Question

What about you?

Are there Stack Overflow questions about git you constantly reference? What are your biggest git hangups? Have you ever had any "oh crap" moments with git?

What’s in the Works at Stack Overflow: Improving Feedback for All Users

Megan Risdal — Tue, 27 Aug 2019 02:06:19 +0000

Hi, I'm the product manager leading the development of Stack Overflow's flagship product, public Q&A. Last week, I shared an update on our blog about what we're working on and why.

Building Stack Overflow in a way that better meets the needs of more and more developers is important to me and my team. I'm cross-posting an abbreviated version here because I want to hear your feedback for us.

Where we're at today

Last year, we kicked off an initiative to make Stack Overflow more welcoming. After launching a new Code of Conduct, an improved question asking experience, and several other changes, we spoke to many Stack Overflow users to hear more about their personal experiences using the site. Through these conversations, we heard that the limitations of the Q&A system continue to create the very environment we set out to discourage. People who need help with coding problems feel attacked when their questions are closed or downvoted, while those curating site content feel blamed for doing what the system has asked them to do.

It’s simply not enough for us to ask people to be nice or change their behavior when the software that underlies everyone’s interactions doesn’t facilitate this. That’s why we’re looking at ways to revitalize the way the core Q&A system works.

What we're working on now

Today, my team's focus is on these initiatives:

How feedback is delivered by the system
Addressing issues with our commenting system
Making a single question asking experience work for everyone

Improving feedback

As Sara, our Director of Public Q&A, noted in her recent blog post, it’s hard not to take feedback personally when it’s piled on, no matter how constructive it is. Unfortunately, this overwhelming pile-on is exactly what users who ask an imperfect question are confronted with.

The first way we're improving the feedback loop among our users is through a redesign of post notices. If you’ve come across a duplicate question or closed question, you’ve probably seen a post notice. These are the pale yellow informational banners that sometimes appear on questions. For people who ask questions today, if your question is closed, feedback that is directed toward you privately is shared publicly with anyone who views your question.

Here’s what our holistic redesign of all post notices will prioritize:

Delivering improved, private feedback to post authors
Not putting users who curate content on the spot
Giving actionable, understandable information for the vast majority of public viewers

While working on building our refresh of post notices, we’re also doing discovery on ways to make our question close workflows and review queues better facilitate feedback and content curation for seasoned moderators, technology experts, and new question-askers alike.

“They told me not to read the comments…”

It probably comes as a surprise to no one that the comments section on Stack Overflow doesn’t always serve its original purpose (facilitating clarifying questions to improve question quality). Comments can be distracting, outright harmful, pure spam, and everything in between.

We encourage people to flag content that doesn’t belong, and our moderators do incredible work to review everything. But there’s a lot of opportunity to reduce the burden for moderators and users who flag inappropriate content in ways that make using Stack Overflow more pleasant for everyone.

For example, some of the things we’re exploring right now:

Better distinguishing the “Answer” and “Comment” actions (due to a high volume of helpful “Not an answer” flags indicating problems with the interface).
Ways to reduce the number of unhelpful comments that are likely to get removed in the first place.

Improved question-asking guidance for everyone

Several months ago, we launched the Ask Question Wizard on Stack Overflow. The new experience defaults users with low reputation to use a guided mode to formulate their questions. So far, we’re pleased with how it’s helping users ask better questions and as a result have a better experience on the site. In fact, we’re so pleased that our next iteration is focused on incorporating aspects of the guided mode into the question-asking form for everyone.

We’re still in early stages, but some of the changes we’re working on so far include:

Upfront guidance for first-time question-askers
Setting expectations for what happens after asking a question
Improved “how-to-ask” guidance while drafting a question
Making it easier to improve question quality by consolidating many dozens of validation messages into a single “review” interface

We’re excited to begin working on these changes because it gives us an opportunity to create a better user experience for everyone. More and more, we hope to look for ways to simplify our platform to seamlessly and intuitively meet the needs of all users who come to Stack Overflow to find and share knowledge.

What do you think?

Millions and millions of users come to Stack Overflow each month and rely on and contribute to the site and community in so many unique ways. Stack Overflow wouldn’t be what it is without contributions from developers like you.

What do you think of this direction? What would make it easier for you to use and participate on Stack Overflow? What are your biggest frustrations today?

What a very bad day at work taught me about building Stack Overflow’s community

Sara Ownbey Chipps — Thu, 18 Jul 2019 16:02:44 +0000

Hi, my name is Sara Chipps, first time Stack blogger, long time Stacker (I’ve always wanted to say that!). I’m the new Director of Public Q&A at Stack Overflow. I’ve been at Stack for a year now, and I’d like to share with you one of my worst days at work, and what it taught me about the Stack Overflow community.

A little about me: I’m Stack Overflow user #4140. I was in the beta and one of the first people to ask a question on the platform. Stack Overflow has been a big part of what I do for a long time. I’ve been an active member of Q&A, a participant on Area 51, and a lurker on Worldbuilding, Cooking, and Code Golf. I’ve looked for new opportunities on our Jobs board and placed job ads when hiring great talent at companies I helped run, like Jewelbots or GDI.

The thing that really connected me to Stack Overflow and its community, however, is the simple fact that I’ve been a software developer for 18 years. I cut my teeth on MS SQL and Data Warehousing. I moved on to C# and .NET in 2006. I was a .NET MVP for 2009 and 2010 before switching to JavaScript, Node.js, and building Nodebots in 2011. I love JavaScript with all my heart, warts and all. Since joining Stack Overflow’s engineering management team in 2018, I’m back in the world of .NET and on the board of the .NET foundation helping the framework build the future of the internet.

I care a lot about representation in technology, and a future where people from underrepresented groups in technology are hired and succeed at the same rates as their peers. Being a part of the team helping to guide the direction and growth of community is an incredible honor.

We’ve been working on exciting things the past few months to make the site more welcome, diverse, and inclusive. To kick off these changes, we started with the tools that our moderators and power users rely on to make Stack Overflow the best site for developers online. The thing I’ve seen our Community Managers push for the most is updating these dated tools, some of which haven’t been touched since we first launched! The team formerly known as DAG (Developer Advocacy and Growth), now part of the Community team, started by rolling out the Tag Synonyms Refresh and the improved Moderator Dashboard. Paying down this debt will continue to be a priority as we work to get our mods best-in-class tools to manage their communities.

The second thing we are prioritizing requires a bit of a story. When I joined Stack Overflow almost a year ago, I was blown away by how kind and generous all my coworkers were, and the engineering team is no exception. As engineering manager for the team charged with working on our Talent product, I got to work closely with brilliant people I really respected. It was a treat to be collaborating with lifetime learners and natural teachers, the kind of engineers that you would want on any team.

About three months in, on a Friday afternoon, we introduced a new company-wide policy that I felt was relatively benign. What happened next was that, from my point of view, the engineering team completely lost it. No one agreed with this policy, and they made it known over seemingly hundreds of Slack pings. After an afternoon of going back and forth, I walked away feeling emotionally drained. What had happened to my amazing coworkers that were so kind and wonderful? I felt attacked and diminished. It seemed people weren’t valuing my work or my judgment.

I went home for the weekend and stewed in my frustration. I replayed everything that happened in my head and each time got more frustrated with the way people reacted. When Sunday rolled around, I decided I wanted to look back at our Slack conversations and see which one of my coworkers was being the rudest and the most unreasonable. I wanted to give them direct feedback that they had hurt my feelings.

As I went back through that Friday afternoon chat log, I was shocked to see that no one had been hurling insults. There was no one saying mean things about me or attacking my efficacy directly. In fact, what I found was that people had some well put together arguments about why they felt this policy was a bad idea. The entire engineering department definitely made their criticisms known, but I didn’t find people questioning my ability as a manager, throwing around insults, or saying anything that that illustrated why I was feeling so targeted.

That was when something became crystal clear: my coworkers hadn’t become monsters, they were still the kind and caring people I thought they were. The monster in this case is not one person, it was created when lots of people, even with great intentions, publicly disagreed with you at the same time. Even kind feedback can come off as caustic and mean when there is a mob of people behind it. No matter how nicely they say it, when a large group of people you really respect publicly challenge something you’ve done it can feel like a personal attack.

When I realized this, some of the confusion I had seen about unwelcomeness on Stack Overflow started to make sense. In our developer survey results we read things like this:

“ Caustic community for new users. There is no excuse for not being kind!” – 6 years coding
“ It feels too scary and unaccessible for new developers” – 3 years coding
“ People could be less brutal” – 6 years coding
“ The attitude is not beginner friendly. Askers are expected to have done a lot of research before asking a question (re: both question format and content), even if they are completely new to the community or topic. Not everyone can understand or even know to look for documentation when they’re completely new to programming.” – 12 years coding experience

However, when our more experienced users hear this feedback they ask us to provide them with definitive examples of WHERE EXACTLY people are being unfriendly? There isn’t a lot of name calling or anger, why are they being accused of being unfriendly?

People tell us they are afraid to participate because of how mean their peers can be. The way the system is currently built, when you ask a question that could use some editing or is a duplicate, a bunch of people come out of the woodwork to tell you you’ve done something wrong.

They could say it in the most neutral possible way, but no matter how you approach it, a dozen people pointing out your errors feels terrible. Not only does it feel terrible, but it can also be not beneficial for overall content quality, not to mention an ineffective way to get someone to improve their question. There is also a big yellow box that gives you the names of engineers that voted your question closed or deleted. That in itself can feel really bad especially as our high rep users skew toward more experienced and respected engineers.

On one hand, our more experienced power users tell us they feel called out for being unfriendly even when they are just trying to be helpful; on the other, our newer users and people that don’t participate tell us they think Stack Overflow is scary and they are afraid of judgement. In the past, we’ve prioritized getting rid of unfriendly comments. We’ve seen improvements there, but we still hear people feel targeted even when there aren’t unfriendly comments. This problem is on us and it’s because of how we designed the question asking and closing process. People are using the product as it was designed and as a result people feel called out or, even worse, discouraged from ever asking a question again.

Over the next few quarters, we’re going to be taking a step back and re-evaluating how we deliver feedback to users about their questions. We want to make sure people are getting necessary feedback without feeling called out or publicly embarrassed. We will be working on new paths to improve content quality and reduce friction between people. Our goal is to have the question asking process be painless and beneficial for new users and Stack Overflow veterans alike.

By improving the way people give each other feedback, we can improve question quality without putting the burden on our users to police the website. We will empower our long time users to become mentors and teachers in order to bring the spirit of Stack Overflow back to what it was in the beginning, a place where people come to share and learn. By thinking hard about how we give feedback, we’ll help people learn instead of driving them away. We’ll get more people involved and improve question quality.

Myself and the community team are really excited to improve the experience that all levels of coders have on Stack Overflow, from new users that are learning front-end for the first time to our respected moderators who have been coding for 20+ years. We all have ideas on how to make the system better. The great news is we have experienced researchers, data scientists, and an amazing product manager that will be gathering feedback from us, the community, and many other places and partners to make educated decisions about solutions.

We think the world of our community, and are excited to hear what you think of improvements as we make them. We’ll make sure, as always, to keep you posted.

The post What a very bad day at work taught me about building Stack Overflow’s community appeared first on Stack Overflow Blog.