ASPInsiders is the wrong way to gather feedback

28Jul/0821

ASPInsiders is the wrong way to gather feedback

A sneak peak at ASP.NET AJAX 4.0’s client-side templating | Encosia:

“For those of us who aren’t MVPs or ASPInsiders, it’s nice to have a chance to offer constructive feedback and generally not be left in the dark.”

Last week I met up with a bunch of people for a geek dinner. A few of them were in town for the ASPInsiders sneak preview/briefing/whatever you call it. I mentioned that I wasn’t an ASPInsider despite having worked with some form of ASP since it was a bunch of idc and htx files. A lot of them kindly offered to nominate me if I wanted to be an ASPInsider. I declined. My reasoning for pointing out that I’m not an ASPInsider isn’t because I think I’m so special that I *deserve* to be included in that group. It’s that if they missed me when sending out invites to the group, who else might they have missed. Probably a lot of people who could provide really great feedback to the team. Is the solution to invite more people and expand the group?

I don’t think so. In fact, I think the group needs to go away. I say that not because I want my friends to miss out on a trip to Redmond to meet with the ASP.NET team. But because I feel the time for small, exclusive groups who have insider information and get to give feedback on code and features that haven’t been released yet is in the past.(2) That’s old school, dinosaur thinking in the software development world. In the case of the MVC team, which has been releasing code and announcing features at LIGHTNING SPEED(1), secrecy and NDA’s really only last a month or two before the feature is released.

I know there are probably legal reasons for the NDA. The team/management is probably nervous about showing features that aren’t fully baked to the general development audience, only to have them change before release. But I think with a little work and more transparency, those issues can be resolved. Ideally, source checkins would be fully transparent. We would have read-only access to the teams source repository. We could build the framework on our machines and maybe even contribute some code back. But even just a wiki/forum/test site where new features could be demoed and discussed openly would be a great start.

(1) Lightning speed for Microsoft in any case.
(2) As J. Ambrose Little pointed out in his comment below, Membership in the ASPInsiders group is not determined by Microsoft. So they can’t be held responsible for defining who gets in and who doesn’t. They are a group of individuals who volunteer their experience and expertise to the ASP.NET team. The debate over whether or not the general public should be able to see the same features they do is a separate debate which continues in the comments and elsewhere.

Filed under: General Leave a comment

John Stockton

You better check that junk mail folder again, you just might be an ASP Insider by now
http://ryanfarley.com Ryan Farley

I completely agree Scott. The guy who’s been writing ASP.NET apps since the beginning of ASP for his own company, that no one really knows about, could provide feedback that is as useful as someone who is very visible in the community – but they’ll never really get that guy’s feedback. But, something is broken because there are a lot of very visible people out there that have been involved with ASP.NET that they’re missing out on (like you). Maybe junk folders are to blame

There are sure to be better ways to do it now days.
http://amundsen.com/blog/ Mike Amundsen

“…the time for small, exclusive groups who have insider information and get to give feedback on code and features that haven’t been released yet is in the past.”

nope. not hardly.

this is not a software phenomenon, either. this is how creating things works. you can’t have everyone in at the ground floor. you can’t build new things (software, hardware, toilet seats, etc.) completely by committee and in the open all the time. you need to start small and work out from there.

using small groups *exclusively* is a dumb idea. hand-picking small groups to make sure the group fits your agenda/POV, etc. is counter productive. i don’t think that’s what is going on w/ ASPInsiders.

also, having a small group that initially provides feedback does not mean you can’t have a large group, too. it’s not one or the other. and i see MSFT doing this quite a bit. opening things up in early betas for testing and feedback.
http://www.west-wind.com/weblog Rick Strahl

Scott – while there are good point in what you say – and I definitely agree on the transparency point – I think this is about manageable feedback to Microsoft with the ability for MS to be able to have a direct conversation. Much of that is often very difficult in a broad public forum especially when dealing with new features, because it quickly gets overwhelmed by too much access and well, unrelated noise and ranting. Take a look at any ‘architecture/vision’ type posts on the ASP.NET Forums and how quickly those get out of hand to get an idea what I mean.

I definitely don’t think that this should be the only feedback and it doesn’t look like Microsoft is treating it that way anyway. Insiders have some input but it’s not a determining factor. It’s more of a first impression or trial run before moving on to a broader forum and currently the move from a ‘private’ forum to general availability or announcement is very short anyway.

Also, source repository access is all good and well for language/framework features, but how do you manage that for the Tools portion?
http://damieng.com Damien Guard

I don’t see ASP Insiders as an exercise in Microsoft finding and rewarding good developers but rather interacting with a vocal passionate group who are already writing posts, books, comments and generally making themselves known to the community through their thoughts, ideas and suggestions.

Then they take this to the next level by getting their feedback earlier in the development cycle and the nature of that requires some level of trust and limited disclosure in such a cutting-edge industry right now.

[)amien
http://dotNetTemplar.Net J. Ambrose Little

Scott,

We apologize for not inviting you sooner. I’m sure it was just an oversight; we work on a peer nomination and voting basis. For what it is worth, your induction was one of the easiest/fastest in our many years of existence; there were comments like “wow, I’m surprised he’s not already a member.”

Now that I’ve addressed the main issue here…

The ASPInsiders does not intend or claim to cover every notable in the community, nor do we or Microsoft think that we are the sum total of those who can provide valuable feedback. In fact, Microsoft folks have reiterated that to us many times over the years. We are just one channel of feedback.

There are forums, blogs (and comments on those), product support, feature request/issue reporting tools, conferences, partners, general design reviews, and more that product managers (not just at MS) can take advantage of. All of these channels have their own value.

Now, the question is viable–is it worth having this particular channel for Microsoft? That is a question only Microsoft can answer, and I know that they have asked it more than once.

My thoughts are that I think Mike and Rick summed up the reasoning behind that pretty well. It’s all about manageability for certain kinds of feedback. You just can’t invite the whole dev world out to Redmond or get the whole dev world involved in other, more direct forms of feedback. It’s pretty common to have these sorts of smaller groups in product management that you work more directly with.

Hope this helps clear things up.
http://weblogs.asp.net/erobillard Eli Robillard

The reason that TAP programs, community councils and insiders groups (including the ASPInsiders) exist, even as product cycles accelerate from years to months is that once the product team mentions a feature publicly, they’re expected to build it. It isn’t necessarily fair, and no, it shouldn’t be that way, but unfortunately that’s the reality of our industry. So while teams can ask each other what they’d like to see all day, that doesn’t give them real-world perspective. Teams need to sanity-check ideas with the real world, and ideally that means a small group of people whom they personally know and trust. MVPs aren’t nominated by trust, they’re nominated by stats and correspondingly have an unfortunate history of leaking information before it’s time (well, except of course for the MOSS MVPs). And that’s why groups like the ASPInsiders exist. It’s work, it requires a time committment, there are few (if any) tangible rewards as compared with the MVP Award. But after several iterations of ASP.NET, I can say that the process continues to result in a better product. I’m glad both that it exists and that I’m fortunate enough to be a part of it.

Cheers,
-Eli.
(ASPInsider and MOSS MVP)
http://www.lazycoder.com Scott

“Now that I’ve addressed the main issue here…”

Ambrose: Honestly, that is LIGHT YEARS away from the main issue here. My feedback isn’t an issue as I’ve always had channels, either via my blog or via friends who know friends on product teams or MVPs, for giving my feedback to Microsoft. My getting nominated as either an MVP or ASPInsider has never been a goal of mine and I’m flattered that anyone would consider my insight to be that valuable. Getting included in a group that include people like you, Rick, and Eli, people who’s blog posts and article help me learn and use frameworks, is weird for me as I’m still the guy that forgets semicolons or to close a bracket or parenthesis. My being an ASPInsider is just a drop in the bucket. I think we *all* need to be ASPInsiders and I think the product will benefit.

Mike, Rick, and Eli: I can certainly see where the volume of feedback might be an issue. But the problem is, usually by the time the public gets to see a new feature, it’s too late in the cycle to change anything based on feedback and we’re told “well, that fix might come in a service pack or a hotfitx.”. That leads to the “if you show us, you have to build it” line of thinking. In the past by the time they’ve shown the public anything, it’s already been built. The key, I think, isn’t to invite a small, or even a larger, group to Redmond, it’s to open up the process to the web more. I think the volume of feedback will result in a greater focus on things that needs to be changed. If you have a group of 30 and 2 people say, “I think this needs to be changed” how can you determine if they are just edge cases or represent a greater need. If you open up the feedback early on to everyone and you get 20,000 responses that say “This needs to be changed”, you have a better idea of how many of your customers would be affected by the change. The most important issues will emerge from the cacophony and you get a better sense of what the feature will mean to developers with a wide variety of skill levels.
I’m not saying that focus groups haven’t worked in the past, I’m saying that more transparency could work *better* than focus groups.

“Also, source repository access is all good and well for language/framework features, but how do you manage that for the Tools portion?”
Rick: Re: dev tools source code – Ehhh, I’m not enough of a OSS radical to suggest that MS, or any company, needs to open source their products. But for things like the .NET Framework, I don’t see any real competition. What other web frameworks are there that work on the Windows platform? JSP, Rails, ColdFusion, etc… Two of those are OSS already. They aren’t really competing with any of those on features, the language you use with each one of those is the biggest feature that makes people choose them. Same with .NET.

“Then they take this to the next level by getting their feedback earlier in the development cycle and the nature of that requires some level of trust and limited disclosure in such a cutting-edge industry right now.”

Damien: See above. I don’t think MS, or any other company, has much, if anything, to lose by involving more people earlier on in the dev cycle. So maybe there is still a need for NDA’s, open up the registration to everyone rather than requiring nominations. The vocal supporters who are interested and want to participate more will still be heard. I agree it’s not about rewarding those developers who have shown an interest, but about interacting more with interested people. I know of two people, at least, who weren’t insiders until they got tired of hearing the “That’s under NDA” answer to all of their questions and said “Well, how do I get under an NDA?” Now, how much easier would it have been if the team member didn’t have to answer, “That’s under NDA”?
http://amundsen.com/blog/ Mike Amundsen

Scott:

if you want more transparency, that’s fine. implying that the ASPInsiders’ efforts are “old school, dinosaur thinking” and the MSFT “team/management is probably nervous about showing features that aren’t fully baked” is not the best way to make your case.
http://damieng.com Damien Guard

@scott: The general public approach suffers from two main problems. The first is that sometimes they are going the wrong way about things and a simple alternative to what they are proposing already exists. The second is that they are not always commited. Sometimes it sounds like a good idea and you go back for more information and it just isn’t coming so you’re left with a half-baked suggestion.

Put this on a very large scale and yes, you’d get design by committe plus potentially people thinking feature A. makes my life easier today without thinking about how it affects maintainability, platform, support etc.

As always there is no golden bullet, we can just try different approaches and learn from our experiences.

[)amien
http://dotNetTemplar.Net J. Ambrose Little

Hi Scott,

The ASPInsiders is an independent entity from Microsoft. Of course, our main thing is direct feedback to the MS ASP.NET teams, but historically, we were founded as a separate entity and our membership has been managed by the ASPInsiders based on a peer nomination and voting basis.

So you see, Microsoft is not the party that is limiting inclusion in the ASPInsiders–it is the ASPInsiders membership itself. Microsoft has been generous in their engagement for that, and we are grateful.

In the end, we are just one of their many inputs, but we are at least a manageable, coherent, and self-selected (which has some value in itself) group of experts (not sure how *I* got in!) in this space that they can draw on for direct feedback *in addition* to their many other forms of feedback.

Keep pushing for transparency, but I hope you might reconsider your tactic, as others have suggested.

Hope this helps.
http://www.lazycoder.com Scott

Mike:I wasn’t implying anything, I was declaring that kind of thinking is old school. I think DevDiv realizes this and they are responding with more openness and transparency. I mean, you can download the SOURCE CODE to the MVC framework from CodePlex!!!

And I think the team and management probably does get nervous showing functions that they are still debating internally and that aren’t fully baked. Developers in general don’t like to show unfinished products to people
http://www.lazycoder.com Scott

J. Ambrose:
Ahhhhhh, so I’m dealing with two separate issues. But confusing them. I was under the incorrect impression that ASPInsiders was an MS Sponsored group. So the membership limiting isn’t coming from MS. Doh!, I’ll update the post to reflect this.

The debate about how much transparency and openness is a separate issue though, as Damien and others continue to make great points.
Scott Galloway

As the person in the ASP.NET team responsible for arranging things like the ASP nsiders Summit you’re referring to (as well as for all the Codeplex releases) I want to comment on this.
Before I joined the team I have the same view of ASP Insiders as you seem to have…an elitist group of ‘names’. Over the past few months I’ve discovered a different side; in general as a product group we use the ASP Insiders to provide very focussed and frankly trusted feedback on features way before release; in fact in many cases we use them to assess the validity of ideas.
Being able to trust this group not to blog about these ideas is important; partly because we want to be able to talk openly without getting involved in quasi-religious discussions / veering off in tangents and partly because, frankly, we have bad ideas sometimes.
We are going to release more and more features earlier and earlier on Codeplex but there’s some stuff that we just can’t release like that e.g., our feature is ‘baked’ into System.Web / someone else’s assembly that we just can’t make public (not everyone can be as open as us).
A lot of us on the team would LOVE to make completely open checkins but again, as ou mention there’s legal (IP) and practical (dependncies on other source which isn’t / can’t be public…think super-secret until release things like Silverlight was).
http://www.lazycoder.com Scott

“Before I joined the team I have the same view of ASP Insiders as you seem to have…an elitist group of ‘names’.”

To be clear, I never thought of the group as “elitist” or that they believed themselves to be better than other developers.

“we want to be able to talk openly without getting involved in quasi-religious discussions / veering off in tangents and partly because, frankly, we have bad ideas sometimes.”

Right, I get that. But why shy away from those situations? Perhaps a better approach would be to expose the bad ideas right away so that you don’t end up converting them into features.
http://haacked.com/ Haacked

Just my personal opinion, not that of my employer, but my guess on the whole NDA issue is that there are reasons we may not want to open up every idea to the public as soon as we think of them. For example, we might want to execute on an idea a bit before we share it so we have a jump on competitors. But before we do that, we want to sanity test the idea with some smart people who won’t leak it.

Consider your own company (not knowing anything about it), do they give away every idea as soon as they consider it? Can you publish your company’s source code online?

Even so, I think we are still being very transparent and getting involved in the hard discussions. As you pointed out, we’re shipping the freaking source code while there’s still time to make a difference. And I think we’re doing it early enough to make changes. Just look at the evolution of MVC thus far.

But that doesn’t negate the need for hosting a small group for more intimate discussions. For example, I’m completely overwhelmed by the tsunami of feedback I get. I try hard, I really do, but I’m drowning. So even without the insiders, there’d be a de-facto insiders. I can only subscribe to so many blogs. I’m commenting on your blog because I know you and respect your opinion (well some of them). What about the person whose blog I’m not commenting in?

I think forums and such are great for being expose to breadth of issues, but what about depth. What about the type of feedback you get from a person to person conversation? We need that. Feedback has more impact when it’s person to person, especially when sharing the pain you have with some tool.

So if not the insiders, what’s the better alternative?

Should we have random drawings on asp.net “Hey! Register here to win a chance to pay your own room and board to Redmond to give Microsoft feedback!” I’m not saying we shouldn’t. If we do, it was my idea first. Do you have a specific suggestion in mind?
http://www.lazycoder.com Scott

“before we share it so we have a jump on competitors. ”

What competitors? Java? The STL? Delphi? Telerik and DevExpress? Again, I’m not talking about making the source available to Visual Studio or Windows. But certainly showing potential features to a wider audience won’t hurt. It works reasonably well with Ubuntu and Rails. The quality of Linux distributions has risen even as more people have become involved in the process, both in development and in usage. Accepting comments from everyone doesn’t mean you have to convert every idea into code, as DHH could tell you, and it doesn’t have to mean you are committing to build a feature if you float it by the general public. What I’m talking about is a modification of the “with enough eyes, all bugs are shallow” idea. With enough eyes, new features get hammered out quicker.

In this particular case, the ASPInsiders got to see more than just MVC I’m guessing. Stuff from other teams who aren’t being as open, but who could probably benefit from more eyes seeing their features.

I can agree that a small, independent group of passionate volunteers can be useful. But why put them under NDA? Why not let them speak about what they saw, gather feedback from a wider, more diverse group, and filter the feedback to you?

Paul Litwin pointed out to me that the big OSS companies like MySQL probably use focus groups too. But I wonder if the focus groups are under NDA.
http://stevesmithblog.com Steve Smith

Scott,
You don’t really believe that Microsoft has no competitors in the development space, do you? If so, I think that’s rather naive. If not, then you must understand that they have real strategic business reasons for wanting to keep certain things under wraps in order to maximize the strategic advantage they get from being first-to-market with some new feature, product, or technology. That’s pretty simple good business. It also follows that such products will be better if they have some early feedback from real world users. So, how do you get real world feedback from people without sharing the goods with competitors?
http://www.lazycoder.com Scott

Steve: I think MS has competitors in the development space same as the do in the Word processing/spreadsheet and OS space, I just think they don’t matter. I’m saying the openness of the other competitor’s processes, and I’m primarily thinking of Ruby/Rails, Java/JBoss/etc, and PHP not competitors to features of the .NET Framework like control/component providers (e.g. DevExpress and Telerik), isn’t hurting their position in the market or their ability to execute. There could be reasons to keep certain technologies secret until they are announced, say the launch of the .NET Framework or Silverlight, but once they are released they would benefit from more involvement. As for competition with component/control vendors, MS still has the huge advantage of shipping their control WITH the framework.

Compare the development process of the MVC Framework, which is one of the most open and transparent products right now, with that of the Entity Framework. Which has had more success and is looked upon favorably by developers?
http://blogs.msdn.com/gblock Glenn Block

Scott:

I completely agree with you that we need to be more and transparent in what we do. I’d also say we’re making tremendous strides in that direction. In the last year alone, .NET Framework 3.5 source, ASP.NET MVC, Dynamic Data on CodePlex and soon MEF are all indications of this.

At the same time, i would disagree that having groups like the insiders is contrary to this. Being open is not mutually exclusive with the fact that you can’t have private discussions with a group of individuals that play more an advisory role. As long as those advisors are a good representation of the community, then what is the problem?

We can and will continue to push the bar on openness, but that doesn’t eradicate the need for having a focused group of advisors which you can turn to to make sure you are not going off a cliff.
http://haacked.com/ Haacked

You think we’re privvy to every decision every OSS project makes? I don’t doubt for a second that sometimes, DHH emails a couple members of the team outside of the ML with an idea, which they say, “Nah, that’s bollocks!” before sharing it with others. Or before having the first commit be a rough implementation, thus getting a jump on others.

If not him, maybe others. OSS projects compete with each other for mindshare and developers. So they do the same type of thing, but on a smaller scale.