(00:40) Jon asks Glenn what Splunk does. Splunk has a product that gathers operational intelligence. It’s got a data analytics platform which understands a lot of log formats. It can handle streaming logs and has a bunch of API’s. It can index in realtime, handles unstructured data, and has some advanced pattern matching features.
(02:12) Glenn talks about some common uses. GitHub and Target both use Splunk. It’s especially liked by IT Admins who can query across multiple servers by timeslice in realtime. There’s a customizable dashboard to surface the information.
(03:24) Glenn says that since Splunk has a powerful API, you can push data into it. You can push data in using HTTP or TCP.
(04:01) You can teach Splunk to fetch data from a source using their app platform. Glenn talks about an Azure app he built for Windows Azure Web Sites diagnostics.
(05:39) Splunk is available in the cloud, but it’s often run on premises. It’s cross-platform. It doesn’t store the data, it just indexes it.
Pricing, free versions, cloud hosted versions
(06:44) Glenn says the pricing is based on data throughput. They have a free license that gives you 500MB/day, a developer license that gives you 10GB/day for a limited time, a free cloud product called Splunk Storm which gives you 20GB/application for a 30 days, and a new enterprise product called Splunk Cloud running in AWS. The enterprise cloud product is especially useful for AWS hosted apps.
(08:20) Jon asks if there’s a planned cloud hosted offering for Windows Azure. Glenn says he’s pushing for it, but in the meantime it’s pretty easy to install it yourself.
(08:58) K. Scott asks about what he’d see if he used Glenn’s Azure app on a Windows Azure Web Site. Glenn lists some of the data and sources.
Developing Splunk apps and language support
(10:03) K. Scott asks about the process of writing a Splunk app. Glenn talks about all the language specific SDK’s they support and describes the process.
(11:20) K. Scott asks how they support so many languages in Splunk. Glenn says it’s pretty Unixy in that it works with streams, so all the language specific SDK’s work with that.
Using Splunk for evented data, not just logs
(12:25) Jon asks about some real world examples of things people are monitoring. Glenn talks about a recent DSL-like feature called data models, which allows business analysts to search through the data, and graphically pivot on it. One of the places people use that is for monitoring the entire dev lifecycle. Security auditing is a huge use case. 50% of the Fortune 100 uses Splunk. Glenn gives an example of how one of his co-workers wrote a Node app using Firebase’s bus feed to show a realtime map with bus location.
(16:00) Jon says this seems to blur the lines between logs and event sourcing. Glenn says it’s not just a log platform, and works really well with evented data.
(16:44) Jon asks what technologies it runs on, and if it’s using Hadoop. Glenn says Hadoop’s great, but not for realtime. They do have a product called Hunk which can access Hadoop HDFS information, though. It’s mostly C++ and Python (Django). They’ve recently rolled out an app frameowrk which makes it easy to customize Splunk using Django. There’s no database, since Splunk really just maintains indexes to data from other sources.
Glenn’s new book: Designing Evolvable Web APIs with ASP.NET
(19:25) Jon asks Glenn what he does in his free time. Glenn talks about the book he (and friends) are just finishing, called Designing Evolvable Web APIs with ASP.NET. It focuses on building a real system using hypermedia using ASP.NET Web API.
(20:35) Jon asks about versioning: are they using headers, URLs, etc.? Glenn says their argument is based on using additional media types and hypermedia. Hypermedia makes it easier to evolve your API because your clients are following links, not using hardcode URLs.
(22:15) Jon says hypermedia sounds great, but developers often want to follow defined links. Glenn says he doesn’t think it as a magical automaton, but both developers and code can look for new links as they’re added.
(23:40) Jon says it’s harder to evolve APIs if you’re thinking RPC style, but once you’re focused on resouces it’s easier. Glenn says this pattern has worked great for the web – clients just ignore things they don’t understand. Jon and Glenn say this is similar also to the move from relational databases to document databases.
(24:30) Glenn says it’s exciting to finally see some hypermedia APIs coming out: PayPal, GitHub, Amazon’s streaming APIs, and NPR’s recent API updates based on hypermedia.
(25:30) Glenn says the book doesn’t try to convince you that this is the only way, just shows the benefits. K. Scott says this sounds really useful to move from the theoretical to some concrete examples.
At NDC Jon and K. Scott talk to Scott Guthrie about his talk Building Real World Apps with Windows Azure, what’s new in Windows Azure, the advantage of provisioning and scaling up and down instantly, and more.
Scott talk: Building Real World Apps with Windows Azure
(00:18) Scott’s talk covered twelve patterns for building cloud apps using things like continuous delivery, transient fault handling, long term failures, etc.
What’s new in Windows Azure
(01:02) Jon asks Scott to overview the highlights of what’s new in Windows Azure over the past year
(01:25) Scott says they generally ship a major release every three weeks
(01:40) Scott talks about how they’re using agile approaches to development, and some services update as often as ten times a day
(02:19) Scott overviews some of the main things that shipped over the past year
Virtual Machine and Virtual Networking
Windows Azure Web Sites
(06:37) K. Scott asks how Auto-Scale came to be. Scott Guthrie tells the story about how it came from an acquisition of an Azure startup incubation project. The team joined at the end of March and the feature shipped in June.
(08:42) Scott talks about how Azure and Cloud Development help you move faster with illustrations of how quickly you can create and integrate services and infrastructure and support multiple regions.
(10:22) Scott talks about the advantages of being able to quickly scale both up and down. He talks about how Troy Hunt was able to scale up Azure instances to crunch through databases of breached passwords to make it easy to see if your password has been compromised, then scaled right back down and spent less than a dollar.
(14:09) K. Scott asks about Node.js support. Scott talks about how they’ve been supporting Node for a long time, and how cloud development lets you easily choose between tools for different applications.
(15:09)Jon asks Scott what books he’s been reading lately.
(15:45) He’s been reading a lot of work related books on things like supply chain management
(16:35) Scott mentions the new Web API book by Glenn Block and friends
(16:46) He went to Australia and read a book called Fatal Shore, a book about the founding of Australia
(00:18) Semantic Merge is a diff tool with a semantic understanding of your code.
(01:01) Jon asks about what languages Semantic Merge supports. It currently supports C#, Visual Basic.NET and Java, and they’re currently working on adding support for C, then C++.
(02:00) Jon noticed that they’re using Roslyn and asks about that. Pablo says that it worked really well, handling the parsing to allow them to focus on the important things like diff calculation and semantic merge calculation
(04:08) Jon asks about how they handle parsing outside of Roslyn and .NET. Pablo lists the different parsers they use for different languages. They’ve opened up the way that languages plug in, which allowed for a community contributed Delphi parser.
(5:33 Scott K. asks about support for Typescript, since it’s more strongly typed. Pablo says that’ll be easier, but they’re working through the language support list in order of demand.
What kind of semantics can Semantic Merge understand?
(06:28) K. Scott talks about what Semantic Merge does at a high level and asks about the different refactorings Semantic Merge can and can’t understand. Pablo explains a common scenario in which you’d be afraid to refactor code while adding or changing functionality if you know someone else is also working on it. Semantic Merge understands the refactorings so it’s easy to merge the actual changes. What Semantic Merge currently doesn’t handle is multi-file semantic merges, e.g. with code being refactored into another file. They’ve got a working prototype for that, but it’s harder to plug into different source control systems since they handle multi-file merges differently.
(08:52) Pablo points out that, while it’s called Semantic Merge, the diff functionality is really useful on its own.
The importance of graphical representation of merge issues
(09:17) Jon talks about how good the graphical representation is – both really easy to read and just generally nice looking. Pablo says they’ve put a lot of work into that and explains why they’ve designed it as they have.
(10:41) Scott K. says that developers are often stuck in a textual viewpoint for diff and merge, but a good graphical representation can be really useful. Pablo says that we’ve seen a recent revolution in source control tools, but we’re still using tools and technologies from twenty years ago. Jon says that the older ways of displaying diff and merge results with plus and minus lines was based on working with the old source control systems and mostly doing two-way merges.
(13:25) Pablo says it’s something that you really miss when it’s not there – big merges with lots of files look scary, but when you see that the actual changes are minimal it’s not such a big deal. Scott K. mentions a joke he saw on twitter about how a ten line code review finds ten issues, but a thousand line review passes easily.
(15:17) Jon asks how Semantic Merge has changed the way their team develops code, for instance by making them more ready to refactor code. Pablo gives an example with working on a year-old branch in which traditional diff gave him tons of merge conflicts but Semantic Merge only gave him one.
(17:19) Jon noticed that many of the samples were able to automatically merge everything and asks how Semantic Merge detects merge conflicts. Pablo explains how Semantic Merge not only is able to detect when changes don’t cause conflicts, but can also detect merge conflicts that other tools won’t find.
Version control integration
(19:36) Jon asks about which version control systems Semantic Merge integrates with. Pablo lists Git, Mercurial, TFS, Perforce, Sourcetree and Subversion and says that it’ll plug into just about anything because just about all version control systems use common conventions for diff / merge tool integration.
(20:51) Jon asks about their recent Linux support and asks if that’s done using Xamarin and Mono. They use Mono for common backend code, but wrote native front-end code for Linux using Gtk#. They’re currently working on an OSX version using MonoMac, which gives it a true native front-end with a standard Mac look and feel.
Pricing model and free licenses
(22:52) Jon asks about the pricing model. There’s a 15 day free trial and a monthly subscription for $4/month. They wanted to experiment with pricing to make it so inexpensive that pricing wasn’t an issue. Jon asks if the subscription checking is complex. Pablo says it give you a lot of leeway so it won’t block you if you’re coding on a plane or something. They don’t obsess over security since it’s such an inexpensive application to begin with.
(25:36) Jon asks about their free licenses for open source developers. Pablo says they use Mono extensively and have been offering open source licenses for Plastic SCM for a while. Pablo mentions some of the open source projects using Semantic Merge, including F-Spot and a lot of other Mono projects.
Semantic based insights
(26:57) Jon asks they could use their information about semantic changes to source code over time to offer other insights to developers. Pablo says that this is something they’ve been doing with Plastic SCM with features like semantic method history, so you can track changes to a method over time across renames, refactoring to other files, etc. They also can offer richer metrics, so you don’t just see lines of code changed but can understand methods changed, refactorings, etc. Their goal for a long time has been to transform version control from a delivery mechanism to a productivity tool for developers.
(29:04) Jon asks how Plastic SCM compares to other version control systems. Pablo recommends going to PlasticSCM.com and look at the branch explorer. It’s as powerful as Git but very easy to use. It’s fully decentralized. It’s very graphical, and you can do almost everything from the branch explorer. It integrates well with enterprise security with support for things like ACL’s. It (of course) offers support for a lot advanced merge scenarios. It’s been under development since 2005, they’re in version 5 right now. It’s free for every team under 15 developers.
(31:25) Jon asks if there’s a way to test-drive Plastic SCM against an existing Git repository. Pablo explains how to do that without changing version control systems, since Plastic SCM can natively use the Git API.
(34:25) K. Scot asks about an old blog post about a small Windows Git application client; Pablo says that’s no longer required as it’s built into Plastic SCM.
(34:55) Jon asks about where listeners can find out more about Semantic Merge and Plastic SCM.
(35:50) Jon mentions that he really likes the team page on the Plastic SCM site – all the faces follow the mouse cursor as you move it around. He’s easily amused.
(00:40) Jon says he heard about Daniel because of SisoDb, a document database running on top of SQL Server. Jon and Daniel talk about what SisoDb does and why it could be useful in a "SQL Server only" shop.
(01:55) Daniel introduced Jon to Max, who works as a developer evangelist at Cloudant and hacks on Node.js and CouchDb.
CouchDb and Cloudant basics, multi-master replication possibilities
(02:30) Jon asks Max what Cloudant does. Max says that Cloudant is a database as a service – a hosted, managed document database based on CouchDb.
(02:58) Max talks about multi-master replication and some of the implications, including PouchDb (which treats your browser as a CouchDb instance), and even running a node on your phone that you can replicate against. Jon’s mind is blown.
(04:30) Jon asks about the latency involved in using an HTTP database as a service. Max says that local caching helps, as well as having your database service physically close to your users (or app/web servers). Queries are always done against precomputed indexes, so query time is always logarithmic. Daniel says you can use replication to bring data as close as possible, and emphasizes the importance of in-application caching.
(06:38) Kevin says that there have been a lot of attempts at replication based systems over the years and asks what CouchDb does differently. Max says that the big difference is in the way CouchDb handles multi-version concurrency control by keeping revision trees. This lets them run a lockless system and investigate changes later. Daniel says that consistent hashing helps with this and explains the terms. Max talks about the use of revision numbers and conflict handling.
(09:08) Daniel says he likes that Cloudant adds clustering support, and he’s excited that Cloudant is contributing this back to CouchDb. Max says that they’ve also added a new administrative interface called Photon, which they’ll be contributing back.
(10:19) Daniel asks if this means that Big Couch is being deprecated. Max says yes, and Kevin asks for more information on what Big Couch is.
Migrating to CouchDb
(10:57) Jon asks about the migration path for applications using traditional RDBMSs to Cloudant or CouchDb. Max explains two options: the do it yourself option (uploading data as CSV’s or similar) or using the WEAVE@cloud service from CloudBees.
(14:13) Daniel asks about task of moving from traditional queries to map – reduce queries. Max talks about some of the migrations he’s been a part of, and talks about the use of Lucene queries as a bridge. Daniel says that in moving to document databases you really need to think differently about how you’ll consume the data, e.g. . Max talks about design documents, which store indexes, list functions.
(16:06) Jon says that when he started looking at document databases, he found that it was also helpful to store additional data in a way that it’s easy to query. Max gives an example using medical data in which you can normalize data as part of the map reduce process, so you rarely have to worry about schema. Daniel says that it’s the result of the map that’s being indexed. Max says that unlike some other systems like Hadoop that do the mapping in a batch, CouchDb updates indexes incrementally.
(18:42) Jon asks how CouchDb compares to Mongo. Max says he found Mongo to be a good transitional system from relational databases because the querying was similar, but it broke down at scale.
(19:50) Jon puts K Scott on the spot and asks how this strikes him due to his recent work with medical data on Mongo. K Scott says it’s good to know CouchDb is an option if they hit scaling issues.
(20:43) Jon asks about the process of migrating from Mongo to CouchDb. Max says he’s written a script that dumps data from JSON in Mongo to be imported into CouchDb. He says that on the surface, Mongo and CouchDb store data similarly so migrating data isn’t that hard – the real differences are in querying and locking.
(22:21) Jon asks how Cloudant compares to other database as a service offerings. Max lists some, and Daniel mentions Iris Couch. Daniel talks about how easy it is to get started with CouchDb on Windows, then migrate to Cloudant.
(24:12) Max talks about some of the features they’ve added recently, like Lucene queries and geographic querying. Max says that they contribute a lot to CouchDb.
(25:30) Jon asks how Cloudant integrates with the hosting providers they’ve got listed. Max says that they work to host Cloudant servers in the same datacenters as their hosting partners.
(27:05) Kevin asks how Cloudant charges. Max says that for dedicated clusters, it’s per-node and dependant on the hosting provider since they all charge differently. For multi-tenancy, it’s on a per-request and per-storage. Migrating between dedicated and multi-tenant is handled using the standard replication mechanism.
(27:39) Jon asks about the process of getting started with the free level. Max explains how it works and says they’ll only charge you if you exceed $5 per month, which is a good amount of use. Daniel says it takes less than 5 minutes to get started.
Client libraries and MyCouch
(29:03) Kevin asks if there are client libraries for most libraries. Max says there are, but most are just adapted from the CouchDb libraries.
(29:43) Daniel built MyCouch as a purely async library that doesn’t hide the domain knowledge of CouchDb. Jon asks about the overall flow of using the MyCouch NuGet package to get started. Daniel says he’s use Portable Class Library support to cover the different the different platforms.
(31:53) Jon asks about the Query.Configure interface to build a query.
(32:44) Jon asks about the history of Daniel’s interest in CouchDb and MyCouch.
Migrating from Cloudant to in-house CouchDb
(33:20) Kevin asks if Cloudant is a hosted version of CouchDb or a fork. Max says that currently it’s a fork, but they contribute a lot back.
(33:43) Kevin then asks about what would be required in bringing a Cloudant-hosted application back in-house to run under vanilla CouchDb. Max say that in addition to losing the managed / hosted value, you’d lose Lucene querying and (soon) the geo-indexing features. Daniel also points out Cloudant’s clustering support.
Questions from Twitter
(34:55) Rob Sullivan asks Daniel how working with SisoDb and CouchDb affect the way he views document databases and RDBMSs. Daniel says that he doesn’t want to see an ORM anymore and he’s noticed that a lot of people are creating hierarchical document structures in SQL Server when a document database would be a better fit. He says that there’s a little less safety in distributed document databases, and you just have to get used to working with that. Kevin asks about some of the application strategies people use to deal with that. Max says that CouchDb provides ACIDity at the document level, so as long as you wrap your transactions into a single document you’re fine. This leads to event sourcing, in which all your transactions are handled as separate documents.
(39:23) Steve Strong asks about the offline story to synchronize change changes to a web client. Max talks about PouchDb and how it works in web clients with intermittent data access.
(41:31) Jon asks if PouchDb and CouchDb could be used in peer-to-peer systems. Max says this is something he’s profoundly interested in. He’s done some conference talks about it and has a project called Quilter which is aimed at feature parity with Dropbox but with full user control, security and privacy by eliminating centralized network infrastructure. Daniel asks if it’s NSA-safe, and Max talks about how you can protect things using HTTPS and friend / reputation systems
(46:13) Kevin asks what it’s like working with CouchDb’s code, since a lot of it’s written in Erlang. Max says that it’s built around building effective distributed systems since it incorporates fault handling. Daniel talks about the low memory footprint and Max talks about the ability to pass native Erlang messages over arbitrary protocols including HTTP.
(49:02) Jon asks where to learn more about Erlang. Max points out the book (available online) called Learn You Some Erlang For Great Good.
(49:44) Jon asks about what it’s like to integrate Erlang into parts of an application.
CouchDb vs. Mongo
(50:51) Kevin asks why Mongo gets more press than CouchDb. Max says that Mongo has a similar interface to traditional RDBMSs, but a lot of it’s just been a marketing victory. He talks about some unappreciated CouchDb advantages, like the fact that it’s got a built-in REST interface. He also says that CouchDb scales better than Mongo due to technical differences such as multi-version concurrency control.
(00:34) Mads works on the ASP.NET team building tools for everything that has to do with web development. He’s also done a lot of open source development – BlogEngine.NET, Web Essentials and some other Visual Studio extensions.
(02:48) Visual Studio 2013 has Browser Link, which allows you to connect any browser with Visual Studio. Any extension in the browser or Visual Studio can talk to each other via a web socket connection. The refresh browser feature in Visual Studio 2013 is just a proof of concept, the real feature is the communications channel.
(04:07) Scott K Asks about the Page Inspector feature and whether that would be integrated with Browser Link. Mads says that Page Inspector was introduced with Visual Studio 2012. It includes browser tools and source mapping which allow you to trace the markup back to what generated it, including C# code and server controls. Mads took over the Page Inspector team almost a year ago, and they’re using the same underlying engine. Right now you don’t get live updates in Page Inspector with Visual Studio 2013, but with the Web Essentials extension you will.
(07:19 Jon asks about how the source mapping works. Mads explains that the ASP.NET runtime injects a script tag at the end of your page, and Visual Studio is listening for it to connect on a localhost endpoint. Mads explains that the Browser Link connection is only made under specific conditions – running locally, in debug, etc.
(09:10) Jon asks about some of the recent extensions Mads has demonstrated, especially the example which tracks unused CSS class names. Mads says this has been a long requested feature, but it’s only possible to do this right from inside the browser. They’re now able to add smart tags into the CSS editor to show unused CSS classes. It’s available now using Visual Studio 2013 and Web Essentials.
(12:24) Scott K asks if it’s possible to see which classes are overriding others. Mads said it’s not there yet, but on the way.
(17:23) Scott K asks if they can hook up the model binder to allow deserializing more complex types. Mads says it’s not available yet, but on the way.
(19:04) Jon says they could also test performance using testing automation, and Mads says that they could do quite a bit more with performance and browser testing by working with browser extensions – Page Speed, SEO, accessibility, etc. They can call off to any service anywhere on the internet.
(21:10) Jon asks about some of the extensions and prototypes he’s worked on. Mads says he’s wrote an extension for LESS and CSS editors which updates the page as you type – without even requiring you to save the CSS document.
(23:50) Mads talks about the inspect mode extension. When you hit ctrl-alt-I in the browser, you can hover over any DOM element and see the source in Visual Studio (including controls, views, Master Pages, partials, etc.).
(25:35) Mad talks about design mode (ctrl-alt-d) which turns any DOM element into a content editable field, which allows you to type in the browser and change the server-side code. He talks about some complexities due to changing the server-side code which throws off the source mapping, and how when they make some future changes to allow updating source maps on the fly they’ll be able to allow pretty complex browser-based design and editing.
(29:35) Scott K asks if they could use the shadow DOM to allow updating the source maps. Mads says that wouldn’t work with older browsers, and there’s some discussion of legacy browser support.
(36:12) Mads talks about the history of Web Essentials. It started out in 2010, but the old Visual Studio HTML and CSS editors limited what he could do. He used Web Essentials as the test project to make sure that Visual Studio 2012 supported the extensibility API’s, then released a new version to correspond with Visual Studio 2013. He open sourced it at BUILD 2013 in June.
(37:36) Jon asks about how Mads migrates features from Web Essentials to Visual Studio. Mads says that he does this on every Visual Studio release (including updates) which allows him to delete a lot of code. There are some features which don’t get migrated – niche features, features for which they’re still testing out the user experience. He talks about some neat features in Web Essentials that he likes, but he doesn’t think enough people use to justify migrating.
(39:57) Jon asks about the language Web Essentials supports. Mads lists Markdown, LESS and Coffeescript. Mads talks about how they were able to include LESS and Coffeescript support from Web Essentials while waiting on the Visual Studio 2012.2 release, then removed it when that update shipped. He talks about the problems they hit due to the editor overlap. Mads said that situation caused him to change his philosophy on features to add in Web Essentials – he’ll no longer include features in Web Essentials which could cause a conflict with Visual Studio, especially compiler related features; that’s why he removed TypeScript support from Web Essentials.
(44:14) Jon mentions robots.txt support in Web Essentials. Mads explains that this is a great example of how his personal web development frustrations turn into Web Essentials features. He’s hoping that open sourcing Web Essentials will lead other developers to contribute as well.
Web Dev Checklist
(46:35) Jon asks about Web Dev Checklist. Mads and Sayed were both working on building out some sites last year, and they came up with a list of important checks for any website – performance, accessibility, SEO, etc. They got on Hacker News and were happy that their site held up well under the traffic.
(49:16) Side Waffle is a Visual Studio extension which gives you a lot of templates so you can add things to your projects which were written the write way, by experts. They’ve got Angular controllers, Durandel, robots.txt, etc. They’re hoping for other developers to add new templates.
(51:36) Mads says that the teams at Visual Studio can’t create and maintain all the templates over time. Jon says he’s seen this again and again – new things get released but don’t always get maintained over the years. Mads says this makes it easy for developers to add and update templates.
(53:28) Mads says Sayed came up with the name from ordering a side order of waffles in a restaurant.
(54:24) Mads explains some of the technical complexities that he and Sayed had to deal with to allow adding new item templates to Visual Studio. Due to the strange ways they worked with MSBuild, Side Waffle isn’t allowed into the Visual Studio Gallery. They register Side Waffle as a Visual Studio gallery provider so when new templates are added, it will show up in the Visual Studio updates list. Jon’s confused, and Mads explains more about what’s going on.
Lightning Round questions from Twitter
(58:31) Barry Dorrans asks why Mads is pushing the #region agenda on unsuspecting HTML files.
(59:08) Warren Buckly asks “There is support for LESS, will there ever be support for SASS?”
(59:34) Jonas Eriksson asks Is it possible to extend the new HTML editor IntelliSense?”
(51:52) Jonas Eriksson asks if it’s possible to start a Grunt task and monitor its output.
(1:00:09) Saul asks how to create a static website in Visual Studio.
(1:01:07) Bret Ferrier asks about getting Angular IntelliSense with TypeScript.