Are you Agile?

This simple question can lead to really heated debate. It's so, because the idea about what the Agile Software Development is can vary significantly among the people. To answer question "Are you Agile?" you have to first define the minimum criteria for being Agile, here is a list of potential candidates: self-organizing teams, incremental development in short iterations, high responsiveness to changing requirements, continuous adjustments, everyday contact with stakeholders, pair programing, TDD, (automated) testing and so on ...

The list may actually be much longer. Those are all desired practices but surly you don't have follow all of them to be Agile right? I figured that the best way to sort out this puzzle will be to check the basis - The Manifesto for Agile Software Development and Principles behind the Agile Manifesto.

What is not there:

• nothing about pair programing, TDD and testing although "Agile processes promote sustainable development" so you don't have write tests and still you can say that you are Agile ;)
  
• surprisingly, short iterations are not a must but remember to "deliver working software frequently"

What is on the list:

• "highest priority is to satisfy the customer" - this is our main goal, in the end ... they pay for our time.
• "business people and developers must work together daily throughout the project" and "the most efficient and effective method of conveying information to and within a development team is face-to-face conversation" - which means that you should avoid, whenever possible, writing documents, emails, using Skype or MSN. Talk to team-mates and stakeholders, ask questions and be proactive ... if you can't talk face-to-face then try to call someone rather then writing an email ... it might be much harder as it requires more effort but it also gives the best results.
• "working software is the primary measure of progress" - I like this rule as it clearly states what really matters ... our job is to deliver working software, not just a random number of classes or lines of code ... job is done when your software works as expected by stackholder and is available for others.
• "simplicity -- the art of maximizing the amount of work not done" and "continuous attention to technical excellence" - this prevents people from releasing shitty software and saying that Agile is about delivering something fast and then improving it later. Agile Development can't be an excuse for poor design and low quality.
• "at regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly" - this principle is absolutely essential, if you follow a rigid process it's obvious that you are not Agile. Agile is about striving for optimal effectiveness therefore you have to be flexible, you have look for ways to improve the process you follow.

So are you Agile?

I my opinion there are two rules which have to be followed to answer "Yes":

• deliver frequently - it guarantees frequent feedback and gives confidence that you are still on track with time and requirements.
• adjust the process to be more effective - you have to constantly try new things, experiment how to make you work more enjoyable and effective.

What criteria do you use to call a process Agile?

Single point estimates are meaningless

How useful is statement "We estimated that project X will take 6 months"? What does it exactly tell you? Does it mean that you can be sure that the project will take 6 months? It's obvious that you can't be sure of that ... so if you were to sign a contract what would you do? It's quite typical approach to add 10% or 20% "just in case" which, in normal thinking, should guarantee meeting a deadline right? The problem is that single point estimates like 6 months for project X are useless without information about probability of success. Let me explain what I mean by that ...

Estimates are based on assumptions, specification etc. ... general rule is that estimates can be as accurate as details you have. So if you have only vague idea about the system then due to uncertainty your estimations can be really far away from the actual time. Some books are saying that actual time in such cases can be 4 times bigger the the estimate but it is also possible that it can be 4 times smaller! In our case, it means that the project X estimated for 6 months can take 1,5 to 24 months! Horrifying, isn't it?

Precision of estimates can be improved from ±4x to ±1,25x by providing details, that is why specification (requirements, UI design, use cases, acceptance criteria) can be so helpful.

What is the conclusion?

A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets‥

Single point estimate doesn't provide a clear view, I believe that to have a clear picture it's required to know the best case and the worst case estimates. Then probability of success can be illustrated like this:



In the best case project will be finished in 20 weeks, but there are only 10% chances for that. In the worst case project will take 32 weeks, but also after week 28 we have 90% chances to get everything done. With those data project manager has a clear picture and can decided what risk is willing to take and plan accordingly.

So if statement "We estimated that project X will take 6 months" is meaningless then how should estimates be stated? Try to use ranges and rephrase statement to "We estimated that project X will take 5 to 7 months". Sometimes, when uncertainty is big, the range between best and worst case can be significant, but wide range is not a result of bad estimation, it's a result of lack of specification!

Data and definition of "good estimate" used in this post are from "Software Estimation: Demystifying the Black Art" book.

Diagnostic Console and Regular Expressions

In my last post I have introduced a Diagnostic Console plugin for EPiServer. Today I would like to show you one more example how it can be used with regular expressions to scan pages for some data.

In this example I want to find all images which are referenced by "MainContent" (XHTML String) property. To do that following script can be used:

   1:  clr.AddReference("System")

   2:  from System.Text.RegularExpressions import RegexOptions

   3:  

   4:  def processProperty(property, page):

   5:   if property.Value and property.Name == "MainContent":

   6:    regexp = "src=\"[a-zA-Z0-9\-_./]*\""

   7:    elements = DiagnosticUtils.FindMatchingElements(property.Value, regexp, RegexOptions.IgnoreCase)

   8:    if elements.Length > 0:

   9:     print str(page.PageLink) + " " + str(elements.Length) + "<br/>"

  10:     for element in elements:

  11:      tempvalue = element.Substring(5)

  12:      url = tempvalue.Substring(0, tempvalue.IndexOf("\""))

  13:      print url + "<br/>"

  14:  

  15:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)

I use DiagnosticUtils class to make those scripts shorter ... writing code without IntelliSense can be very annoying! I use ProcessProperties() method to scan all pages under selected root and one additional method FindMatchingElements(). FindMatchingElements was added just 2 days ago so you have to download the latest version of the plugin to have it. This method encapsulates creation of Regex object and as a result returns arrays of all matches. In above example I'm simply displaying all URLs but I can as well replace them with some different URL and update the property like this:

   1:  def processProperty(property, page):

   2:   if property.Value and property.Name == "MainContent":

   3:    regexp = "src=\"[a-zA-Z0-9\-_./]*\""

   4:    elements = DiagnosticUtils.FindMatchingElements(property.Value, regexp, RegexOptions.IgnoreCase)

   5:    if elements.Length > 0:

   6:     writableClone = page.CreateWritableClone()

   7:     for element in elements:

   8:      tempvalue = element.Substring(5)

   9:      url = tempvalue.Substring(0, tempvalue.IndexOf("\""))

  10:      filename = url.Substring(url.LastIndexOf("/") + 1)

  11:      print str(page.PageLink) + "   [" + url + "]   [" + filename + "]<br/>"

  12:  

  13:      updatedProperty = writableClone.Property.Get(property.Name).Value.Replace(url, "/images/news/2008/" + filename)

  14:      writableClone.Property.Get(property.Name).Value = updatedProperty

  15:  

  16:     DataFactory.Instance.Save(writableClone, SaveAction.Publish);

  17:  

  18:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)

Here you can download the latest DLLs and source code.

Ultimate diagnostic tool for EPiServer

Recently I have written yet another admin plugin for EPiServer. The sole purpose of this plugin is to help developers finding pages, properties or to enable execution of some small scripts. Of course there are lots of diagnostic plugins for EPiServer so what is so special about this one? With this plugin it is possible to write your own scripts. Plugin called, Diagnostic Console, is responsible for executing your scripts and presenting results. You can use all .Net Framework libraries as well as all classes built on top of .Net (so EPiServer as well).

Here is an example of a simple script which job is to find all pages of specific page type. Lets say that we want to find all instances of News page type:

   1:  import clr

   2:  clr.AddReference("EPiServer")

   3:  from EPiServer import *

   4:  clr.AddReference("Cognifide.EPiServer.Plugin.DiagnosticConsole")

   5:  from Cognifide.EPiServer.Plugin.DiagnosticConsole import DiagnosticUtils

   6:  

   7:  def processPage(page):

   8:   if page.PageTypeName == "News":

   9:    print str(page.PageLink) + " " + str(page.PageName) + "<br/>"

  10:  

  11:  DiagnosticUtils.ProcessPages(PageReference("1"), processPage)

First 5 lines are importing required types, the interesting part is in lines 7 to 11. As you can see it's possible to write 4-5 lines of not-very-sophisticated code to find all instances of any arbitrary page type.

How is that possible?

The above script is written in IronPython. IronPython enables dynamic compilation and execution of a code and additionally it has access to .Net libraries. All the rest, like DiagnosticUtils class are just helpers to make certain operations easier.

What else can you do with Diagnostic Console?

I'm using this plugin only for a short time but I have already written a few useful scripts:

1. This script checks all properties if they contain some specific string. ProcessProperties() method takes two parameters: reference to a root page and delegate which will be used to process properties:
   
   

      1:  def processProperty(property, page):
   

      2:   if property.Value and property.Value.ToString().Contains("/Upload/images"):
   

      3:    print str(page.PageLink) + " " + str(property.Name) + " " + property.Value + "<br/>"
   

      4:  
   

      5:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)
   
   

2. This script is very similar to the previous one, the difference is that it uses ProcessPages() method which allows you to check all pages. Again, method takes two parameters: root page and delegate in which you can put your own operations. Version presented here sets a property MainImageAlt on all instances of News page type. It's very useful when you want to create a new mandatory property which should be then set on all existing pages:
   
   

      1:  def processPage(page):
   

      2:   if page.PageTypeName == "News":

      3:   DiagnosticUtils.UpdateAndSave(page, "MainImageAlt", "test2", SaveAction.Publish

      4:   print str(page.PageLink) + " " + str(page.PageName) + "<br/>" 

      5:  
   

      6:  DiagnosticUtils.ProcessPages(PageReference("1"), processPage)
   
   

   DiagnosticUtils.UpdateAndSave() method will create writable clone, update the page and save it.
   


3. Here is another interesting script:
   
   
   

      1:  clr.AddReference("System.Web")

      2:  from System.Web import HttpRuntime

      3:  
   

      4:  print "<ul>"

      5:  
   

      6:  cache = HttpRuntime.Cache

      7:  for cacheItem in cache:

      8:   if not cacheItem.Key.StartsWith("EP") and not cacheItem.Key.StartsWith("PlugIn"):

      9:    if cacheItem.Value:

     10:     print "<li>" + str(cacheItem.Key) + ": [" + DiagnosticUtils.HtmlEncode(cacheItem.Value) + "]</li>"

     11:  
   

     12:  print "</ul>"

   
   
   This one displays all items stored in Http Cache. Please note that EPiServer items are filtered out.
   


4. And finally a script which allows you to clean Http cache:
   
   
   

      1:  clr.AddReference("System.Web")

      2:  from System.Web import HttpRuntime

      3:  HttpRuntime.Close()

As you can see based on above examples you are not limited to EPiServer classes, you have access to literally everything. It makes this tool very flexible but also very dangerous so be careful and make sure that only good people can access it ;)

Last words...

• if you guys find it interesting I can publish the source code on EPiCode, for now you can get the source code from here. Binary version to play with can be downloaded from here.
  Update: Diagnostic Console is already available on EPiCode.
  
• big thanks to Aleks Sumowski who showed me the IronPython

Related articles:

• Diagnostic Console and Regular Expressions
  
• Dummy Content Generator for EPiServer