Thursday, 29 January 2009

Are you Agile?

This simple question can lead to really heated debate. It's so, because the idea about what the Agile Software Development is can vary significantly among the people. To answer question "Are you Agile?" you have to first define the minimum criteria for being Agile, here is a list of potential candidates: self-organizing teams, incremental development in short iterations, high responsiveness to changing requirements, continuous adjustments, everyday contact with stakeholders, pair programing, TDD, (automated) testing and so on ...

The list may actually be much longer. Those are all desired practices but surly you don't have follow all of them to be Agile right? I figured that the best way to sort out this puzzle will be to check the basis - The Manifesto for Agile Software Development and Principles behind the Agile Manifesto.

What is not there:
  • nothing about pair programing, TDD and testing although "Agile processes promote sustainable development" so you don't have write tests and still you can say that you are Agile ;)
  • surprisingly, short iterations are not a must but remember to "deliver working software frequently"

What is on the list:
  • "highest priority is to satisfy the customer" - this is our main goal, in the end ... they pay for our time.
  • "business people and developers must work together daily throughout the project" and "the most efficient and effective method of conveying information to and within a development team is face-to-face conversation" - which means that you should avoid, whenever possible, writing documents, emails, using Skype or MSN. Talk to team-mates and stakeholders, ask questions and be proactive ... if you can't talk face-to-face then try to call someone rather then writing an email ... it might be much harder as it requires more effort but it also gives the best results.
  • "working software is the primary measure of progress" - I like this rule as it clearly states what really matters ... our job is to deliver working software, not just a random number of classes or lines of code ... job is done when your software works as expected by stackholder and is available for others.
  • "simplicity -- the art of maximizing the amount of work not done" and "continuous attention to technical excellence" - this prevents people from releasing shitty software and saying that Agile is about delivering something fast and then improving it later. Agile Development can't be an excuse for poor design and low quality.
  • "at regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly" - this principle is absolutely essential, if you follow a rigid process it's obvious that you are not Agile. Agile is about striving for optimal effectiveness therefore you have to be flexible, you have look for ways to improve the process you follow.

So are you Agile?

I my opinion there are two rules which have to be followed to answer "Yes":
  • deliver frequently - it guarantees frequent feedback and gives confidence that you are still on track with time and requirements.
  • adjust the process to be more effective - you have to constantly try new things, experiment how to make you work more enjoyable and effective.
What criteria do you use to call a process Agile?

Wednesday, 21 January 2009

Single point estimates are meaningless

How useful is statement "We estimated that project X will take 6 months"? What does it exactly tell you? Does it mean that you can be sure that the project will take 6 months? It's obvious that you can't be sure of that ... so if you were to sign a contract what would you do? It's quite typical approach to add 10% or 20% "just in case" which, in normal thinking, should guarantee meeting a deadline right? The problem is that single point estimates like 6 months for project X are useless without information about probability of success. Let me explain what I mean by that ...

Estimates are based on assumptions, specification etc. ... general rule is that estimates can be as accurate as details you have. So if you have only vague idea about the system then due to uncertainty your estimations can be really far away from the actual time. Some books are saying that actual time in such cases can be 4 times bigger the the estimate but it is also possible that it can be 4 times smaller! In our case, it means that the project X estimated for 6 months can take 1,5 to 24 months! Horrifying, isn't it?

Precision of estimates can be improved from ±4x to ±1,25x by providing details, that is why specification (requirements, UI design, use cases, acceptance criteria) can be so helpful.

What is the conclusion?

A good estimate is an estimate that provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets‥


Single point estimate doesn't provide a clear view, I believe that to have a clear picture it's required to know the best case and the worst case estimates. Then probability of success can be illustrated like this:



In the best case project will be finished in 20 weeks, but there are only 10% chances for that. In the worst case project will take 32 weeks, but also after week 28 we have 90% chances to get everything done. With those data project manager has a clear picture and can decided what risk is willing to take and plan accordingly.

So if statement "We estimated that project X will take 6 months" is meaningless then how should estimates be stated? Try to use ranges and rephrase statement to "We estimated that project X will take 5 to 7 months". Sometimes, when uncertainty is big, the range between best and worst case can be significant, but wide range is not a result of bad estimation, it's a result of lack of specification!

Data and definition of "good estimate" used in this post are from "Software Estimation: Demystifying the Black Art" book.

Monday, 19 January 2009

Diagnostic Console and Regular Expressions

In my last post I have introduced a Diagnostic Console plugin for EPiServer. Today I would like to show you one more example how it can be used with regular expressions to scan pages for some data.

In this example I want to find all images which are referenced by "MainContent" (XHTML String) property. To do that following script can be used:


   1:  clr.AddReference("System")
   2:  from System.Text.RegularExpressions import RegexOptions
   3:  
   4:  def processProperty(property, page):
   5:   if property.Value and property.Name == "MainContent":
   6:    regexp = "src=\"[a-zA-Z0-9\-_./]*\""
   7:    elements = DiagnosticUtils.FindMatchingElements(property.Value, regexp, RegexOptions.IgnoreCase)
   8:    if elements.Length > 0:
   9:     print str(page.PageLink) + " " + str(elements.Length) + "<br/>"
  10:     for element in elements:
  11:      tempvalue = element.Substring(5)
  12:      url = tempvalue.Substring(0, tempvalue.IndexOf("\""))
  13:      print url + "<br/>"
  14:  
  15:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)

I use DiagnosticUtils class to make those scripts shorter ... writing code without IntelliSense can be very annoying! I use ProcessProperties() method to scan all pages under selected root and one additional method FindMatchingElements(). FindMatchingElements was added just 2 days ago so you have to download the latest version of the plugin to have it. This method encapsulates creation of Regex object and as a result returns arrays of all matches. In above example I'm simply displaying all URLs but I can as well replace them with some different URL and update the property like this:


   1:  def processProperty(property, page):
   2:   if property.Value and property.Name == "MainContent":
   3:    regexp = "src=\"[a-zA-Z0-9\-_./]*\""
   4:    elements = DiagnosticUtils.FindMatchingElements(property.Value, regexp, RegexOptions.IgnoreCase)
   5:    if elements.Length > 0:
   6:     writableClone = page.CreateWritableClone()
   7:     for element in elements:
   8:      tempvalue = element.Substring(5)
   9:      url = tempvalue.Substring(0, tempvalue.IndexOf("\""))
  10:      filename = url.Substring(url.LastIndexOf("/") + 1)
  11:      print str(page.PageLink) + "   [" + url + "]   [" + filename + "]<br/>"
  12:  
  13:      updatedProperty = writableClone.Property.Get(property.Name).Value.Replace(url, "/images/news/2008/" + filename)
  14:      writableClone.Property.Get(property.Name).Value = updatedProperty
  15:  
  16:     DataFactory.Instance.Save(writableClone, SaveAction.Publish);
  17:  
  18:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)


Here you can download the latest DLLs and source code.

Thursday, 15 January 2009

Ultimate diagnostic tool for EPiServer

Recently I have written yet another admin plugin for EPiServer. The sole purpose of this plugin is to help developers finding pages, properties or to enable execution of some small scripts. Of course there are lots of diagnostic plugins for EPiServer so what is so special about this one? With this plugin it is possible to write your own scripts. Plugin called, Diagnostic Console, is responsible for executing your scripts and presenting results. You can use all .Net Framework libraries as well as all classes built on top of .Net (so EPiServer as well).

Here is an example of a simple script which job is to find all pages of specific page type. Lets say that we want to find all instances of News page type:



   1:  import clr
   2:  clr.AddReference("EPiServer")
   3:  from EPiServer import *
   4:  clr.AddReference("Cognifide.EPiServer.Plugin.DiagnosticConsole")
   5:  from Cognifide.EPiServer.Plugin.DiagnosticConsole import DiagnosticUtils
   6:  
   7:  def processPage(page):
   8:   if page.PageTypeName == "News":
   9:    print str(page.PageLink) + " " + str(page.PageName) + "<br/>"
  10:  
  11:  DiagnosticUtils.ProcessPages(PageReference("1"), processPage)


First 5 lines are importing required types, the interesting part is in lines 7 to 11. As you can see it's possible to write 4-5 lines of not-very-sophisticated code to find all instances of any arbitrary page type.

How is that possible?

The above script is written in IronPython. IronPython enables dynamic compilation and execution of a code and additionally it has access to .Net libraries. All the rest, like DiagnosticUtils class are just helpers to make certain operations easier.

What else can you do with Diagnostic Console?

I'm using this plugin only for a short time but I have already written a few useful scripts:
  1. This script checks all properties if they contain some specific string. ProcessProperties() method takes two parameters: reference to a root page and delegate which will be used to process properties:

       1:  def processProperty(property, page):
       2:   if property.Value and property.Value.ToString().Contains("/Upload/images"):
       3:    print str(page.PageLink) + " " + str(property.Name) + " " + property.Value + "<br/>"
       4:  
       5:  DiagnosticUtils.ProcessProperties(PageReference("1"), processProperty)

  2. This script is very similar to the previous one, the difference is that it uses ProcessPages() method which allows you to check all pages. Again, method takes two parameters: root page and delegate in which you can put your own operations. Version presented here sets a property MainImageAlt on all instances of News page type. It's very useful when you want to create a new mandatory property which should be then set on all existing pages:

       1:  def processPage(page):
       2:   if page.PageTypeName == "News":
       3:   DiagnosticUtils.UpdateAndSave(page, "MainImageAlt", "test2", SaveAction.Publish
       4:   print str(page.PageLink) + " " + str(page.PageName) + "<br/>" 
       5:  
       6:  DiagnosticUtils.ProcessPages(PageReference("1"), processPage)

    DiagnosticUtils.UpdateAndSave() method will create writable clone, update the page and save it.

  3. Here is another interesting script:

       1:  clr.AddReference("System.Web")
       2:  from System.Web import HttpRuntime
       3:  
       4:  print "<ul>"
       5:  
       6:  cache = HttpRuntime.Cache
       7:  for cacheItem in cache:
       8:   if not cacheItem.Key.StartsWith("EP") and not cacheItem.Key.StartsWith("PlugIn"):
       9:    if cacheItem.Value:
      10:     print "<li>" + str(cacheItem.Key) + ": [" + DiagnosticUtils.HtmlEncode(cacheItem.Value) + "]</li>"
      11:  
      12:  print "</ul>"

    This one displays all items stored in Http Cache. Please note that EPiServer items are filtered out.

  4. And finally a script which allows you to clean Http cache:

       1:  clr.AddReference("System.Web")
       2:  from System.Web import HttpRuntime
       3:  HttpRuntime.Close()

As you can see based on above examples you are not limited to EPiServer classes, you have access to literally everything. It makes this tool very flexible but also very dangerous so be careful and make sure that only good people can access it ;)

Last words...
  • if you guys find it interesting I can publish the source code on EPiCode, for now you can get the source code from here. Binary version to play with can be downloaded from here.
    Update
    : Diagnostic Console is already available on EPiCode.
  • big thanks to Aleks Sumowski who showed me the IronPython
Related articles:

Friday, 19 December 2008

The best IT blog of 2008

End of each year is a great moment for all sort of reviews, people tend to publish various different rankings like "The Best of ..." in all possible categories. This year I will add my 2 cents in a category which I find the most interesting: "The best IT blog of 2008".

I follow a few blogs as well as sites aggregating articles in some particular area (Agile Development, .Net, EPiServer). During this year I found lots of interesting and usefull posts there so choosing the best blog was difficult. My intention was to avoid nominating blog talking about one particular area or product. Main reason behind that decision was that people move from one project to another, from one job to another, I wanted to find a blog which can be relevant for us all the time.

Okey, let's move to the point ... in my opinion the best IT blog of 2008 is Jeff Atwood's Coding Horror!

There are a few reasons:

  1. New posts every 2-3 days
  2. Vast number of comments, Jeff's posts are just a start for a discussion.
  3. And at last but not least .. lots of great, thought-provoking posts ... my personal top 5:
Which blog is your "The best of 2008" ?

Wednesday, 17 December 2008

Fluent NHibernate - Integration Tests

In last two posts I have covered the Flunet NHibernate introduction and Coneventions together with AutoPersistenceModel. In this part I would like to show how Fluent NHibernate can speed up writing integration tests which should guarantee that mappings are correct. If there is a problem with mappings then for sure that is something you would like to want know about as fast as possible therefore some sort of automated tests are necessary.

Lets check what Fluent NHibernate has to offer:

1) SessionSource class which helps to deal with NHibernate configuration, session management and recreating database schema. Why is it useful? "Just" to check if mappings are correct you don't need to have any test data, you need only clear schema and database to connect with. You can use SQLite to create in-memory database. This is a neat and robust solution and moreover it eliminates external dependency -- we don't need any external database any more. Check the example:



   1:  [SetUp]
   2:  public void SetUp()
   3:  {
   4:      // create and configure persistance model including changes in Conventions
   5:      var persistenceModel = new PersistenceModel();
   6:      persistenceModel.Conventions.GetTableName =
   7:          type =>
   8:          String.Format("[{0}.{1}]", type.Namespace.Substring(type.Namespace.LastIndexOf('.') + 1), type.Name);
   9:      persistenceModel.Conventions.GetPrimaryKeyNameFromType = type => type.Name + "ID";
  10:      persistenceModel.Conventions.GetForeignKeyNameOfParent = type => type.Name + "ID";
  11:      persistenceModel.Conventions.GetForeignKeyName = prop => prop.Name + "ID";
  12:  
  13:      // add mappings
  14:      persistenceModel.addMappingsFromAssembly(typeof (Product).Assembly);
  15:  
  16:      // configure nhibernate using SQLite database
  17:      var config = new SQLiteConfiguration()
  18:          .InMemory()
  19:          .ShowSql();
  20:  
  21:      sessionSource = new SessionSource(config.ToProperties(), persistenceModel);
  22:  
  23:      // create NHibernate session
  24:      session = sessionSource.CreateSession();
  25:  
  26:      // recreate schema
  27:      sessionSource.BuildSchema(session);
  28:  }


Here is an implementation of BuildSchema(...) method:



   1:  public void BuildSchema(ISession session)
   2:  {
   3:      IDbConnection connection = session.Connection;
   4:  
   5:      string[] drops = _configuration.GenerateDropSchemaScript(_dialect);
   6:      executeScripts(drops, connection);
   7:  
   8:      string[] scripts = _configuration.GenerateSchemaCreationScript(_dialect);
   9:      executeScripts(scripts, connection);
  10:  }


As you can see, it uses NHibernate methods to generate DDL for dropping and creating tables for selected dialect (SQLite in this case). One thing worth noticing is that DDLs will be as accurate as mappings. So if you skip some information (like nullable fields) then don't expect to see it there!

2) Thanks to the SessionSource class we can query in-memory database and we have access to NHibernate Session ... it's time to check our mappings, we can use PersistenceSpecification class to do that:



   1:  [Test]
   2:  public void ProductReviewTest()
   3:  {
   4:      new PersistenceSpecification<ProductReview>(session)
   5:          .CheckProperty(x => x.Comments, "some nice comment")
   6:          .CheckProperty(x => x.EmailAddress, "test@test.com")
   7:          .CheckProperty(x => x.ModifiedDate, DateTime.Today)
   8:          .CheckProperty(x => x.Rating, 4)
   9:          .CheckProperty(x => x.ReviewDate, DateTime.Today)
  10:          .CheckProperty(x => x.ReviewDate, DateTime.Today)
  11:          .CheckProperty(x => x.ReviewerName, "test name")
  12:          .CheckReference(x => x.Product, CreateNewProduct())
  13:          .VerifyTheMappings();
  14:  }

Under the hood PersistenceSpecification class will save the object (ProductReview) to the database and then using another connection it will fetch the object back to make sure that all properties have correct values set. It's not a revolution but for sure it can save lots of time and thanks to neat and readable code it will increase maintainability.

Source Code

As always you can download the source code, and the whole sample web application. You will find there separate project for tests. I have created an AbstractTestBase class which is responsible for configuration and it also exposes NHibernate session. There are also tests for mappings, which use exposed NHibernate session and at least on my machine all tests pass ;)


(EDIT: Examples in this post have been updated on 8.02.2009 to reflect changes in Fluent NHibernate API)

Useful Links:
Other related posts:

Monday, 15 December 2008

Fluent NHibernate - Conventions and AutoPersistenceModel

Last time I have introduced the Fluent NHibernate, this time I would like to move further and show you how to use Conventions and AutoPersistenceModel.

But before we move to the point lets take a quick look on things that have changed in our sample web application:

1) Global.asax

NHibernate SessionFactory is created only once, on application start, then it is reused to create Sessions for each request. This approach allows us to use NHibernate Session in code behind of ASP.NET pages.

2) To demonstrate more interesting things I had to add one more table, so now we are playing with two tables:


I am, of course, still using AdventureWorks database. With a new table and relations between those two tables, our mappings have changed:

  • Product:


   1:  public class ProductMap : ClassMap<Product>
   2:  {
   3:      public ProductMap()
   4:      {
   5:          WithTable("Production.Product");        
   6:          Id(x => x.Id, "ProductID");
   7:   
   8:          Map(x => x.SellEndDate);
   9:          .
  10:          .
  11:          .
  12:          Map(x => x.ModifiedDate).Not.Nullable();
  13:   
  14:          HasMany<ProductReview>(x => x.ProductReview).WithKeyColumn("ProductID").AsBag().Inverse();
  15:      }
  16:  }

  • ProductReview:



   1:  public class ProductReviewMap : ClassMap<ProductReview>
   2:  {
   3:      public ProductReviewMap ()
   4:      {
   5:          WithTable("Production.ProductReview");
   6:   
   7:          Id(x => x.Id, "ProductReviewID");
   8:   
   9:          Map(x => x.Comments).WithLengthOf(3850);
  10:          Map(x => x.EmailAddress).Not.Nullable().WithLengthOf(50);
  11:          Map(x => x.Rating).Not.Nullable();
  12:          Map(x => x.ReviewDate).Not.Nullable();
  13:          Map(x => x.ReviewerName).Not.Nullable().WithLengthOf(50);
  14:          Map(x => x.ModifiedDate).Not.Nullable();
  15:   
  16:          References(x => x.Product, "ProductID");
  17:      }
  18:  }


Please note that I have skipped some irrelevant parts of the mappings but you can download the sample project to get source code. A bit of clarification:
  • Product can have 0 or multiple reviews. From the code point of view, Product class has additional IList property.
  • Review is about a product, therefore there is a not null foreign key in the ProductReview table.


Conventions

Now we can move to the point ... as you can see in above mappings, in some places it's required to specify column name or table name. This is because AdventureWorks database doesn't follow the default convention. (check Convention over Configuration design pattern) Differences:
  • table name is different then class name (Product vs Production.Product)
  • id property has different name then primary key column (Id vs ProductID)
  • properties representing links between tables have different names then foreign key column names (Product vs ProductID)
Luckily for us we don't have to repeat the same changes for all our mappings ... we can change the default convention ... here is how it can be done:



   1:  var models = new PersistenceModel();
   2:   
   3:  // table name = "Production." + class name
   4:  models.Conventions.GetTableName = type => String.Format("{0}.{1}", "Production", type.Name);
   5:   
   6:  // primary key = class name + "ID"
   7:  models.Conventions.GetPrimaryKeyNameFromType = type => type.Name + "ID";
   8:   
   9:  // foreign key column name = class name + "ID" 
  10:  //
  11:  // it will be used to set key column in example like this:
  12:  //
  13:  // <bag name="ProductReview" inverse="true">
  14:  //   <key column="ProductID" />
  15:  //   <one-to-many class="AdventureWorksPlayground...ProductReview, AdventureWorksPlayground, ..." />
  16:  // </bag>
  17:  models.Conventions.GetForeignKeyNameOfParent = type => type.Name + "ID";
  18:   
  19:  // foreign key column name = property name + "ID"
  20:  //
  21:  // it will be used in case like this:
  22:  // <many-to-one name="Product" column="ProductID" />
  23:  models.Conventions.GetForeignKeyName = prop => prop.Name + "ID";
  24:   
  25:  models.addMappingsFromAssembly(typeof(Product).Assembly);
  26:  models.Configure(config); 


and our mapping can be simplified to this:



   1:  public class ProductReviewMap : ClassMap<ProductReview>
   2:  {
   3:      public ProductReviewMap()
   4:      {
   5:          Id(x => x.Id);
   6:   
   7:          Map(x => x.Comments).WithLengthOf(3850);
   8:          Map(x => x.EmailAddress).Not.Nullable().WithLengthOf(50);
   9:          Map(x => x.Rating).Not.Nullable();
  10:          Map(x => x.ReviewDate).Not.Nullable();
  11:          Map(x => x.ReviewerName).Not.Nullable().WithLengthOf(50);
  12:          Map(x => x.ModifiedDate).Not.Nullable();
  13:   
  14:          References(x => x.Product);
  15:      }
  16:  }

EDIT: API for conventions was changed completely therefore code which is above is no longer valid, you can find update in this post - Conventions After Rewrite


AutoPersistenceModel

In fact ... in our mappings, there is not much left ... for sure you won't find there anything particularly creative therefore why not get rid of it completely? Yes ... it's possible ... if everything is 100% in accordance with the convention then you can simply use the following code to get NHibernate configured:



   1:  var models = AutoPersistenceModel
   2:      .MapEntitiesFromAssemblyOf<ProductReview>()
   3:      .Where(t => t.Namespace == "AdventureWorksPlayground.Domain.Production" );
   4:   
   5:  models.Conventions.GetTableName = prop => String.Format("{0}.{1}", "Production", prop.Name);
   6:  models.Conventions.GetPrimaryKeyNameFromType = type => type.Name + "ID";
   7:  models.Conventions.GetForeignKeyNameOfParent = type => type.Name + "ID";
   8:  models.Conventions.GetForeignKeyName = prop => prop.Name + "ID";
   9:   
  10:  models.Configure(config);


And that is all what you need ... a few POCO objects representing database tables, AutoPersistenceModel and you are ready to go. For sure it allows you to start very fast with development but what worries me is that there is no way to say that some properties are mandatory or have length limit. Specifying those additional data may help you to discover data related problems faster and moreover, it should increase performance of the NHibernate ... but is it worth it? What do YOU think?

(EDIT: Examples in this post have been updated on 6.02.2009 to reflect changes in Fluent NHibernate API)

Links:
Other interesting posts about Conventions and AutoPersistenceModel: