Posts archived in .NET

I’d been trying to do a little reporting in RavenDB and needed to do a group several items.

The secret is that you can group into a new anonymous type, which I found out about on David Klein’s blog posting a few years ago.

So, my Reduce statement looks something like:

Reduce = results => from result in results
                    group result by
                        new
                            {
                                result.Date,
                                result.Application
                            }
                    into g
                    select
                        new
                            {
                                g.Key.Date,
                                g.Key.Application,
                                Count = g.Sum(x => x.Count)
                            };

I was surprised it worked so easily once I knew the secret ingredient.

RavenDB is a document database, and unlike a relational database (like say Microsoft SQL Server) – you can’t do your usual group-by type queries quite as simply.

Consider this scenario: I have a series of documents showing when users have performed certain actions within an application.
I want to produce a unique count of users, and a count of Application starts.

public class StatisticEntry
    {
        public DateTime Timestamp { get; set; }
        public string UserId { get; set; }
        public string Action { get; set; }
    }

 

My documents will look something like:

{ "Timestamp": "2010-01-01", "UserId": "Able", "Action": "Open"} 
{ "Timestamp": "2010-01-01", "UserId": "Barry", "Action": "Open"} 
{ "Timestamp": "2010-01-02", "UserId": "Barry", "Action": "Close"} 
{ "Timestamp": "2010-01-03", "UserId": "Barry", "Action": "Open"} 
{ "Timestamp": "2010-01-03", "UserId": "Charlie", "Action": "Open"} 

In SQL Server, I would do something like:

SELECT UserID, count(Timestamp) as Count 
FROM StatisticEntries 
WHERE Action=”Open” 
GROUP BY UserID

This would give me:

Row # UserID Count
1 Able 1
2 Barry 2
3 Charlie 1


For RavenDB instead I need to use a Map-Reduce query. The basics of this are that the first part (the Map) runs over the documents, and finds the matches. The reduce then takes the found matches, and summarises (reduces) the result set.

To make this magic happen, we need to know the results we want out at the end, and define this as a class:

public class StatisticResult
{
    public string UserId { get; set; }
    public int Count { get; set; }
}

We then create our index like so, adding a reference to Raven.Abstractions.dll

public class StatisticsUniqueUsers : AbstractIndexCreationTask<StatisticsEntry, StatisticResult> 
{ 
    public StatisticsUniqueVisitors() 
    { 

        Map = docs => from doc in docs 
                      where doc.Action == "Open"
                      select new 
                                 { 
                                     UserId = doc.UserId, 
                                     Count = 1 
                                 }; 
        Reduce = results => from result in results 
                            group result by result.UserId 
                            into g 
                            select new 
                                       { 
                                           UserId = g.Key, 
                                           Count = g.Sum(x => x.Count) 
                                       }; 
    } 
}

You then feed this class to RavenDB using this bit of magic (This took me a while to find):

Raven.Client.Indexes.IndexCreation.CreateIndexes(typeof(StatisticsUniqueUsers).Assembly, store);


NB: This will result in scanning this entire assembly for indexes. All of them will be created at that time. If any are invalid, you’ll get exceptions.

The CreateIndexes method should be called when your application starts – but be aware that you may get stale results temporarily while RavenDB produces the index.

So, Let me break the index class we created above:

The Map gives us a list of matching documents, with the intermediate results being:

{ "UserId": "Able", "Count": 1} 
{ "UserId": "Barry", "Count": 1} 
{ "UserId": "Barry", "Count": 1} 
{ "UserId": "Charlie", "Count": 1}

The Reduce then summarises these values and gives us the summarised values:

{ "UserId": "Able", "Count": 1} 
{ "UserId": "Barry", "Count": 2} 
{ "UserId": "Charlie", "Count": 1}

…so, we then have this index, which gives us a per-user count. But we just wanted unique users and total Opens?  Quite right.. It’s a little bit of a round trip, but we get there in the end.

We get the results by performing two queries:

int uniqueUsers = _documentSession.Query<StatisticsEntry, StatisticsUniqueUsers>().Count(); 
int totalOpens = _documentSession.Query<StatisticsEntry>().Where(s=>s.Action=="Open").Count();

The biggest ‘wtf’ for me was how to actually load the indexes into RavenDB – since there’s no apparent functions to do so on the IDocumentStore interface.

A few months ago for some unknown reason the Output pane in Visual Studio stopped displaying output from my application.

I’d get the build notices, exceptions, and thread/process exit information but any calls to Debug or Console to output information wouldn’t display.

tickprogramoutputIt turns out that you can de-select “Program Output” – and somehow it’d become deselected. Even now it still turns itself off, apparently by random.

Right clicking in the Output pane should let you re-select that value.

Edit: This post was linked from Stack Overflow – you might want to check back there for more/better discussion about the topic.

Recently Jeff Attwood wrote about how they are using ELMAH to get more information about the types of errors occurring in Stack Overflow.

Effectively ELMAH is designed as a ‘drop in’ fault capturing system for ASP.NET. It works really well there, and for many situations you can get along just fine without even needing to recompile your application (it does need some editing of the web.config though).

I wanted a way to capture more detail about the faults occurring in our dev and production environments, especially when working with WCF – since a lot of error detail tends to be hidden, or is difficult to reproduce.

Dropping in ELMAH into a WCF application will by default mean you miss the vast majority of errors – WCF swallows the error, and doesn’t let it get back up to ASP.NET.

There’s two ways you can go about fixing this:
Side Note: If you’re not hosting WCF in ASP.NET, then Option 2 may not be directly possible for you without some modification.

#1 – Wrap everything in try/catch blocks (if you didn’t already) and sprinkle this line around everywhere:

Elmah.ErrorSignal.FromCurrentContext().Raise(YourExceptionHere);

#2 Add a HttpHandler, and Decorate your Service(s) with an Error Handling attribute.

I borrowed the ServiceErrorBehaviourAttribute code from somewhere else, and I can’t find the source of it at the moment. Effectively this was so I could manipulate the HTTP Status Codes going back to the client when there was an error. It just so happens that this is a great way of capturing Exceptions and sending them to ELMAH at the same time.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;
using System.ServiceModel.Dispatcher;
using System.ServiceModel.Channels;
using System.ServiceModel.Description;
using System.Collections.ObjectModel;
using System.Net;
using System.Web;
using System.IO;
using Elmah;
namespace YourApplication
{
	/// <summary>
	/// Your handler to actually tell ELMAH about the problem.
	/// </summary>
    public class HttpErrorHandler : IErrorHandler
    {
        public bool HandleError(Exception error)
        {
            return false;
        }

        public void ProvideFault(Exception error, MessageVersion version, ref Message fault)
        {
            if (error != null ) // Notify ELMAH of the exception.
            {
                Elmah.ErrorSignal.FromCurrentContext().Raise(error);
            }
        }
    }
	/// <summary>
	/// So we can decorate Services with the [ServiceErrorBehaviour(typeof(HttpErrorHandler))]
	/// ...and errors reported to ELMAH
	/// </summary>
	public class ServiceErrorBehaviourAttribute : Attribute, IServiceBehavior
    {
        Type errorHandlerType;

        public ServiceErrorBehaviourAttribute(Type errorHandlerType)
        {
            this.errorHandlerType = errorHandlerType;
        }

        public void Validate(ServiceDescription description, ServiceHostBase serviceHostBase)
        {
        }

        public void AddBindingParameters(ServiceDescription description, ServiceHostBase serviceHostBase, Collection endpoints, BindingParameterCollection parameters)
        {
        }

        public void ApplyDispatchBehavior(ServiceDescription description, ServiceHostBase serviceHostBase)
        {
            IErrorHandler errorHandler;
            errorHandler = (IErrorHandler)Activator.CreateInstance(errorHandlerType);
            foreach (ChannelDispatcherBase channelDispatcherBase in serviceHostBase.ChannelDispatchers)
            {
                ChannelDispatcher channelDispatcher = channelDispatcherBase as ChannelDispatcher;
                channelDispatcher.ErrorHandlers.Add(errorHandler);
            }
        }
    }
}

Once you’ve added that, then it’s just a matter of decorating your Service like so:

    [ServiceContract(Namespace = "http://example.com/api/v1.0/")]
    [ServiceErrorBehaviour(typeof(HttpErrorHandler))]
    public class MyServiceService
    {
      // ...
    }

…and then making sure ELMAH is added as a reference, and adding it’s entries to your web.config.

Then you’ll be getting a whole stack of errors you otherwise may not have seen.

It’s also possible to log exceptions from higher up the chain (eg Databases, Files, etc) by using the line of code from Option 1.

Issues

Whilst ELMAH is great for capturing information about the request, I havn’t yet found any way to capture the original HTTP Request – this would be the ultimate goal for me.

It’s also not particularly easy to capture additional information (such as database records, or objects in cache, etc) without rolling your own copy of ELMAH.

All in all though – for a few minutes work, it’s one additional way to capture errors that your existing code may not be able to.

Yes, ELMAH even captures errors (in most situations) when your WCF services can’t start up (eg your fubar’ed some attributes).

Hope that helps.

NB: Use this code at your own risk, don’t blame me if it brings down your multi-million-dollar-per-hour application and causes you to go bankrupt.

Two methods that I keep finding myself needing are a way to Serialize and Deserialize objects in .NET 3.5.

Either for Unit Testing against a WebService of some kind, or for storing objects in memory to disk in XML for human-readable niceties.

Don’t forget to add in appropriate error handling code as needed.

With .NET 3.5 SP1, these methods will serialize (almost) any object to either XML or JSON, it was based in part off an example given in a long since forgotten forum post.

 

using System.IO;
using System.Runtime.Serialization; // System.Runtime.Serialization.dll (.NET 3.0)
using System.Runtime.Serialization.Json; // System.ServiceModel.Web.dll (.NET 3.5)
using System.Text;
namespace Serialization
{
    public static class Helpers
    {
        /// <summary>
        /// Declare the Serializer Type you want to use. 
        /// </summary>
        public enum SerializerType
        {
            Xml, // Use DataContractSerializer
            Json // Use DataContractJsonSerializer 
        }

        public static T Deserialize<T>(string SerializedString, SerializerType UseSerializer)
        {
            // Get a Stream representation of the string.
            using (Stream s = new MemoryStream(UTF8Encoding.UTF8.GetBytes(SerializedString)))
            {
                T item;
                switch (UseSerializer)
                {
                    case SerializerType.Json:
                        // Declare Serializer with the Type we're dealing with.
                        var serJson = new DataContractJsonSerializer(typeof(T));
                        // Read(Deserialize) with Serializer and cast
                        item = (T)serJson.ReadObject(s);
                        break;
                    case SerializerType.Xml:
                    default:
                        var serXml = new DataContractSerializer(typeof(T));
                        item = (T)serXml.ReadObject(s);
                        break;
                }
                return item;
            }
        }

        public static string Serialize<T>(T ObjectToSerialize, SerializerType UseSerializer)
        {
            using (MemoryStream serialiserStream = new MemoryStream())
            {
                string serialisedString = null;
                switch (UseSerializer)
                {
                    case SerializerType.Json:
                        // init the Serializer with the Type to Serialize
                        DataContractJsonSerializer serJson = new DataContractJsonSerializer(typeof(T));
                        // The serializer fills the Stream with the Object's Serialized Representation. 
                        serJson.WriteObject(serialiserStream, ObjectToSerialize);
                        break;
                    case SerializerType.Xml:
                    default:
                        DataContractSerializer serXml = new DataContractSerializer(typeof(T));
                        serXml.WriteObject(serialiserStream, ObjectToSerialize);
                        break;
                }
                // Rewind the stream to the start so we can now read it. 
                serialiserStream.Position = 0;
                using (StreamReader sr = new StreamReader(serialiserStream))
                {
                    // Use the StreamReader to get the serialized text out
                    serialisedString = sr.ReadToEnd();
                    sr.Close();
                }
                return serialisedString;
            }
        }
    }
}

Hopefully others will find this useful.

 

Updated - A little more generic now – can serialize to either Json or Xml as needed by altering the type param.

So one of the things you find in C# when you move from VB.NET is that you can’t do Optional parameters on methods.

Actually, you can – it’s just that the compiler in VB.NET is giving you a free ride by generating overloaded values for you automatically. Correction, Looks like C# is being difficult, see this FAQ

Anyway, say you have a VB.NET function like so:

Public Sub TestFunction(Param1 as String, Optional Param2 as String = “Default Value”)
    // Do Stuff
End Sub

In C# you can achieve the same thing by writing a stub like so:

public void TestFunction(string Param1)
{
TestFunction(Param1, “Default Value”);
}

public void TestFunction(string Param1, string Param2)
{
// Do Stuff
}

Yeah, it’s a little more code to write – but it’s really not that difficult to manage unless you have truely obscene numbers of optional parameters.

TechRepublic has a look at the same issue too, and gives Parameter arrays as an alternative. Under .NET 3.0+ with Anonymous types you can use some reflection niceties to do some other, similar things too.

On a different note, reading VB code after nearly a year straight of C# development feels a little strange.

I’m working on a new project at work where we’re dealing with data that updates frequently, at unpredictable times, used in across several different front-end services, and needs to scale to pretty decent traffic levels without going nuts on buying more hardware.

So, given all that, one of the things we’re looking at is using a distributed object caching layer, such as memcached.  If you’re not sure what this technology does, the quick summary is that it is used to store commonly accessed data in memory on your servers. One of the most common uses is to cache results from database queries. 

memcached started it’s life at Danga Interactive to solve issues scaling LiveJournal at 20 million+ pageviews per day. It has a proven track record in the Unix world, and a fairly significant base of knowledge on what works and various workarounds and solutions.

Whilst memcached is from Unix, there are also Windows based ports of the server, and also .NET clients so using it in our environment shouldn’t be an issue from the technical side. 

Recently Microsoft also announced their entry into this space with a project code named Velocity. It’s pretty similar to memcached, but also has some additional functions allowing things like Tagging and Regionalising (Partitioning) data.  There’s also more support at the moment for different cache expiry methods, and the roadmap includes additional redundancy bits too.

For anyone who is considering how their applications will scale up, there’s plenty more to read on the subject.

Dare Obasanjo has a post from July 2007 about memcached on Windows, and also more recently about Velocity.  Scott Hanselman (Who I’m happy to say is coming to Tech.Ed Australia 2008!) has a podcast up about Velocity, talking with Anil Nori – one of the smart fellows responsible for Velocity.

I’ll write some more on this as we progress down the build of this application.

So, today I discovered an issue which related to me doing two calls something a little like this:

- Execute dc.sp_Proc1
- If some condition exists, execute dc.sp_Proc2, and then Execute dc.sp_Proc1 again with the same parameters.
- Insert some records into the database.

The problem is, the first time you execute the sproc, it caches the result. This would be okay for most instances, but in mine – I’m actually after the updated result.

A quick bit of googling revealed this post by Chris Rock. This approach of “turn off object tracking” works Ok if you don’t need to insert records on that Data Context.

My quick, dirty, and (possibly) really wrong approach was just to spin up a new Data Context, and re-execute that sproc.

I promise I’ll find a more sane way of fixing this :)

With many web 2.0 applications there’s a basic three-tier architecture..   In our case the client is a Flex 3/Caringorm application, the Services are WCF/ASP.NET Web Services, and the Database SQL 2005.

One of the typical approaches to creating Web Services for this type of system is to use a CRUD type pattern. That is: all methods are based around either Creating, Retrieving, Updating, or Deleting records.  In most usually done on a per-table basis, and means that you’re effectively making the Web Services a HTTP enabled SQL client.

For our situation, this wasn’t really appropriate for a number of reasons, including complex relationships between tables, and a need to reduce the amount of network traffic.

Another concern, although relatively minor, is to reduce the amount of work needed by the Flex team to implement the Web Services. 

Ideally, we wanted to be able to share business objects as widely as possible, to reduce the amount of rework needed by everyone involved in implementing the interfaces.

Therefore we chose to go with task, or semantic based methods, and using the objects as needed by the Flex front-end.  The work of validation, and mapping to appropriate tables would be done by the Web services.

An example of this might be that a Document had many properties, such as Media Items (pictures, video, etc), Tags, Authors, etc.  However, within the database there might be a necessity to track Document Versions, What versions are Live, the relationships between Documents, Document Versions and Media Items. 

Because the objects that I needed to send/receive didn’t match the objects that needed to be saved in the database, I needed to write a lot of “left hand/right hand code”: ServiceDocument.Property =   SQLDocument.Property.  Most of this was fairly simple code to write, but tracking the places where this takes place can be grow to become quite a challenge when the solution grows to dozens of tables.

This is an approximate list of what I need to do to add a property to one table:

  • Add the Property to the Service Types
  • Add conversion pieces to transpose the Service Type to/from the LINQ to SQL Object equivalents.
  • Add the column to the Table in the Database Model for LINQ to SQL
  • Add the column to all Stored Procedures in the Database Model which reference this, removing and re-adding them if this means new properties too.  Don’t forget to ensure the return types on the re-added Stored Procedures are set correctly.
  • Add the columns to the actual Stored Procedures, update parameters, etc
  • Add the column to the actual Table

I can only imagine the Version Control conflict chaos that would ensue if you had several people making these changes concurrently.

I highly recommend grouping changes into a per-table basis, because it can take a while to go through all the additional pieces you have referencing the LINQ to SQL and Service Type object equivilents.

This is the first in (hopefully) a series of quick things I’ve picked up whilst tackling the previously mentioned project

So, I have a table something like this:

CREATE TABLE [dbo].[Product](
 
[ProductID] [int] IDENTITY(1,1) NOT NULL,
 
[Name] [nvarchar](100) NOT NULL,
   [Price] [int] NOT NULL,
    [LastSaveTimestamp] [datetime] NOT NULL CONSTRAINT [DF_Product_SaveTimestampDEFAULT (getutcdate())
) ON [PRIMARY]

The key here is the default value on the column: LastSaveTimestamp.

If I then try to, say insert a new column into this table, for example using this code:

  DatabaseContext dc = new DatabaseContext();
  Product product = new Product();
  product.Name = “test product”;
  product.Price = 50;
  dc.Products.InsertOnSubmit(product);
  dc.SubmitChanges(System.Data.Linq.ConflictMode.FailOnFirstConflict);

Then I’d get an exception like:

System.Data.SqlTypes.SqlTypeException: SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM..

The fix is actually really simple – In the table designer / DBML, you need to tell it that the column is auto-generated. Unfortunately this doesn’t seem to be automatically detected. It’s one of a few ‘just plain weird’ situations. 

AzamSharp has the fix details, with a handy-dandy screenshot over on his blog.