A Better Way
There comes a time in the career of every programmer when they take an earnest look at their way of going about their job and mutters to her or himself for the first time: "There must be a better way." For me, that first period of introspection came in 2001 - roughly three years into my career writing business applications. At the time, I had a litany of desktop and web applications all with their own databases and conventions. Many of these applications were one-off programs written to scratch minor itches, but once you start providing value through software development, expectations rise and the spigot opens.
As the number of applications rose, so did the number of potential users. Since I was developing for a music school, I was inevitably going to run up against a Mac user or two and anything developed in Access, Visual Basic or Borland C++ was going to alienate them. This was the point where I totally committed to the web as my primary development platform going forward and decided to convert whichever existing desktop applications I could to the web medium. I'd like to thank the web standards project and mozilla.org for creating the climate to make such a commitment possible.
As my plan was playing out, another problem became evident, my newer web apps were much better written than my older efforts. This wasn't really a surprise as experience naturally makes us better at pretty much any task, but it was especially painful during maintenance when I would have to make changes to older applications with less elegance in design.
One App to Rule Them All
As a solution to this problem I decided to build a base library with commonly used implementations, but as I pursued that effort I had a much more interesting idea. Instead of building a base library module and implementing it in separate applications, why not build a base application and implement specialized functionality in a modular fashion? The idea was a "mother" app that implemented all of the plumbing and child modules that used this plumbing in a unified manner.
The primary benefit at the time was being able to share authentication and authorization allowing single sign on functionality to the growing suite of applications I had in development. In addition, I found that there was opportunity to integrate the applications themselves with one another. For example, one of the applications was a concert attendance program where students swiped their ID cards into a desktop application which saved the current date, time, and concert to a database server connected to a website where students could log in and see their stats. For various reasons, the stats could be wrong, so I developed a "dispute" process where students could challenge the count. When the student created a dispute, I saved them into the work tracking application I used for support requests and assigned the incident to the relevant school administrator for processing. Also, the calendars from a calendar application I wrote to power the main school website fed in as the authoritative list of events for the concert attendance application.
I called this platform xNET. The idea was that an arbitrary (x) number of applications could share functionality through an application NETwork facilitated by shared data. The first iteration of xNET was completed in 2002 and written in ASP 3.0. ASP.NET was around at the time, but I was still learning it and there was such a paradigm shift between the two platforms that I figured it was better to write it in the familiar ASP and port it later than to stumble through it on the new technology.
I did end up porting xNET to .NET a year or so later and with that port came my first object-oriented design. My early .NET days were a little rough as I was still very stuck on Dreamweaver as my web development IDE. Dreamweaver was great for ASP but terrible with ASP.NET and I probably moved at a quarter of the speed I would have had I used Visual Studio. That's right, in my early .NET days I debugged code manually and also manually compiled all of my assemblies. I had a great ASP.NET book but it wasn't focused on Visual Studio at all. I kind of shake my head when I think about that primitive setup and hours upon hours of lost productivity from not having an interactive debugger, but being exposed directly to .NET without the massive obfuscation of Visual Studio gave me a good understanding of how the platform worked. Having said that, I still wouldn't recommend that route to anyone trying to learn.
I did eventually move to Visual Studio 2003 and it was then that I really appreciated a design based on objects instead of database recordset. Even though my design wasn't really n-Tiered yet (all data access was in my object methods, not a data access layer) the ability to encapsulate related functionality in one object instead of a base library of functions resonated. I took my first object-oriented programming class (C++) in 1995, but it wasn't until I had years of programming under my belt and was forced to adjust by .NET that I fully understood all the hubbub.
The Great Leap Forward
xNET cruised along for a few years as I got more acquainted with the .NET Framework, in 2005 I changed jobs and left the music school to work in the University research administration office as a web developer. They had a number of needs for custom software development so I naturally brought xNET along with me. While working on my first major project, I began to feel like this might be time for the platform to grow up.
At this point, xNET was object-oriented in the barest sense - it exposed data and logic using objects. This created a clean way to access data and perform operations, but I was starting to feel the pain of the massive amount of code required to create objects of even medium complexity. Creating member variables and all of the logic to populate those fields by itself was cumbersome and prone to induce a ton of copy/paste errors. I had people working for me that could work through those things but I still ended up with lots of problematic code. It was here that I once again said to myself: "There must be a better way."
This time, a "better way" meant a better way to develop software in general. I had years of experience under my belt at this point and felt like I had come a long way since I wrote my first production desktop application in 1998 but it was clear I had room for improvement. I wanted to create a way to make myself and my team more productive. My criteria for more productive was for us to have the ability to create basic object code quickly and correctly and to make the particulars of the database schema irrelevant when doing day to day coding. The latter requirement came from my frustration writing tons of convoluted SQL joins in object constructors and property collections that were the result of many-to-many associations along with a SQL injection attack that exposed vulnerabilities in my process.
It was at this point that I read a couple of books that would allow me to take the next step in systems design. The first was Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides (commonly known as the "Gang of Four") and the second was Patterns of Enterprise Application Architecture by Martin Fowler. I took a look at the Gang of Four book around the time I first got into object-oriented programming but I hadn't seen enough to understand it. This time with years of cases under my belt and a deep interest in honing my craft, the book made perfect sense and allowed me to instantly make my applications better. Most of the concepts I had brushed upon one way or another, but the simple elegance of the GoF solutions was like water in the desert. After completing the GoF book, I became obsessed with patterns. Most of the other books I read were pretty much rehashes of the GoF book (I highly recommend O'reilly's Head First Design Patterns book as a more user-friendly alternative to the GoF text) until I came across Martin Fowler's awesome tome on Enterprise Architecture.
The biggest takeaway from PoEAA for me was a solid understanding of how the domain and data access layers should interact. Much of what I had seen up to that point was a DAL that essentially marshaled some sort of database recordset object to the domain object, it was Fowler's DataMapper pattern that made me understand that the DAL should do more than shuttle recordsets around, it should normalize the way the domain object interacts with data irrespective of the source. The irony here is that, in the book, Fowler doesn't actually say this as his discussion is exclusively in the context of a database source, but the revelation isn't always planned by the revealer.
Almost as important for me was the dichotomy of the Domain Layer and Service Layer. Neither of these concepts was new, in fact the Service Layer is simply a GoF facade pattern, but these ideas in the context of fine / course graining and tight / loose coupling take encapsulation to another level and are critical components for thinking about interoperability.
With the knowledge gained from these books and my own experiences, I started designing xNET2 on the .NET2 platform, a radical redesign of the platform with numerous upgrades including the following:
- Standardized schema design in order to enable automated entity generation
- Auto-generation of test classes for each entity
- Decreased code required for new entities by factoring out every method possible into abstract base objects
- Elimination of the use of SQL so developers can focus on objects and not data
- The ability to create many-to-many object relationships without making schema changes
- Implementation of strongly-typed collection classes for object-oriented search
- XSS and SQL Injection threat mitigation through a DAL-level implementation of SQL statement parameterization and HTML filtering
The roll out of xNET2 paid off immediately. Thanks to code simplification, dramatic refactoring into tiers, and a new code generation process the quality and time to deployment of code improved dramatically. Seriously, the number of issues encountered after code was deployed to production fell by over 90% because of the testing and automation of class generation. It was at this point I felt fully actualized as a software developer.
Further Refinements
The core architecture hasn't changed much. In the xNET3 release, the only notable architecture change was my reducing the amount of tweaking after code generation by moving my module-oriented factories into late-bound instantiation operations using activator.createinstance(). xNET3 was really about getting xNET on the .NET 3.0 platform so I could take advantage of the Workflow Foundation and get my SOA on with the Windows Communication Foundation. I also updated my formerly ConllectionBase-derived strongly-typed collections to use the generic System.Collections.ObjectModel.Collection in order to add LINQ compatibility.
Currently I'm working on a .NET4.0-based xNET4 which is really a clean up of some of the things I implemented in version 3. I'm making a point to leverage the much-improved workflow foundation and round out my service layer implementation. I've gone from thinking about web services as a method to expose custom application functionality for external consumption to using them as a mechanism for cleanly exposing an entire research domain where data originates from an ugly amalgamation of disparate sources.
More importantly, service layers are great for decoupling logic from client applications which enables rapid expansion into new territory. I was able to write a full administrative interface for my portal application natively in Windows Phone 7 over a couple of hours and can't imagine an iPhone or Android implementation being much more difficult. In our current time where client platforms are rising and falling with the tides (iOS, Honeycomb, ONX, Windows Phone, WebOS, etc) the argument for making business logic generically accessible makes an incredible amount of sense. This also sparks conversations about service repositories and the like but I have philosophical issues to work through there.
Finally, I'm looking into the Entity Framework as a database source for my data mappers. It addresses almost all of the issues that designed xNET objects to fix and the tools to use it are built in Visual Studio as opposed to my solution that requires using a third party tool to generate the object code. My only concern is moving away from DataTables as my principal data marshaling objects as they are awesome for caching and storing data from diverse sources. However, I have been wanting to get a true unit of work implementation in place - I have a bit of a hack now - and entities comes with it prefabricated. That might be worth the loss of flexibility.
The Future
Going forward, I want to focus on client development and high performance backends. Since I've got a good handle on making business logic available, I would love to play around with some of these client technologies and come up with cool ways to make the business go, so to speak. Also, for various reasons, performance hasn't been a huge concern for most of my projects. The business case could be met within reasonable performance boundaries. However, I can see how the diversification of clients and the opening of systems to parties not completely indoctrinated in my shop can put pressures on infrastructure requiring scalably performant solutions.
I look forward to taking on these challenges and discussing the various issues that come along with resolving them.