Very exciting news - an article co-authored by Barry Lawrence of BSTGlobal and myself based on a system now in production at Matrix has just been published in the Microsoft Architecture Journal.

The article is on the subject of Enterprise Identity Synchronization and describes a synchronization pattern and associate system architecture for keeping enterprise identity data up to date between disparate systems. In effect - it is a system for replicating identity memes around an organization.

This has proven to be a fascinating project from a number of standpoints, and I have learned a great deal from the experience of being involved in designing and building a services oriented workflow architecture. The implementation of the system has also been very fascinating, and is a real case study in how systems, people and processes are inherently intertwined. I will write a number of blog posts on the outcome of this project over the next few weeks.

I am attending the Patterns and Practices conference in Seattle at Microsoft. There were a number of other conferences I looked at attending, but I have wanted to visit the Microsoft campus for a long time, and it seemed to be a good excuse to do so.

The lecture schedule at the conference is pretty intense, and is held at the Microsoft conference center, so there is not a lot of opportunity to tour around either Seattle or even the campus itself. I am hoping that I can get a bit of a look around later in the week.

The conference features an excellent set of speakers, and is incredibly well organized and run. It has a nice blend of theoretical, practical, practices, methodologies. The conference center food has been outstanding, and there have been none of the issues with long lineups you often find at conferences.

They also have had some great after hours events, so far all involving beer (a very good thing). There are some great local beers - tried Fat Tire, Mginties(?), EBS so far. Tonight we have an evening with Microsoft Research, which should prove very interesting.

The hotel is a bit mundane (typical sort of suburban style), but is quiet and at least has a decent weight room.

I would highly recommend it to anyone who is involved in working with developing, designing or managing software projects.

One of the things I really like about this confrence is that it does not have a 'microsoft' centric focus. It is not a 'sales' pitch, which is something I was a bit concerned about when I first signed up for the confrence.

There are sessions that focus on Microsoft's development tool offerings of course, but the focus is not on the marketing but on the application of these tools to solve problems.

I have been busy refactoring the network, server and information infastructure layers for the company I work for recently. Now that that unpleasantness is nearly complete, I can again return to the more pleasant world of creating software.

However, even that has its dark side - of which one ongoing point of pain is the Data Access Layer.

There have been some changes made to the way the Enterprise Library is setup for the .net 2.0 framework. I will document the differences and how to here as I figure things out.

The first is that they have dropped the configuration block, as the 2.0 framework has a System.Configuration namespace that handles the configuration management internally.

Here is a sample:

Found this while looking for some info on the crypto enterprise library block: http://www.pnplive.com/

The patterns team does great stuff - use it all the time. Unfortunately I just heard Ward Cunnignham is leaving Microsoft and the patterns team to join the Eclipse project. I had the honour of having a beer with Ward at last years OOPSLA confrence in Vancouver...

Was looking around the google code competition. Interesting - but primarily algorithm centric contest.

I would need to do some serious study to get my algorithms back up to par again before getting involved something like that. Even so - algorithms are something that is on my list of continuous improvement study items. From that contest link, I found the Top Coders site, which has a great set of tutorials and resources on algorithms:

http://www.topcoder.com/tc?module=Static&d1=tutorials&d2=alg_index

At some point I will be getting into multiple configuration deployments for the code... So: in order to do that, it is convienient to have multiple configs for different environments.

This blog has a post and a plug-in for doing this...

Investigating this forms processing engine I am working on has lead me to software factories. I have been reading the Software Factories book - its good.

Here are some additional resources I have found while investigating this.

I am just working on setting up my in the latest m$ marketing speak, Smart Client application with the Enterprise Library updater block.

I am going to see if I can get it working off a linux server just for fun as well.

Method to be posted here as I figure it out.

Good example of a singleton pattern in C#

http://codeguru.earthweb.com/columns/DotNet/article.php/c7617/

I have a registry singleton source code pattern I will add up here once I get a few minutess

Been working on trying to get sorted collections figured out in C#. My first cut at it was a bit gnarly - (the post is on here), but in my latest quest, I found this article on sorting custom collections by J Ambrose Little.

 (More)

Secrets of Great Architects

By Don Awalt and Rick McUmber
RDA Corporation

July 2004

Summary: All great architects have mastered the ability to conceptualize a solution at distinct levels of abstraction. By organizing the solution into discrete levels, architects are able to focus on a single aspect of the solution while ignoring all remaining complexities. Presents techniques for applying levels of abstraction to IT solutions, and compares this to other engineering disciplines. (16 printed pages)

Contents

Applying Levels of Abstraction to IT Solutions
Levels of Abstraction: A Powerful Weapon for All Engineers
Core Principles When Applying Levels of Abstraction
Applying Levels of Abstraction to IT Systems
Simple Framework: Four Levels of Abstraction
Evolve the Levels Through Iterations
Revisit the Levels of Abstraction Core Principles
Scaling Levels to Support Enterprise Solutions
Benefits
Conclusion
Self Assessment

Applying Levels of Abstraction to IT Solutions

Enterprise architects are challenged by the vast amount of complexity they face. It's one thing to develop an isolated, departmental application that automates a business task. However, it is quite another to design and assemble a worldwide network of IT labs filled with applications, servers, and databases that support tens of thousands of IT users, all supporting various business activities. Compounding the complexity, the IT network must always be available, responsive, and protect the corporation's precious information assets. In addition to all of this, the IT network must be flexible enough to support the ever-changing needs of the business and to adopt new technologies as they emerge.

Some architects clearly stand out and thrive in this complexity. We've been fortunate to work side by side with some truly great analysts and architects over our careers. Reflecting on these experiences, we've analyzed what makes an exceptional architect.

Without exception, all great architects have mastered the ability to conceptualize a solution at distinct levels of abstraction. By organizing the solution into discrete levels, architects are able to focus on a single aspect of the solution while ignoring for the moment all remaining complexities. Once they stabilize that part of the solution, they can then proceed to other aspects, continuously evolving and refining the layers into a cohesive model, one that can ultimately be implemented.

Most software developers know that they should decompose the solution into levels of abstraction. However, this is very difficult to apply in practice on actual projects. Upon encountering the first hurdle, it's easy to abandon the levels in the rush to start coding. Great architects work through the challenges and are disciplined to maintain the levels throughout the entire project lifecycle. They realize that if they don't, they'll eventually drown in the complexity.

This article presents techniques for applying levels of abstraction to IT solutions. We first demonstrate the approach through a simple example, and then propose a structure of system artifacts based on formal levels of abstraction.

Levels of Abstraction: A Powerful Weapon for All Engineers

Other engineering disciplines, such as civil engineers, have been coping with complexity by leveraging levels of abstraction for centuries. Let's study how other, more mature, engineering disciplines apply levels of abstraction, starting with the electrical engineers who design computer systems that grow increasingly complex with each new generation.

Hardware Engineers

System designers model computer systems using levels of abstraction. Each level is well defined and provides a distinct perspective of the system. Many systems are designed at three primary levels: system, subsystem, and component, as shown in Figure 1

.

Layering enables the engineers to integrate an enormous amount of complexity into a single, working computer system. It's impossible to comprehend a computer strictly at the level of its atomic parts. There are approximately 25,000,000 transistors on a single Intel Itanium® chip alone.

This approach of breaking complexity down into layers of abstraction is of course not unique to IT-related disciplines. A similar approach is used in countless other disciplines, from aeronautical engineering to microbiology.

Core Principles When Applying Levels of Abstraction

All engineers follow this core set of principles when applying levels of abstraction. These principles also hold true when applying levels of abstraction to software.

The number and scope of the levels are well defined; in order for engineers to collaborate on a complex system, all team members must share the same understanding of the levels. So as designers make design decisions, they must file those decisions at the appropriate level of detail.

The three levels of abstraction are defined as follows:

Figure i. The three levels of abstraction defined

Figure ii. A simple framework for the levels of abstraction

Multiple views within each level

The complexity within a single level can become too much to grasp all at once. In this case, engineers present the design within a single level through multiple views. Each view presents a particular aspect of the design, yet remains at the same level of abstraction. For example, the motherboard engineer creates a view for each layer of the board, modeling the design of the connection pathways for that layer.

Figure 1. Levels of abstraction for a computer system

Must maintain consistency among the levels

In order for the system to function as intended, each subsequent layer must be a proper refinement of its parent layer. If the computer system designer switches from an IDE bus to a SCSI bus, then the interface specifications for all devices must also switch to SCSI. If the levels are not synchronized, the system won't perform as envisioned at the top level.

Applying Levels of Abstraction to IT Systems

Now that we've examined how other disciplines apply levels of abstraction, let's apply the technique to IT solutions¹. The following sections present techniques for applying levels of abstraction to model the requirements, design, and implementation of a typical IT application. The techniques are presented through a simple, instructional example of an online order system for a hypothetical retailer. In our example, we not only include the architecture, but expand the scope to include the system requirements and the business context as defined by the retail industry.

Simple Framework: Four Levels of Abstraction

Our simple example defines the following four levels of abstraction for an IT solution:

• Domain
• Business process
• Logical
• Physical

Within each level, we present both the dynamic view and static view of the behavior for that particular level. Whereas the dynamic view models the messaging among the objects, the static view models the structure and relationships among the objects.

Domain Level of Abstraction

Applying the scoping rules above, the retailer serves as the black box central actor in the domain level. The customer serves as the external actor. The domain level is modeled from the perspective of the customer. Only the purchase interactions are modeled. The communication mediums used to complete the purchase are not included at this level but are introduced at the business process level.

Figure 2. Domain-level dynamic view for purchasing items from a retailer

Figure 3. Domain level static view for purchasing items from a retailer

Dynamic view

The dynamic view within the domain level models the interactions between the customer and the retailer. The figure below summarizes the domain context and contains a simple use case narration of the business interactions.

Figure 4. Business process level dynamic view for purchasing items online from a retailer

Static view

The static view of the domain level models the class structure and their relationships of the objects witnessed in the use case. In other words, it tells what objects the customer needs to understand in order to accomplish the purchase transaction at this level of abstraction. Figure 5 presents the class diagram for the static view of the domain level.

Figure 5. Business process level static view for purchasing items online from a retailer

The customer is an instance of Person. The relationship between the customer and the retailer is embodied as an Account. All Purchases are associated with the customer's Account. The Purchase is associated with each of the Items being purchased. Each Item is of a specific Product, where Product follows the meta-class pattern. Instances of Product are in effect classes themselves. Modeling Product as a meta-class makes our model more flexible in that adding additional Products to the Catalog is purely a data-driven process and doesn't impact the class model. Rounding out the classes, each Payment is associated with its Purchase.

As you can see, the model at this level is representative for most any retailer—online or brick-and-mortar, large or small. This illustrates why the [Industry] Domain Model should indeed define the company as the black box central actor. Companies in the same industry tend to support the same set of business interactions with their external actors. Moreover, domain models exclude the specific business processes of the company, as they can vary widely among companies in the same industry.

The domain level focuses strictly on business interactions viewed from the perspective of the external actor. We must be careful to not include implementation mechanisms for accomplishing the interactions. These details belong at the next level of abstraction. So in this case, we only model browse, select, purchase, and pay. We do not model how these interactions are accomplished—through telephone, US Mail, e-mail, Web application, in person, check, credit card, or cash.

Business Process Level of Abstraction

The next level of abstraction models the company's business processes for realizing the interactions that were captured at the domain level. The system level "zooms inside" the company black box and identifies all employees and systems that collaborate in order to perform the business transaction. At this level, the system to be developed serves as the black box central actor.

Applying the scoping rules for the system level, the online order system serves as the black box central actor. The customer and the employee serve as external actors. The system level is modeled from the perspective of the customer and the employee. The customer performs the purchase online. Payment is made by credit card. The order is fulfilled by shipping the items to the customer's shipping address. Notification of shipment is sent by e-mail.

Dynamic view

The dynamic view replays the domain level purchase transaction, this time exposing the internal business processes of the retailer. Figure 4 summarizes the business process context and contains a simple use case narration of the interactions among the system and its actors.

Static view

The static view at this level refines the class model to capture the objects witnessed in the business process level use cases. In other words, "What objects do the customer and employee need to understand in order to create an order online and fulfill the order?" Figure 5 presents the class diagram for the business process static view. We refine the domain class model to capture the perspective at this level of abstraction. The Person, Account, and Company abstractions remain the same, as well as Catalog and Product. However, the abstract Purchase event from the domain model is replaced with an Order.

Orders contain LineItems, which are associated with the Product in the Catalog. Since this level models the company's internal business process, we need to capture the inventory on hand (an attribute of a stock-keeping unit (SKU), which represents an inventory of items at a particular location). We also model the customer's UserAccount, which provides access to the online system. Payment is accomplished by using a CreditCardAccount. Location represents a geopoint within the US and serves as the billing address, as well as the Order's shipping address. Shipment contains Items included in the Shipment.

The system level of abstraction usually requires a great deal of creativity because it is at this level that we invent ways to streamline business processes. In doing so, it is common to realize a single domain level transaction using several different mediums at the business process level. For example, a purchase can be accomplished online, over the phone, by mailing or faxing an order form, or in-person at the retail store. Each of these would need to be modeled at the business process level. Notice that even though the Credit Authorizer is an external actor to the retailer, it is introduced at this level because it is only needed to implement a business process that first appears at this level.

Lastly, notice that the system is technology independent. Our online purchase system could be implemented with any Web technology. Selecting technologies inside the system black box is an architecture decision.

Logical Level of Abstraction

The logical level zooms inside the system black box, exposing the high-level design of the system. The architect selects the technology and defines the high-level system structure. In our simple example, the system is comprised of a Microsoft IIS/Microsoft ASP.NET server that hosts the presentation, business, and data access layers and a Microsoft SQL Server database server that hosts the persistent data.

Dynamic view

The dynamic view at the logical level traces the message flows through the major components of the system. As an example, Figure 6 traces the flow when submitting the ConfirmOrder Web form.

Figure 6. Logical level dynamic view for purchasing items online from a retailer

Static view

The static view at this level also switches our perspective to the system internals. Whereas the business process level modeled the real-world abstractions that appear in the business processes, this level models the abstractions as they are to be represented inside the system. In an actual system, the architect would design the classes for each software layer (presentation, business, and data access). To keep this article brief, Figure 7 presents just the static design for the business layer to show how the system-level abstractions are refined to the design.

Figure 7. Logical level static view for purchasing items online from a retailer

The architect refines the system-level classes to design the business-layer interface.

All accounts and customers in the system are the retailer's, so it's not practical to create a single instance of Company and associate it with all accounts, so Company is omitted at this level. Rather than creating a separate instance for each CreditCardAccount, we simply store the credit card number and billing address with the Payment. Further, it is not practical for the system to create an instance for every Item sold, so Item is removed from the model and instead the model tracks the quantity of items ordered in the LineItem and the quantity of items shipped in the new ShippedItems class.

The architect also defines the granularity of the services that the business layer exposes. For this example, the business layer exports Create, Read, Update, and Delete (CRUD) services for Account, UserAccount, Order, Shipment, and Catalog. The ellipses indicate the CRUDing granularity.

Note that even though the classes at this level are not a proper superset of the business process classes, the architect arrives at this design by directly refining the business process classes, changing the perspective from outside the system to inside the system.

Physical Level of Abstraction

The physical level of abstraction captures the structure of the system implementation. The system is implemented as a network of nodes, each node configured with hardware and software. The three software layers in the logical view (presentation, business, and data) are physically implemented as code and deployed onto these nodes. Persistent classes in the logical view are physically stored in relational tables in a SQL Server database.

Dynamic view

The dynamic view traces the message flows through the nodes of the physical configuration. The ConfirmOrder HTTP post flows from the customer's browser through the Internet through the retailer's firewall to the Web server where Microsoft Windows forwards it to IIS, which passes it to Microsoft ASP.NET, which then dispatches ConfirmOrder.aspx. Fortunately, modern development tools insulate us from the majority of the physical network. Architects, however, need to understand the physical layer in order to avoid network bottlenecks and security exposures.

Static view

The static view (Figure 8) refines the persistent classes in the logical view to their physical representation. In our retail example, the business-layer classes are stored in the following SQL Server tables.

Figure 8. Physical level static view for purchasing items online from a retailer

Classes map to relational tables and attributes are implemented as columns. One-to-one relationships and one-to-many relationships are implemented using a foreign key. Optimistic concurrency is implemented by assigning a datetime field to each parent class being CRUDed.

When designing the logical level, the architect focuses mainly on implementing system functionality. Confident that the system functionality is covered, the architect can focus on optimizing the implementation at the physical level.

Evolve the Levels Through Iterations

Having established this framework, the architect evolves the solution over several iterations. Each iteration incorporates additional functionality—invoices, back orders, order in person, order by phone, and so on. In each case, the architect updates the appropriate level of abstraction, and then refines the updates to the physical implementation level.

Revisit the Levels of Abstraction Core Principles

Let's test our example against the core level of abstraction principles.

• The number and scope of the levels are well defined: We have four distinct levels; company black box, system black box, logical design inside the system, physical implementation
• Multiple views within each level: In this simple example, we presented a dynamic view and static view at each level.
• Must maintain consistency among the levels: If a change is made to the domain model, the impact of the changes must flow down to the lower levels. For example, if the retailer decides to offer maintenance contracts for its products, the analyst would add MaintenanceContract to the domain model and refine it to its physical representation. Synchronizing all levels is critical for maintaining large systems. As enhancement requests are submitted, the analyst performs an impact assessment to the appropriate level of detail. Some enhancements impact the domain level (and therefore all subsequent levels). Others merely impact the physical level.

Scaling Levels to Support Enterprise Solutions

Now that we've presented a simple example with four levels of abstraction, let's scale the approach to support solutions for IT enterprises. Figure 9 presents a Rational Unified Process (RUP) configuration that organizes project artifacts in to well defined levels of abstraction.

The levels in the table are described below.

Figure 9. RUP configuration organizing project artifacts into well defined levels of abstraction

• Domain. The domain level captures the business context for a project.
• Project vision. The project vision communicates the business impact the system will have on the business. It quantifies this impact in a return on investment analysis. The project vision represents the highest level of abstraction of the project.
• Business process. The system level models the business processes within the company. For extremely complex organizations, this level can be subdivided into sublevels: division, interdepartmental, and intradepartmental.
• UI specification. The UI specification designs the user interface that realizes the business processes. It is comprised of a UI-design document and a functional UI prototype.
• Detailed requirements. The detailed requirements specify the lowest level abstraction of the system requirements. It includes details such as data type formats and detailed business rules. It also contains the proficiency requirements such as performance, availability, security, internationalization, configuration, scalability, and flexibility requirements.
• Architecture. The system architecture is organized into six views:
  • Logical. Defines the software layers and major abstractions that perform the system functionality.
  • Concurrency. Captures the parallel aspects of the system, including transactions, server farms, and resource contention.
  • Security. Defines the approach for authentication, authorization, protecting secrets, and journaling.
  • Deployment. Defines the network topology and deployment configuration of the system.
  • Component. Defines the system components, their interfaces and dependencies.
  • Data. Defines the design structure of the persistent data.

Benefits

Organizing system artifacts into discrete levels of abstraction delivers several benefits:

• It separates the system requirements into three distinct levels of abstraction: Business Processes, UI specification, and detailed requirements. No longer do we overwhelm the business users with a single, monolithic functional specification. Instead, we communicate the system requirements in three refined levels of detail.
• Analysts and architects are able to harness the complexity into a single, integrated model of the system.
• Architects can focus on a single aspect of the system and integrate those decisions into the overall solution.
• The levels of abstraction form the structure of the system artifacts. For example, the software architecture document dedicates a subsection for each view.
• The levels of abstraction provide direct traceability from requirements to design to implementation. Traceability enables a team to perform an accurate impact assessment when evaluating change requests.
• After developing several systems using the same framework, patterns emerge at each level of abstraction. Organizations can catalog these patterns and other best practices within each level of abstraction. This catalog of best practices serves as the foundation of a process improvement program.

Conclusion

All engineering disciplines apply formal levels of abstraction in order to cope with complexity. Software is no exception. In order to realize the benefits of levels of abstraction, projects must:

• Formally identify the layers, each with a well-defined scope.
• Split complexity within a level into multiple views.
• Maintain consistency among the levels.

This article demonstrated how to apply levels of abstraction through a simple example, then scaled the approach to support enterprise IT solutions. It offered a RUP configuration framework that organizes system artifacts into well-defined levels of abstraction.

Self Assessment

• Does your current project apply levels of abstraction?
• Are the levels well-defined?
• Are the levels well understood by the project team?
• If the complexity becomes too great within a level, does the team split it into views?
• Does the team maintain consistency among the levels?
• Would your project benefit from levels of abstraction?

Great architects follow these principles instinctively. The rest of us must consciously apply levels of abstraction and exercise discipline to maintain the levels throughout the project lifecycle.

Resources

Cockburn, Alistair. Writing Effective Use Cases. New Jersey: Addison-Wesley, 2001

Kroll, Per and Kruchten, Philippe. The Rational Unified Process Made Easy: A Practitioner's Guide to the RUP. Boston MA: Pearson Education and Addison-Wesley, 2003

DeMarco, Tom and Plauger, P J Structured Analysis and System Specification. Prentice Hall PTR, 1979

Online copy of DoD standard 2167A can be found at http://www2.umassd.edu/SWPI/DOD/MIL-STD-2167A/DOD2167A.html.

Footnotes

1   Many have successfully applied levels of abstraction to software. Ed Yourdon and Tom DeMarco proposed structured analysis and structured system design in 1979. Many branches of the US Government standardized on DoD's 2167A standard, which requires systems to be composed of a hierarchy of hardware and software configuration items. The DBA community frequently applies levels of detail to model relational databases. In particular, the Bachman toolset and James Martin's Information Engineering Methodology (IEM) model databases logically, then physically. A Google search of "software levels of abstraction" returns several results; however, most are from the academic community and seem to focus on formal computer languages.

This is something I picked up on a while back - and now it looks like its coming into the fore:

http://www.softwarefactories.com/TheBook.html

http://msdn.microsoft.com/architecture/overview/softwarefactories/default.aspx?pull=/library/en-us/dnmaj/html/aj3softfac.asp

The Case for Software Factories

Jack Greenfield
Microsoft Corporation

July 2004

Summary: Briefly presents the motivation for Software Factories, a methodology developed at Microsoft. A Software Factory is a development environment configured to support the rapid development of a specific type of application. Software Factories are just a logical next step in the continuing evolution of software development methods and practices. However, they promise to change the character of the software industry by introducing patterns of industrialization. (10 printed pages)

Contents

Scaling Up Software Development
Facing the Changes Ahead, Again
Innovation Curves and Paradigm Shifts
Raising the Level of Abstraction
Industrializing Software Development
Can Software Be Industrialized?
Economies of Scale and Scope
What Will Industrialization Look Like?
Conclusion

Scaling Up Software Development

Software development, as currently practiced, is slow, expensive and error prone, often yielding products with large numbers of defects, causing serious problems of usability, reliability, performance, security and other qualities of service.

According to the Standish Group [Sta94], businesses in the United States spend around $250 billion on software development on approximately 200 projects each year. Only 16 percent of these projects finish on schedule and within budget. Another 31 percent are cancelled, mainly due to quality problems, for losses of about $81 billion. Another 53 percent exceed their budgets by an average of 189 percent, for losses of about $59 billion. Projects reaching completion deliver an average of only 42 percent of the originally planned features.

These numbers confirm objectively what we already know by experience, which is that software development is labor intensive, consuming more human capital per dollar of value produced than we expect from a modern industry.

Of course, despite these shortcomings, the products of software development obviously provide significant value to consumers, as demonstrated by a long-term trend of increasing demand. This does not mean that consumers are perfectly satisfied, either with the software we supply, or with the way we supply it. It merely means that they value software, so much so that they are willing to suffer large risks and losses in order to reap the benefits it provides. While this state of affairs is obviously not optimal, as demonstrated by the growing popularity of outsourcing, it does not seem to be forcing any significant changes in software development methods and practices industry-wide.

Only modest gains in productivity have been made over the last decade, the most important perhaps being byte-coded languages, patterns, and agile methods. Apart from these advances, we still develop software the way we did ten years ago. Our methods and practices have not really changed much, and neither have the associated costs and risks.

This situation is about to change, however. Total global demand for software is projected to increase by an order of magnitude over the next decade—driven by new forces in the global economy—like the emergence of China and the growing role of software in social infrastructure, by new application types like business integration and medical informatics, and by new platform technologies like Web services, mobile devices, and smart appliances.

Without comparable increases in capacity, it seems inevitable that total software development capacity is destined to fall far short of total demand by the end of the decade. Of course, if market forces have free play, this will not actually happen, since the enlightened self interest of software suppliers will provide the capacity required to satisfy the demand.

Facing the Changes Ahead, Again

What will change, then, to provide the additional capacity? It does not take much analysis to see that software development methods and practices will have to change dramatically.

Since the capacity of the industry depends on the size of the competent developer pool and the productivity of its members, increasing industry capacity requires either more developers using current methods and practices, or a comparable number of developers using different methods and practices.

While the culture of apprenticeship cultivated over the last ten years seems to have successfully increased the number of competent developers and average developer competency, apprenticeship is not likely to equip the industry to satisfy the expected level of demand for at least two reasons:

• We know from experience that there will never be more than a few extreme programmers. The best developers are up to a thousand times more productive than the worst, but the worst outnumber the best by a similar margin [Boe81].
• As noted by Brooks [Bro95], adding people to a project eventually yields diminishing marginal returns. The amount of capacity gained by recruiting and training developers will fall off asymptotically.

The solution must therefore involve changing our methods and practices. We must find ways to make developers much more productive.

Innovation Curves and Paradigm Shifts

As an industry, we have collectively been here before. The history of software development is an assault against complexity and change, with gains countered by losses, as progress creates increasing demand. While great progress has been made in a mere half century, it has not been steady. Instead, it has followed the well known pattern of innovation curves, as illustrated in Figure 1 [Chr97].

Figure 1. Innovation Curves

Typically, a discontinuous innovation establishes a foundation for a new generation of technologies. Progress on the new foundation is initially rapid, but then gradually slows down, as the foundation stabilizes and matures. Eventually, the foundation loses its ability to sustain innovation, and a plateau is reached. At that point, another discontinuous innovation establishes another foundation for another generation of new technologies, and the pattern repeats. Kuhn calls these foundations paradigms, and the transitions between them paradigm shifts [Kuh70]. Paradigm shifts occur at junctures where existing change is required to sustain forward momentum. We are now at such a juncture.

Raising the Level of Abstraction

Historically, paradigm shifts have raised the level of abstraction for developers, providing more powerful concepts for capturing and reusing knowledge in platforms and languages. On the platform side, for example, we have progressed from batch processing, through terminal/host, client/server, personal computing, multi-tier systems and enterprise application integration, to asynchronous, loosely coupled services. On the language side, we have progressed from numerical encoding, through assembly, structured, and object-oriented languages, to byte coded languages and patterns, which can be seen as language-based abstractions. Smith and Stotts summarize this progression eloquently [SS02]:

The history of programming is an exercise in hierarchical abstraction. In each generation, language designers produce constructs for lessons learned in the previous generation, and then architects use them to build more complex and powerful abstractions.

They also point out that new abstractions tend to appear first in platforms, and then migrate to languages. We are now at a point in this progression where language-based abstractions have lagged behind platform-based abstractions for a long time. Or, to put it differently, we are now at a point where tools have lagged behind platforms for a long time. Using the latest generation of platform technology, for example, we can now automate processes spanning multiple businesses located anywhere on the planet using services composed by orchestration, but we still hand-stitch every one of these applications, as if it is the first of its kind. We build large abstract concepts like insurance claims and security trades from small, concrete concepts like loops, strings, and integers. We carefully and laboriously arrange millions of tiny interrelated pieces of source code and resources to form massively complex structures. If the semiconductor industry used a similar approach, they would build the massively complex processors that power these applications by hand soldering transistors. Instead, they assemble predefined components called Application Specific Integrated Circuits (ASICs) using tools like the ones shown in Figure 2, and then generate the implementations.

Figure 2. ASIC Based Design Tools⁷

Can't we automate software development in a similar way? Of course we can, and in fact we already have. Database management systems, for example, automate data access using SQL, providing benefits like data integration and independence that make data driven applications easier to build and maintain. Similarly, widget frameworks and WYSIWYG editors make it easier to build and maintain graphical user interfaces, providing benefits like device independence and visual assembly. Looking closely at how this was done, we can see a recurring pattern.

• After developing a number of systems in a given problem domain, we identify a set of reusable abstractions for that domain, and then we document a set of patterns for using those abstractions.
• We then develop a runtime, such as a framework or server, to codify the abstractions and patterns. This lets us build systems in the domain by instantiating, adapting, configuring, and assembling components defined by the runtime.
• We then define a language and build tools that support the language, such as editors, compilers, and debuggers, to automate the assembly process. This helps us respond faster to changing requirements, since part of the implementation is generated, and can be easily changed.

This is the well-known Language Framework pattern described by Roberts and Johnson [RJ96]. A framework can reduce the cost of developing an application by an order of magnitude, but using one can be difficult. A framework defines an archetypical product, such as an application or subsystem, which can be completed or specialized in varying ways to satisfy variations in requirements. Mapping the requirements of each product variant onto the framework is a non-trivial problem that generally requires the expertise of an architect or senior developer. Language-based tools can automate this step by capturing variations in requirements using language expressions, and generating framework completion code.

Industrializing Software Development

Other industries increased their capacity by moving from craftsmanship, where whole products are created from scratch by individuals or small teams, to manufacturing, where a wide range of product variants is rapidly assembled from reusable components created by multiple suppliers, and where machines automate rote or menial tasks. They standardized processes, designs, and packaging, using product lines to facilitate systematic reuse, and supply chains to distribute cost and risk. Some are now capable of mass customization, where product variants are produced rapidly and inexpensively on demand to satisfy the specific requirements of individual customers.

Can Software Be Industrialized?

Analogies between software and physical goods have been hotly debated. Can these patterns of industrialization be applied to the software industry? Aren't we somehow special, or different from other industries because of the nature of our product? Peter Wegner sums up the similarities and contradictions this way [Weg78]:

Software products are in some respects like tangible products of conventional engineering disciplines such as bridges, buildings and computers. But there are also certain important differences that give software development a unique flavor. Because software is logical not physical, its costs are concentrated in development rather than production, and since software does not wear out, its reliability depends on logical qualities like correctness and robustness, rather than physical ones like hardness and malleability.

Some of the discussion has involved an "apples to oranges" comparison between the production of physical goods, on the one hand, and the development of software, on the other. The key to clearing up the confusion is to understand the differences between production and development, and between economies of scale and scope.

In order to provide return on investment, reusable components must be reused enough to more than recover the cost of their development, either directly through cost reductions, or indirectly, through risk reductions, time-to-market reductions, or quality improvements. Reusable components are financial assets from an investment perspective. Since the cost of making a component reusable is generally quite high, profitable levels of reuse are unlikely to be reached by chance. A systematic approach to reuse is therefore required. This generally involves identifying a domain in which multiple systems will be developed, identifying recurring problems in that domain, developing sets of integrated production assets that solve those problems, and then applying them as systems are developed in that domain.

Economies of Scale and Scope

Systematic reuse can yield economies of both scale and scope. These two effects are well known in other industries. While both reduce time and cost, and improve product quality, by producing multiple products collectively, rather than individually, they differ in the way they produce these benefits.

Economies of scale arise when multiple identical instances of a single design are produced collectively, rather than individually, as illustrated in Figure 3. They arise in the production of things like machine screws, when production assets like machine tools are used to produce multiple identical product instances. A design is created, along with initial instances, called prototypes, by a resource-intensive process, called development, performed by engineers. Many additional instances, called copies, are then produced by another process, called production, performed by machines and/or low-cost labor, in order to satisfy market demand.

Figure 3. Economies of Scale

Economies of scope arise when multiple similar but distinct designs and prototypes are produced collectively, rather than individually, as illustrated in Figure 4. In automobile manufacturing, for example, multiple similar but distinct automobile designs are often developed by composing existing designs for subcomponents, such as the chassis, body, interior, and drive train, and variants or models are often created by varying features, such as engine and trim level, in existing designs. In other words, the same practices, processes, tools, and materials are used to design and prototype multiple similar but distinct products. The same is true in commercial construction, where multiple bridges or skyscrapers rarely share a common design. However, an interesting twist in commercial construction is that usually only one or two instances are produced from every successful design, so economies of scale are rarely, if ever, realized. In automobile manufacturing, where many identical instances are usually produced from successful designs, economies of scope are complemented by economies of scale, as illustrated by the copies of each prototype shown in Figure 4.

Figure 4. Economies of Scope

Of course, there are important differences between software and either automobile manufacturing or commercial construction, but it resembles each of them at times.

• In markets like the consumer desktop, where copies of products like operating systems and productivity applications are mass produced, software exhibits economies of scale, like automobile manufacturing.
• In markets like the enterprise, where business applications developed for competitive advantage are seldom, if ever, mass produced, software exhibits only economies of scope, like commercial construction.

We can now see where apples have been compared with oranges. Production in physical industries has been naively compared with development in software. It makes no sense to look for economies of scale in development of any kind, whether of software or of physical goods. We can, however, expect the industrialization of software development to exploit economies of scope.

What Will Industrialization Look Like?

Assuming that industrialization can occur in the software industry, what will it look like? We cannot know with certainty until it happens, of course. We can, however, make educated guesses based on the way the software industry has evolved, and on what industrialization has looked like in other industries. Clearly, software development will never be reduced to a purely mechanical process tended by drones. On the contrary, the key to meeting global demand is to stop wasting the time of skilled developers on rote and menial tasks. We must find ways to make better use of precious resources than spending them on the manual construction of end products that will require maintenance or even replacement in only a few short months or years, when the next major platform release appears, or when changing market conditions make business requirements change, whichever comes first.

One way to do this is to give developers ways to encapsulate their knowledge as reusable assets that others can apply. Is this far fetched? Patterns already demonstrate limited but effective knowledge reuse. The next step is to move from documentation to automation, using languages, frameworks, and tools to automate pattern application.

Semiconductor development offers a preview into what software development will look like when industrialization has occurred. This is not to say that software components will be as easy to assemble as ASICs any time soon; ASICs are the highly evolved products of two decades of innovation and standardization in packaging and interface technology. On the other hand, it might take less than 20 years. We have the advantage of dealing only with bits, while the semiconductor industry had the additional burden of engineering the physical materials used for component implementation. At the same time, the ephemeral nature of bits creates challenges like the protection of digital property rights, as seen in the film and music industries.

Conclusion

This article has described the inability of the software industry to meet projected demand using current methods and practices. A great many issues are discussed only briefly here, no doubt leaving the reader wanting evidence or more detailed discussion. Much more detailed discussion is provided in the book Software Factories: Assembling Applications with Patterns, Models, Frameworks and Tools, by Jack Greenfield and Keith Short, from John Wiley and Sons. More information can also be found at http://msdn.microsoft.com/architecture/overview/softwarefactories and http://www.softwarefactories.com/, including articles that describe the chronic problems preventing a transition from craftsmanship to manufacturing, the critical innovations that will help the industry overcome those problems, and the Software Factories methodology, which integrates the critical innovations.

Copyright Declaration

This appears to be what I have been looking for:

Adaptive Object-Model Architecture - http://www.adaptiveobjectmodel.com/ - its the use of metadata in building an object model stored in a database. I think the open form class application for storing forms based data is well suited to this.

Implementing the Singleton Pattern in C#

http://www.yoda.arachsys.com/csharp/singleton.html

The singleton pattern is one of the best-known patterns in software engineering. Essentially, a singleton is a class which only allows a single instance of itself to be created, and usually gives simple access to that instance. Most commonly, singletons don't allow any parameters to be specified when creating the instance - as otherwise a second request for an instance but with a different parameter could be problematic! (If the same instance should be accessed for all requests with the same parameter, the factory pattern is more appropriate.) This article deals only with the situation where no parameters are required. Typically a requirement of singletons is that they are created lazily - i.e. that the instance isn't created until it is first needed.

There are various different ways of implementing the singleton pattern in C#. I shall present them here in reverse order of elegance, starting with the most commonly seen, which is not thread-safe, and working up to a fully lazily-loaded, thread-safe, simple and highly performant version. Note that in the code here, I omit the private modifier, as it is the default for class members. In many other languages such as Java, there is a different default, and private should be used.

All these implementations share four common characteristics, however:

• A single constructor, which is private and parameterless. This prevents other classes from instantiating it (which would be a violation of the pattern). Note that it also prevents subclassing - if a singleton can be subclassed once, it can be subclassed twice, and if each of those subclasses can create an instance, the pattern is violated. The factory pattern can be used if you need a single instance of a base type, but the exact type isn't known until runtime.
• The class is sealed. This is unnecessary, strictly speaking, due to the above point, but may help the JIT to optimise things more.
• A static variable which holds a reference to the single created instance, if any.
• A public static means of getting the reference to the single created instance, creating one if necessary.

Note that all of these implementations also use a public static method GetInstance as the means of accessing the instance. In all cases, the method could easily be converted to a property with only an accessor, with no impact on thread-safety or performance.

First version - not thread-safe

public sealed class Singleton
{
    static Singleton instance=null;

    Singleton()
    {
    }

    public static Singleton GetInstance()
    {
        if (instance==null)
            instance = new Singleton();
        return instance;
    }
}

As hinted at before, the above is not thread-safe. Two different threads could both have evaluated the test if (instance==null) and found it to be true, then both create instances, which violates the singleton pattern. Note that in fact the instance may already have been created before the expression is evaluated, but the memory model doesn't guarantee that the new value of instance will be seen by other threads unless suitable memory barriers have been passed.

Second version - simple thread-safety

public sealed class Singleton
{
    static Singleton instance=null;
    static readonly object padlock = new object();

    Singleton()
    {
    }

    public static Singleton GetInstance()
    {
        lock (padlock)
        {
            if (instance==null)
                instance = new Singleton();
            return instance;
        }
    }
}

This implementation is thread-safe. The thread takes out a lock on a shared object, and then checks whether or not the instance has been created before creating the instance. This takes care of the memory barrier issue (as locking makes sure that all reads occur logically after the lock acquire, and unlocking makes sure that all writes occur logically before the lock release) and ensures that only one thread will create an instance (as only one thread can be in that part of the code at a time - by the time the second thread enters it,the first thread will have created the instance, so the expression will evaluate to false). Unfortunately, performance suffers as a lock is acquired every time the instance is requested.

Note that instead of locking on typeof(Singleton) as some versions of this implementation do, I lock on the value of a static variable which is private to the class. Locking on objects which other classes can access and lock on (such as the type) risks performance issues and even deadlocks. This is a general style preference of mine - wherever possible, only lock on objects specifically created for the purpose of locking, or which document that they are to be locked on for specific purposes (e.g. for waiting/pulsing a queue). Usually such objects should be private to the class they are used in. This helps to make writing thread-safe applications significantly easier.

Third version - attempted thread-safety using double-check locking

public sealed class Singleton
{
    static Singleton instance=null;
    static readonly object padlock = new object();

    Singleton()
    {
    }

    public static Singleton GetInstance()
    {
        if (instance==null)
        {
            lock (padlock)
            {
                if (instance==null)
                    instance = new Singleton();
            }
        }
        return instance;
    }
}

This implementation attempts to be thread-safe without the necessity of taking out a lock every time. Unfortunately, there are four downsides to the pattern:

• It doesn't work in Java. This may seem an odd thing to comment on, but it's worth knowing if you ever need the singleton pattern in Java, and C# programmers may well also be Java programmers. The Java memory model doesn't ensure that the constructor completes before the reference to the new object is assigned to instance. The Java memory model is going through a reworking for version 1.5, but double-check locking is anticipated to still be broken after this.
• It almost certainly doesn't work in .NET either. Claims have been made that it does, but without any convincing evidence. Various people who are rather more trustworthy, however, such as Chris Brumme, have given convincing reasons why it doesn't. Given the other disadvantages, why take the risk? I believe it can be fixed by making the instance variable volatile, but that slows the pattern down more. (Of course, correct but slow is better than incorrect but broken, but when speed was one of the reasons for using this pattern in the first place, it looks even less attractive.) It can also be fixed using explicit memory barriers, but experts seem to find it hard to agree on just which memory barriers are required. I don't know about you, but when experts disagree about whether or not something should work, I try to avoid it entirely.
• It's easy to get wrong. The pattern needs to be pretty much exactly as above - any significant changes are likely to impact either performance or correctness.
• It still doesn't perform as well as the later implementations.

Fourth version - not quite as lazy, but thread-safe without using locks

public sealed class Singleton
{
    static readonly Singleton instance=new Singleton();

    // Explicit static constructor to tell C# compiler
    // not to mark type as beforefieldinit
    static Singleton()
    {
    }

    Singleton()
    {
    }

    public static Singleton GetInstance()
    {
        return instance;
    }
}

As you can see, this is really is extremely simple - but why is it thread-safe and how lazy is it? Well, static constructors in C# are specified to execute only when an instance of the class is created or a static member is referenced, and to execute only once per AppDomain. Given that this check for the type being newly constructed needs to be executed whatever else happens, it will be faster than adding extra checking as in the previous examples. There are a couple of wrinkles, however:

• It's not as lazy as the other implementations. In particular, if you have static members other than GetInstance, the first reference to those members will involve creating the instance. This is corrected in the next implementation.
• There are complications if one static constructor invokes another which invokes the first again. Look in the .NET specifications (currently section 9.5.3 of partition II) for more details about the exact nature of type initializers - they're unlikely to bite you, but it's worth being aware of the consequences of static constructors which refer to each other in a cycle.
• The laziness of type initializers is only guaranteed by .NET when the type isn't marked with a special flag called beforefieldinit. Unfortunately, the C# compiler (as provided in the .NET 1.1 runtime, at least) marks all types which don't have a static constructor (i.e. a block which looks like a constructor but is marked static) as beforefieldinit. I now have a discussion page with more details about this issue. Also note that it affects performance, as discussed near the bottom of this article.

One shortcut you can take with this implementation (and only this one) is to just make instance a public static readonly variable, and get rid of the method entirely. This makes the basic skeleton code absolutely tiny! Many people, however, prefer to have a method in case further action is needed in future, and JIT inlining is likely to make the performance identical. (Note that the static constructor itself is still required if you require laziness.)

Fifth version - fully lazy instantiation

public sealed class Singleton
{
    Singleton()
    {
    }

    public static Singleton GetInstance()
    {
        return Nested.instance;
    }
    
    class Nested
    {
        // Explicit static constructor to tell C# compiler
        // not to mark type as beforefieldinit
        static Nested()
        {
        }

        internal static readonly Singleton instance = new Singleton();
    }
}

Here, instantiation is triggered by the first reference to the static member of the nested class, which only occurs in GetInstance. This means the implementation is fully lazy, but has all the performance benefits of the previous ones. Note that although nested classes have access to the enclosing class's private members, the reverse is not true, hence the need for instance to be internal here. That doesn't raise any other problems, though, as the class itself is private. The code is a bit more complicated in order to make the instantiation lazy, however.

Performance vs laziness

In many cases, you won't actually require full laziness - unless your class initialization does something particularly time-consuming, or has some side-effect elsewhere, it's probably fine to leave out the explicit static constructor shown above. This can increase performance as it allows the JIT compiler to make a single check (for instance at the start of a method) to ensure that the type has been initialized, and then assume it from then on. If your singleton instance is referenced within a relatively tight loop, this can make a significant performance difference. You should decide whether or not fully lazy instantiation is required, and document this decision appropriately within the class.

Conclusion

There are various different ways of implementing the singleton pattern in C#. The final two are generally best, as they are thread-safe, simple, and perform well. I would personally use the fourth implementation unless I had some other static members which really shouldn't trigger instantiation, simply because it's the simplest implementation, which means I'm more likely to get it right. The laziness of initialization can be chosen depending on the semantics of the class itself.