Remember the old metaphysical question: “if a tree falls in the forest, and no one’s around to hear it, does it make a sound?”

Well, here’s an equally unanswerable question: “if someone designs a web site, and no one’s around to measure its performance, was it a success?” If you designed or manage that web site, your job depends on being able to answer that question. However, many user experience designers have historically avoided the question with the enigmatic and frustrating non-answer: “it depends.” Well, the time is overdue for us to take ownership of measuring the performance of our designs. We can start by understanding what performance means.

“Performance” implies that someone is observing the performance – the audience. And there’s an implication of critique, or measurement of the performance. In order to guide the evolution of the design to fulfill some strategy, the design’s performance must be measured.

Performance implies an audience.

The performance of any creative effort is subject to interpretation by the people observing the performance, and those people are called “the audience”. To reiterate the argument I made in Part 1, user experience designers aren’t creating art for self-expression. We’re creating art for use. My design is made to perform a particular function, made useful by having an audience that values the design’s function.

If I design a web application behind a firewall that no one ever finds or uses, what have I accomplished, other than self-expression? By definition, user experience design requires an other (the user) to experience the work. My design must be used in order to fulfill its destiny.

Performance implies measurement.

The audience is neither monolithic nor homogeneous. Each audience member will have their own unique set of values, motivations, cultural filters, and personality traits, which will affect how they view the performance. They will each bring their own criteria by which they will judge the performance, whether they know it or not.

In the movie theater, I might judge a performance based on the quality of the story, the set design, or the acting. A film critic might judge the character development, the narrative structure, or the impact on the world of filmmaking. A marketer might focus on the product placements, the brand opportunities, or the potential for a companion video game and a line of toys. Conversely, my mom might judge the performance on how comfortable the seats were, the temperature of the air conditioning, or the cleanliness of the restrooms.

Thankfully, there are some criteria that are common amongst audience members. We can group people together who have common attributes and common ways of judging performance. In the digital marketing business, they are called segments. In user experience design, these groups are called personas (I’ll save the discussion of the difference between marketing segments and personas for another day). For each audience segment or persona, the user experience designer can identify which criteria they will design for.

Who you perform for determines what you measure.

By choosing an audience to please, the designer must deal with the accompanying criteria that audience uses to judge performance. The criteria you choose to measure determines how you will design for performance.

For a typical web site, here are just a few examples of the different types of audiences and their different measures of performance:

You might notice that these last two are more cross-disciplinary and cross-audience. This is because they’re based on values, rather than on roles. Value-centered design (see Jess McMullin’s Boxes & Arrows article) and value-sensitive design (see Batya Friedman’s work at UW) are approaches to design that define success by how well the design aligns to commonly-held definitions of worth or ethics, respectively (I’m oversimplifying).

Some of the above approaches are better than others for different contexts. A user experience designer might gravitate toward a completely user-centered approach, but their job might depend on business- or politics-centered measurements. For a company whose entire business is conducted through their web site, depending solely on user-centered measurements of success might have disastrous consequences. Especially if the web server is crashing due to some poorly tested code.

As an online user experience designer, I must deliver a design that the users love, but that is also on time, within budget, functional, revenue-generating, and adheres to the values of the company and its customers. I also need to create an experience that will evolve as those values and the market landscape change over time.

How you perform determines how you EVOLVE.

Which brings us back around to evolution. Adapting a design to the changing environment will ensure its survival. Even if you don’t intentionally vary the design, your site or product WILL change over time. HOW it changes depends on how you measure its performance. What do you value? The user? The money? The environment? Design for the audience(s) you must satisfy to survive. Measure what matters to that audience. Keep what works, toss out what doesn’t, explore new approaches, measure again, and repeat.

Next up: Part 3 – Process & Deliverables

Evolution is really very simple: life begets life, with a little variation in each iteration (due to entropy). Some variants perform better than others, giving them an advantage in the face of predators and environmental factors. Over time and zillions of variations, specialization occurs.

Evolution in a nutshell

1. Live
2. Reproduce with variation
3. What performs well, survives
4. What survives, reproduces
5. Repeat

Evolution is not random

It’s important to note that the process of evolution is not random. If it were, we’d have 3-eyed frogs with 6 legs. The reason we don’t is that, if 3-eyed, 6-legged frogs did occur, they didn’t survive any better than the 2-eyed, 4-legged frogs. That’s not random. 2-eyed, 4-legged frogs are simply better at surviving.

What is random about evolution is the mutation and the environment in which this cycle of life and death take place. In nature, evolution is not intentionally guided, as far as we know. Catastrophic events and chance encounters with predators cannot be perfectly controlled in the wild, so evolution tends to take a wandering course.

Evolution with a purpose = breeding

If you wanted to intelligently guide the evolution of an organism, you’d need to look no further than your local AKC kennel or county fair. Humans have been breeding plants and animals to select for certain characteristics for millennia. Corn, rice, wheat, bananas, apples, chickens, cows, horses, goats, sheep, dogs, cats – practically every plant and animal we eat or live with has been bred intentionally. They didn’t evolve naturally – humans pollinated, inseminated, incubated, hybridized, and culled millions of generations of wild organisms to get the highly specialized cadre of useful, nutritious, cuddly, non-toxic, and benign farmyard animals and plants we eat and love.

What’s different between evolution and breeding is that most of the random variables have been removed. Farms are very controlled environments, with relatively few random predator attacks, competitors, or catastrophic weather events. This is because we humans protect our plants and animals with weeding, fencing, shelter, medicine, and a steady supply of food. The randomness of mutation remains but is minimized by selecting breeding pairs that resemble the desired characteristics as much as possible.

These domesticated breeds wouldn’t survive in the wild. Instead, they have helped us survive and become the dominant species on the planet. It was the transition to agriculture – intentional cultivation of plants and animals – that alleviated the need for humans to wipe entire herds of mastodons to feed their burgeoning population. In short, guided evolution allowed humans and their companion organisms to survive. It also allowed and all the beauty and variety of human culture, knowledge, and art to blossom. And it might – in a return to sustainable agriculture – be what allows us to stop catastrophic climate change and survive for another million years.

Art with a purpose = design

Art is self-expression. It’s an act of creation, or, in some cases, an act of destruction or deconstruction. Whatever art is, it rarely has a purpose beyond making visible/audible/legible/tangible the vision of the individual artist. As with the natural world, the world of art evolves based on predation, competition, and the sociopolitical weather. Like evolution, art is not random, but it isn’t necessarily guided either. At least, not guided sufficiently by factors outside the artist.

Design, on the other hand, is guided art. Design is art that is meant to be used by someone other than the artist. There are varying degrees to this – and probably libraries full of these that discuss this subject ad nauseam – but my operational definition of design is art with a purpose. User experience designers are in the business of creating systems that fulfill a purpose – namely, meeting the needs of users.

Evolutionary Design

The problem with the design agency world, where I work, is that design is often confused with art. We strive to create the One True Design, a work of art so perfect that it will be viewed as useful, usable, elegant, and beautiful to all. And it will make money, further the brand, and pacify business stakeholders and partners. We are so enamored with this idea that we structure our projects to conduct research, create a design concept, test that concept, and deliver the One True Design after a few rounds of revisions. Most designs fail the first time. Sometimes miserably. And if you bet the farm on that one design, you only get that one chance.

Here’s an alternative: forget the One True Design. It is an illusion. There are no perfect designs for all time, there are only appropriate designs for specific contexts. Just as orchids and tree frogs are perform well in their niches in the tropical rain forest but not the frozen tundra, certain designs only work well in certain environments. And those environments are subject to change over time. Thus, design must change along with the often catastrophic shifts in markets, customer needs, and business strategy.

At this moment in 2009, we are all facing catastrophic economic change. If we want to create user experiences that have a better chance of surviving the chaos of the market, it may help to adopt the same guided evolutionary approach in our design process.

Evolutionary design process

1. Strategize
2. Generate design variants
3. What performs well, survives
4. What survives, generates more variants
5. Repeat

This looks very similar to the guided evolutionary process (i.e. breeding), with one key difference: randomization. Unlike with breeding dogs or corn, a guided evolutionary approach to design could be nearly free of random factors. Designers can control the mutation in design variants, creating only those variants that have a chance of surviving. No 3-eyed, 6-legged frogs. But maybe a frog with wings. Or a frog with X-ray vision. Or a frog with the ability to digest pesticides. However, in order to understand whether our flying superfrog has left us better off than with a regular frog, we must measure its performance against some goal. Otherwise, we’re just wasting our time and torturing frogs.

Next: Part 2 – Performance-Driven Design

To the panelists, evolution was a thing to fear, as if we are as powerless as dinosaurs in the face of an approaching asteroid. I found myself wondering how we might embrace evolution as a tool for advancing the discipline or, more ambitious, develop an evolution-inspired approach to advancing the process of how IA is done. How would we re-conceive of the process of IA as the guided evolution of information spaces? Might we guide the evolution of the discipline itself, utilizing evolutionary principles to improve what how IA is done and ensure the survival of IA as a discipline?

That evening, I realized that the book I’m co-writing (more on this later) on the blending of analytics, optimization, and user experience already described this evolutionary approach. Early the next morning, I signed up for a session slot to co-present the idea with fellow ZAAZ user experience architect (and optimization test designer) Rachel Elkington. Our session title: “Darwin Does Design: Measuring & Optimizing the User Experience.”

You can view the slides of our presentation on SlideShare. Audio coming soon!

Stop thinking about designing the perfect system. Perfection is expensive, illusory, and unattainable. Perfection will get you laid off. Good-enough will allow you to survive until the next round.

Design sustainable systems. Create revenue, efficiency, and value feedback loops. Design iterative workflows and self-sustaining user flows that add value, increase engagement, and make the system smarter.

Just design something that works. If it can’t be built on time and under budget, you have failed as a designer. Every subsequent iteration is a chance to improve the quality and return-on-investment of your user experience. But there must be something in place to iterate on.

Understand how your users will evolve. Look beyond the first site visit or the first use. Look at the entire lifetime relationship you form with your users. Design for their first visit, their 10th visit, and their 100th visit. And know how to measure the value of each visit.

Measure the performance of your design. Measure what will keep you your job — revenue, customer satisfaction, efficiency. Measure pre-design. Measure post-design. Fix what’s broken and measure the performance of your fixes. If you can point to measurable improvements, you’ll keep your job.