Design Science

Last month I was an invited speaker at the 4th Annual Education and Technology Forum at the Mayo Clinic. The title of my talk was the importance of Design Science for Improving Education. The words science and design are not often used together, with design more often being associated with art, as in fashion design, interior design or graphic design. My first task was to explain why “design” was a more general activity that all humans do, it is not just about making things look good or artistic.

I gave the example of Personal Information Management (PIM) where people evolve their own designs for their information spaces at work. Once we see that design is a general human activity, we can then consider how we scientifically study the processes of design. There are two paths we can go down here. We can evolve our own approaches based on an informal design process and hope they work. Or we can systematically observe, measure and compare different approaches to show which ones are more effective and/or efficient. In other words, we can take a scientific approach to determining how to improve design (whatever the domain).

Among the many things researchers have founds studying PIM is that there are two main approaches: filing and piling. The illustration below shows a mixture of both approaches where papers important to the faculty member are piled into separate categories, but there is also an informal filing using shelves and post-it notes. This is a system that evolved through the person thinking about different possibilities, and arranging things to suit their idea of how things might work best for them. They may have used a different system in the past, made changes over time or borrowed ideas from seeing how others approached this problem.


The illustration above is in a sense a theory of how best to organize an information space. One that can be tested against other theories. Researchers studying this space can test how fast it is to find a given piece of information in the design. They can test how easy it is to construct, learn, maintain and use the specific design compared to others. Computer researchers are very interested in this specific domain to help them design better tools for facilitating organization and access to digital information.

This Google Tech Talk by Manuel Perez-Quinones is a good overview of a design science approach to personal information management:

Design processes can be informal and idiosyncratic, like the above example, or highly formalized and collectively determined like the ones that are described in textbooks of design-related disciplines (e.g. engineering, architecture). The US has been the pioneer in developing what is akin to an engineering discipline for the design of instruction. In the field of instructional design [see instructional design central], researchers study different process models, tools and methods used to develop instruction. From this they have developed formal systematic approaches to instructional design, similar to those found in other design-related disciplines. Instructors in higher education often have no training in formal approaches and develop their own idiosyncratic approaches. Many institutions have found that they can obtain better quality instruction by employing instructional designers to educate and support their faculty in the design process.

A brief introduction to instructional design can be found here:

Learning Innovation Showcase

One issue I have to deal with in my new job at Arizona State University (ASU) is learning about existing work and looking for ways to connect it. There is a famous quote from the early days of the field of knowledge management:

“If only HP knew what HP knows, we would be three times more productive.” Lew Platt, CEO, Hewlett-Packard

I was reminded of this when finding out about various innovations in teaching and learning occurring at ASU. If only ASU knew what ASU knows.  As a result, the institute I now lead organized the ASU Learning Innovation showcase. I asked all the Deans at ASU to nominate people in their units who they thought were doing innovative work. I did not define “innovative” or set restrictions, given I wanted a diversity of things on show. Out of over eighty nominations we had fifty people agree to showcase. The work was showcased over a two-hour period using a posters and demos.


Many who attended remarked upon the variety of work on show and many were able to find new collaborators or others interested in applying the innovations to their own teaching. You can see a list of the presentations and more photographs of the event here.

Given I often promote active learning; I am glad that the event created an atmosphere of active learning among the participants. If we had adopted the more usual conference format only a handful of people would have been able to show their work and the level of participant interaction would be restricted to a few questions in between presentations.

We are now planning a similar event for next year, only on a much larger scale and in the spirit of the ASU design aspiration of being social embedded, we intend to involve not educators from the local community. So look out for ASU Learning Innovation Showcase 2, January 11, 2017.

Innovation is a constant theme at ASU:

The Hype Cycle

There is no shortage of new technologies available to educators. However, from its earliest days’ educational technology has often failed to live up to the high expectations given to it. For example, in 1922 Thomas Edison proclaimed, “I believe the motion picture is destined to revolutionize our educational system and that in a few years it will supplant largely, if not entirely, the use of textbooks.” The Gartner group identified the “Hype Cycle” for emerging technologies (see figure 1), which is of relevance here. The hype cycle suggests that new technologies tend to get overhyped based on their assumed potential. This leads to disillusionment when a lack of success at scale relevant to the hype becomes apparent.
Hype Cylce
Figure 1: the hype cycle

After the disillusionment, a second wave of the technology often arises accompanied by new methodologies and practices that quietly begin to make a real impact over time. Although motion pictures did not supplant textbooks or radically transform education as predicted, they have over time become an important tool for educators.

The skills involved in this second wave research and development are probably undervalued, but no less important in making a technology successful. The problem here involves not just making a technology work in a controlled environment. It involves making it work with clear evidence of effectiveness in complex organizational structures that are often resistant to change.

We have over the past decade seen an increasing number of technologies move up the hype cycle (artificial agents, adaptive tutors, educational games, MOOCs, interactive classrooms, virtual worlds, learning analytics, social media, learning management systems). Often these are seen as isolated “silver bullet” solutions that on face value have potential to transform education. They are not always viewed in the context of the systems of education we currently have, or the ones we want to create. The problem is not a lack of new ideas or technologies; it is lack of methodologies and cultures for integrating them into new designs of educational systems. These are the important problems being solved by people who are working on the second wave without any hype.

Letting go of the horse

horse drawn carriage early automobile that was referred to as a horseless carriage

For many years after the invention of the automobile the predominant design thinking was influenced by the preceding age of horse-drawn transportation. In fact, early automobiles were referred to as horseless carriages. It took a number of decades for auto design to break out of this confine and evolve into the many automobile variants we see today -none of which look like a carriage. We have moved from the age of the horseless carriage toward the age of the driverless car.

Google car

The world-wide web is a break-out technology for education in the same way the automobile was for transportation. However, the predominant design thinking in education is still confined by the pre-internet age. In many ways we are in a horseless carriage phase of addressing technology in education although there are a number of areas, such as active learning space design and assessment, where this is beginning to change.

One institution that has made a large-scale commitment to breaking out of the traditional thinking in educational design is Arizona State University, which bills itself as the “New American University”. It tries not to be constrained by such academic traditions as high entrance requirements, 15-week semesters or fixed academic disciplines. As a result, some interesting ideas have developed and the university has been able to thrive and grow even in the face of substantial cuts in state support. In this video, ASU president Michael Crow outlines the concept and the design thinking behind it:

This is a good example of the educational equivalent of letting go of the horse. It involves truly embracing the new world created by the Internet and rethinking what an educational institution is and how it should operate. I have noted in a previous Blog what history tells us about organizations that cannot break out of previous paradigms.

What does an A mean?

In US education student achievement is measured primarily by assigning letter grades at the end of a course and averaging all the grades to come up with an overall measure of how “good” a student is relative to their peers. There are several flaws in this system.

• Although failure is part of the learning process, in this system every failure can have a cost to the student.

• Because there are no standards, the determination of what a letter grade means can vary greatly between different instructors.

• Grade inflation has been introduced into this system to the extent that what used to be considered average (a C grade) is often seen as equivalent to failing.

At the same time as grades have been increasing in both high school and college [1,2], college entrants are being required to do more remedial course work [3] and employers have been complaining about the abilities of graduates [4]. If grades are going up and quality is going down, there is something wrong with the system.

I remember being made acutely aware of this situation many years ago when a new graduate student applied to work in my lab. His GPA was 4.0 and because I was looking to employ someone to do object-oriented programming, I was glad to see he had taken two courses in this area. During the interview I asked him to explain object orientated programming in his own words. He was unable to do so even after some prompting. I then showed him some code containing a number of basic errors and he was unable to see anything wrong with it. His grades had been largely based on multiple-choice tests. He had passed his courses with high grades but had failed to acquire even the basic skills needed by an employer.

Just as some advocate completely changing the way we educate in the Internet age, there are many who would advocate changing the way educational attainment is measured. The GPA is no longer useful.

A new approach that is gaining prominence is competency-based education [5]. Basically, this is breaking down assessment into specific skills or competencies. So rather than a student’s receiving an arbitrary grade for an entire course, they receive a certification for each learning outcome that has been defined. In the example above, a student would have to be certified in the component skills of program error detection and key concept explanation before graduating with a qualification in object oriented programming.

Competency-based education requires more instructional design effort to define the competencies and create reliable measures of student performance. Measures may require gradations, e.g. an acceptable level of performance versus an exceptional or expert level. Course completion and graduation are based on a demonstration of competence instead of a prescribed number of credit hours and knowledge recognition memory through multiple-choice tests. In this system, an instructor’s time is likely to be spent more on coaching and assessing competency performance and less on presenting information. The Internet can be employed for online skills demonstrations and instruction via video. Competency-based education is a perfect complement to active learning and flipping the classroom discussed in prior blogs (Sept, 2012, Dec, 2014).


[1] Rising GPAs making it harder to get into college. Justin Pope.

[2] GPAs have steadily risen over decades. Jessica Lichter.

[3] High School Graduates Still Struggle With College Readiness. Allie Bidwell.

[4] Are they really ready to work? Joint report of the Conference Board, Corporate Voices for Working Families, the Partnership for 21st Century Skills, and the Society for Human Resource Management.

[5] What is competency-based education? Dr. Robert Mendenhall, President, Western Governors University.


Competency-Based Education as a Potential Strategy to Increase Learning and Lower Costs. Stephen R. Porter and Kevin Reilly

Doomed to Repeat

“Those who don’t know history are doomed to repeat it.”
― Edmund Burke

This well-known quote is certainly true when it comes to the history of educational technology. I recently participated in a NERCOMP event that considered the use of tablet computers in education. The opening presentation touched on some failures, specifically a spectacular failure of a $1B iPad program for Los Angeles schools (see this LA Times article). This is not the only example.

The iPad is one in a long series of technologies to be promoted as something that would revolutionize education. For example, Thomas Edison once said in 1922, “I believe the motion picture is destined to revolutionize our educational system and that in a few years it will supplant largely, if not entirely, the use of textbooks”. Since then, many technologies have been hyped and adopted without a revolution occurring (see October, 2012 post).

The problem with this and similar initiatives is that it concentrates on getting the new technology into the learning environment without first creating a strategic plan for how the technology will be used and how the users (particularly the teachers) will be supported. The assumption is made that all that is needed is the new technology and everything will work by some sort of magic. Prior to the recent focus on tablets, many schools and colleges adopted policies requiring students to obtain a laptop but had no strategies for how they were to be used in classes. As a result some instructors have banned laptops from classes, seeing them as more of a distraction than a help (e.g. Laptops vs. Learning – Washington Post)

Fortunately, the NERCOMP event did include some success stories. A professor from Baypath University described how he was initially a skeptical and unwilling participant with little technical experience but now is an enthusiastic supporter of the use of tablets in the classroom. The key ingredient of the success story was the training provided alongside the adoption of tablets. The extensive training, provided by edtechteacher, focused on what students can do with the tablets in class instead of providing details about the technology or all the apps available. Seven guiding principles aimed at promoting active learning with the technology were adopted:

1. Exploration, not planning
2. Activities, not Apps
3. Flexible tools (e.g. video creation), not discipline specific apps
4. Creation not consumption (by students)
5. Student uses, not Faculty uses
6. Task challenges (to students), not step-by-step instructions
7. Student choice of tools, not instructor-assigned apps

The initially skeptical professor said the experience has transformed his perspective on teaching and sees the changes as a dramatic improvement. He is often amazed by what his students do with the technology.

diagram of SAMR

Key to this approach is focusing on the higher levels of the Substitution Augmentation Modification Redefinition Model, which defines levels of technology use in teaching and learning. The most basic and most used level is substitution, where technology is used to perform the same task as was done before the use of computers. Thus, the instructor uses PowerPoint to replace acetate slides and chalkboards and the students use laptops to replace paper notebooks. The highest level is redefinition, where technology results in new learning tasks that were previously inconceivable.

The approach to learning, not the type of technology, is the key. Training should be focused on facilitating learning outcomes not merely technology features and operation. This is the way to change history not repeat it.

Further Reading:

The Human Touch by Lowell Monke, Education Next, Fall 2004.

Research on in class use of laptops and other devices. The Teaching Center Journal, Washington University in St.Louis, Sept, 2014.