Evaluation and Selection of Web Course Management Tools
[Research completed at: Robert H. Smith School of Business, University of Maryland, College Park ]
Abstract
World Wide Web has provided higher education institutions an effective medium to distribute course materials over the Internet. Unfortunately there is no single standard that describes how courses should be created and distributed over the Web. There has been a proliferation of Web course development tools that can be used to create course materials and have these accessed by students from remote sites. This document describes applications of Web Course Development Tools, their evaluation and selection for large campus wide deployment. The purpose of this document is not to provide vendor specific evaluations, but to describe the methodology and selection process for course development tools that provided best fit based on existing infrastructure, faculty experience, needs, and additional criteria which emerged during the evaluation process.
Introduction
Before Internet access became widely available, Distance Education used phone lines, cable TV, videotapes, or postal mail for instructors to deliver asynchronous instruction to distance learning students enrolled in virtual classes. During the past three years Distance Education using Internet has become widely accepted in higher education institutions. The World Wide Web has been recognized as a powerful medium for distributing course based information. Hundreds of courses in almost all areas can be accessed from sites such as the World Lecture Hall.
Virtual universities such as Western Governor's University, Phoenix University Online , Walden University, University of Maryland University College use courses to attract students who may not be able to attend traditional colleges and universities due to various reasons. In majority of cases students enroll in distance learning institutions because of convenience while working towards a diploma, or enhancing professional skills that may result in a promotion or changing careers. Other than 'virtual universities' mentioned above traditional universities are also offering distance education courses either as standalone courses delivered entirely on the Internet without any class meetings, or by using Web based materials for supplementary instruction.
Several different methods exist for developing Web courses. Although the basic framework of Web documents is built on HTML tags, today it is not necessary to know HTML programming to develop course material for the Web. This is due to development of software that convert documents to HTML format, or provide a WYSIWYG front-end that is similar to creating a document using any common word processor. There are even organizations such as UOL and Ecollege.com (formerly Real Education) who are in the business of education to develop and host courses for colleges and universities. Depending on background, experience, interest, and technical sophistication, faculty members also individually choose HTML editors, templates, authoring programs or commercial Web course development tools to develop and distribute online material.
This document describes applications of Course Development and Management Web Tools, their evaluation and selection for large campuswide deployment. The purpose of this document is not to provide vendor specific evaluations (which is available elsewhere), but to describe the methodology and selection process for course development tools that provided 'best fit' depending on existing infrastructure, faculty experience, needs, and additional requirements that emerged during the evaluation process.
The University of Maryland at College Park is a comprehensive Carnegie Research-I University with 25,000 undergraduate students, 8000 graduate students, and 3000 faculty. According to the National Research Council ranking, the university has top-twenty ranked programs in engineering, computer science, physics, journalism, art history, economics, criminology, mathematics, oceanography, astronomy, education, and agricultural economics. In its 13 schools, 98 Undergraduate Majors, 87 Graduate degrees are offered. During the academic year there are on average 3000 undergraduate and 1000 graduate courses offered by the university. With such a large student enrollment and the possibility of every course as potential users of Web course development tools, it was important to select a standard tool that faculty could learn and use to develop pedagogically useful course materials. This would offer students a consistent, virtual, meaningful, interactive environment to promote learning.
Online Teaching and Learning
Traditional classrooms have used objectivist model of learning which is based on Skinner's theory of transfer of knowledge from teacher to the learner. In this model, the instructor controls the material and pace of learning. Describing this model, Cuban (1993) mentions that instruction is directed to the whole class as a large group, the pace of learning, is controlled by the teacher, and curricular and instructional decision making is guided by the textbook as the primary medium.
On the other hand, constructivist model is student centered (Hofstetter, 1998). Here the instructor acts as a moderator primarily responsible for facilitating learning. In this model, most instruction occurs in small groups, students help choose the content to be organized and learned, teachers permit students to determine the rules of behavior, classroom rewards, and punishment. (Cuban, 1993). Because of the capability of Internet to use discussion groups and text, graphics, audio, video, file transfers over electronic mail, in asynchronous format and also videoconferencing, whiteboards, chat in synchronous real time modes, a different learning medium has evolved that is closely based on constructivist approach to learning. Since Web Based Instruction is such a new medium, evidence of effectiveness of online courses compared to traditional instruction is lacking. Although there have studies e.g. Schutte (1996), showing virtual classes perform better, other researchers such as Neal (1998) have disputed these by questioning research design and methodology used to arrive at these results.
Because the Web is a different medium, teaching styles have to be adapted to this new environment. Faculty have to adjust to the new pedagogy that uses technology as an integral component in teaching. Many faculty who have not used technology in the past to accomplish course objectives, now have to be trained to teach with technology and very often include a component in the course that provides information to students about the technology itself.. Beaudoin (1998) feels that as we rapidly approach the next millennium, the role of professoriate within the context of an evolving teaching/learning environment is undergoing profound changes. Not all faculty have embraced the idea of moving courses for online delivery. Faculty resistance has resulted from perceived notion of being intimidated about rapidly changing newer technologies, constantly keeping up with a need to update skills, losing control over the teaching process, dealing with copyright issues by accessing or making available materials for open access.
Faculty pioneers have offered online courses that simulate traditional classroom environment by using syllabus, schedule, course notes, assignments, discussion rooms. In addition students are also provided the opportunity to communicate with the instructor or other students by using e-mail, bulletin boards, live chat rooms. Capabilities such as online assessment, simulations, multimedia, course delivery, access to external resources, provide potential advantages over lecture-only classes. Effectiveness of these experiential type learning has been supported by researchers. Vygotsky (1986) emphasizes the use of social dialog and interaction to be an essential part of the learning process. Web based testing represents a cognitive behavior modification technique designed to help students develop goal setting behavior, planning, and self-monitoring (Good & Brophy, 1995) and provides opportunity for students to master the concepts (Bloom, 1981).
Instructional design principles must be applied to develop pedagogically effective learning materials. Ritchie and Hoffman (1997) emphasize that well designed courses include elements that motivate the learner, specify what is to be learned, prompt the learner to recall and apply previous knowledge, provide new information, offer guidance and feedback, test comprehension, and supply enrichment or remediation. Web Based Instruction must be designed to accommodate individual learning styles. This does not mean using all available technologies but instead using those appropriate technology mechanisms that will directly contribute to enhance learning.
EDUCOM has identified three main obstacles for providing effective online materials and learning environment. these are:
1. Lack of support for the collaborative and dynamic nature of learning.
2. Lack of standards for locating and operating interactive platform-independent materials.
3. Lack of incentives and structure for developing and sharing content.
Web course development tools attempt to address the first two items mentioned above by providing a collaborative environment and standard interface for developing and distributing course content.
Web Course Development Tools
Developing courses using integrated features of Web Course Development Tools offers a single authentication scheme, directory structure, consistent interface, and simple way to publish and update content. HTML layout editors that were initially used to create course content produced static material which were rigid and non-conforming to different learning environments. The new generation of Web course development tools provide features that let instructors adapt components according to learning outcomes of the course. Use of such tools can promote collaborative learning, enhance critical thinking skills and give every student an equal opportunity to participate in classroom discussions.
Course material is developed by using a familiar graphical Web interface. Web course development tools can be considered as a workbench for putting together text, graphics, video, and audio files. This can be done without any knowledge of HTML. Many tools also have 'wizards' that guide the developer in creating course elements. In addition it also offers additional features such as bulletin board, chat, e-mail, discussion groups, calendars, and online assessment. Administration features allow creating (or import) of student accounts, archiving e-mail messages student discussion groups and graded assessment. In general there is a three way interaction: 1) student with content, 2) student with instructor, 3) student with other student(s).
Use of Web course development tools can piggy-back on huge investments higher education institutions have made in not only installing the hardware and software but also planning the network infrastructure to link offices, libraries, classroom, and student dormitories for local, wide area, and Internet connectivity. With sound pedagogical design, web based instruction can create meaningful learning environments by engaging students in active application of knowledge, concepts and giving them an opportunity to control pace and monitor learning which will help them grow and evolve as the course progresses.
With such a promising future and acceptance by college and universities, many Web Course Tool vendors are also forming partnerships with book publishers and companies that host web courses. This could possibly result in courses becoming commodities, developed by faculty as content experts and packaged in multimedia format by instructional designers and programmers, and sold to other institution as pre-packaged components. Eventually electronic publishing could replace textbooks because of the advantage of having material contained in the course tool shell dynamic in nature so it can be updated easily and made available instantly.
Products
With many course development products currently available and new ones being developed (both commercially and as university projects), there does not seem to be any standardization and interoperability between course development systems (as noted earlier) which would make it possible to exchange materials between courses. Although the lowest common denominator is HTML since all courses are web based, there has been an initiative to develop a set of specifications to facilitate the growth and viability of distributed learning on the Internet. EDUCOM's Instructional Management Systems (IMS) Project involves academic, commercial, and government organizations with a goal of developing technical and design specifications that helps in development of common framework for generating and leveraging information integral to the process of learning. Draft of IMS specification that provides more details on member organizations, project goals, and defines technical content requirement, design, architecture, services has been made available by EDUCOM.
For long term benefits, it was important that the selected tool conform to IMS specifications since this would provide a standardized environment endorsed by the premier organization in higher education.
Methodology
Very often technology based decisions are made by technical personnel who base their decision on personal use, attendance at vendor sponsored workshops, reading about it in trade publications, or having used other products from the same vendor. This type of decision making process does not take into account the needs and capabilities of the customers, users, or client. Since implementation of a standard platform would directly affect how instructors teach and students learn, it was important to get primary developers i.e. faculty involved in the decision making process.
In its first meeting, the committee identified possible tools from review of literature, previous conference presentations, web resources, discussion groups, listserv, and published vendor materials.
The following products emerged as possibilities:
Vendors were contacted for demo versions of products and the committee met regularly after creating test courses to discuss advantages, disadvantages, features, and problems encountered during course creation. Over the course of many such meetings, three products emerged worthy of further consideration. For the focus group meetings, based on ease-of-use, and extended feature set, one of these products was chosen for demonstration purposes.
Focus Groups
Faculty members in several colleges were contacted and invited to two focus group sessions. Approximately 50 faculty accepted the invitation to attend a two hour focus group meeting. The sessions were held in the University of Maryland Teaching Theaters which have networked computers with computer mediated communication tools installed. These tools can be used to facilitate collaborative work such as electronic brainstorming, voting, and anonymous threaded discussions. Moran (1993) and Krueger (1994) provide additional insights into focus group composition, objectives, participant characteristics, topic orders, and scenario development.
The session started with self running slide presentation on each participant computer that presented information on use of software for course delivery (standalone editors versus integrated packages), existing features, and advantages to students. These pre-organizers were intended to be pointers to items and topics that would be addressed during the focus group sessions.
At the end participants were asked to complete an electronic One Minute Paper that would anonymously record their comments, questions, and suggestions to be forwarded to the committee.
After the self running demo, faculty were introduced to task force committee members, purpose of the meeting was explained with emphasis that specific outcome in each session would be to determine interest for faculty use and applications of Web Course Development tools and also identify characteristics that the selected tools should have. It was made clear that although there would be a sample tool used for demonstration, the purpose was not to evaluate this particular tool based on its features.
One of the task force members then provided Student View of a sample course. Procedures such as login, navigation, course schedule, class notes, online assessment were demonstrated. Focus group was once again reminded that product being used represented generic class of Web Course Development tools and individual features may differ in the final selection.
Faculty members were then given the opportunity to work on the course as students. During this time, the task force members observed reactions, offered help, and answered questions posed by individual faculty. By this time, it was clear that many faculty appeared to be interested in finding out more about the development side.
Another committee member then created a course from scratch to replicate the student-view that faculty had just seen. Because of time constraints, not all features were shown. At the end of this demonstration, the focus group now had a much clearer idea of development, presentation of online course material using a web course management tool.
The next hour was dedicated to getting as much input and feedback from faculty based on their brief experience with online course development tools. All comments were noted. there were interesting issues brought up such as: Security, stability, reliability, dial-in time limitations, honor system for assessment, course setup, training, cost, creating online groups, multimedia capability, importing existing courses, exporting completed courses, access and licenses for extension services and executive programs, guest scholars, archiving, course ownership, copyrights etc.
Evaluation Criteria
Based on Committee discussions, review of literature, on-line resources, and faculty focus group meetings, the following general criteria emerged. Products were evaluated based on these criteria and while it was understood that no single product could possibly meet all criteria (especially integration with University of Maryland in-house developed systems) , it was equally possible that any product meeting all criteria may not be the best due to interface, technical, functional, or cost reasons.
| Features | Back-end | Other |
|
|
|
A Ratings sheet was also developed that listed Rank order of importance as perceived by faculty for the categories mentioned above and a check list of features available in the top two products. To provide a standard for evaluation of tools by each committee member, the ratings sheet was by individual committee members only. Final decision was made on discussions rather than a 'number' since no attempt had been made to validate the ratings sheet as a standardized instrument for measuring Web course tools features and/or effectiveness. Other online resources listed below also provide information on quantifying the selection process.
Results
Based on committee discussions, online evaluation, vendor demonstration, review of literature, individual contacts, testimonials from existing users and institutions, scalability, integration with current infrastructure, and ratings comparison, the Web Tools Committee recommended adopting WebCTcourse development tool. WebCT is a full featured product currently shipping and usable under the Unix operating system (dedicated course server of choice for University of Maryland) with fairly easy to use interface, good technical support, and strong pedagogical components. It was however obvious from the two faculty focus group meetings and numerous committee discussions that selection of one single tool for the entire campus may not meet needs of individual faculty members in each and every department. (A follow-up report on implementation and acceptance of the selected tool is planned for the future)
Technology Training has always been recognized as an important component in the University's Technology Strategic Plan. Some examples of training are: Peer Training program for students, and Institute for Instructional Technology, aimed at preparing faculty to incorporate technology into their teaching. Teaching With Technology symposium, which is sponsored with the Center for Teaching Excellence, brings faculty from throughout the University together to share their experiences with using technology in teaching. Faculty training sessions for Web Course Development tools are being planned during Summer and Fall 1998. These training sessions will be developed by Academic Information Technology Services at University of Maryland, College Park, and made available to any faculty member interested in using the selected web tool for developing and implementing standalone or supplementary online course material.
Committee Members
| Ellen Yu Borkowski (Chairperson) |
| Dan Cabirac |
| Sunil Hazari, Ed.D |
| Mike Landavere |
| Rosmaria Morales |
| Diane Patrick, Ph.D |
| Jamie Spriggs |
References
Beaudoin, M. F. (1998). A new professoriate for the new millenium. Distance Learning Online Symposium, 8(5).
Bloom, B. S. (1981). All our children learning. New York: McGraw-Hill.
Cuban, L. (1993). How teachers taught (2nd Ed.). New York: Teachers College Press.
Good, T. L. & Brophy, J. (1995). Contemporary educational psychology (5th ed.). NY: Longman.
Hofstetter, F. (1998). Cognitive versus behavioral psychology. http://www.udel.edu/fth/pbs/webmodel.htm
Krueger, R. A. (1994). Focus groups: A practical guide for applied research. CA: Sage Publications.
Moran, D. L. (ed.) (1993). Successful focus groups: Advancing the state of the art. CA: Sage Publications.
Neal, E. (1998, June). Does Using Technology in Instruction Enhance Learning? or, The Artless State of Comparative
Research. http://horizon.unc.edu/TS/commentary/1998-06.asp
Ritchie, D., & Hoffman, B (1997). Incorporating instructional design principles with the world wide web. In B. Khan (Ed.), Web based instruction(pp. 135-138). NJ: Educational Technology Publications.
Schutte, J. G. (1996). Virtual teaching in higher education: The new intellectual superhighway or just another traffic jam. http://www.csun.edu/sociology/virexp.htm
Vygotsky, L. (1986). Thought and language. Cambridge: MIT Press.
Online Resources
- Sheridan College WebCT Info and Resources.
- University of Georgia selection process (listserv message)
- University of Alberta - "Fit Analysis" (listserv message)
- Committee on conference Tool comparison (Univ. of Pittsburgh)
To cite this article use:
| Hazari, S. I. (1998). Evaluation and selection of web course management tools. Retrieved from http://www.sunilhazari.com/education |
Related article:
Evaluation of Blackboard Course Tool at the Robert H. Smith School of Business, University of Maryland, College Park
