Course

Instructor

Mary Annette Rose                          Email: arose@bsu.edu

Assistant Professor

Department of Technology

Ball State University                          Office Hours: By Appointment

Applied Technology 207C                  Office Phone: (765) 285-5648

Muncie, IN 47306                              Facsimile: (765) 285-2162

This course provides an overview of the models and analytical methods that are employed to make judgments about the worth and merit of curriculum and curricular programs. Students will plan, conduct, and report an evaluation study of a curricular program which builds technological literacy or technical skills. The course improves students' ability to make evidence-based curricular decisions at the program level.

 _____________________________________________

 _____________________________________________

What is the nature of evaluation?

What issues, principles, processes, and methods underpin the evaluation of curricular programs which build students' technological literacy and technical skills? 

How can empirical research methods inform curriculum and program evaluation?

How might evaluation inform program and curriculum improvement?

What processes, criteria, and standards are commonly used to evaluate technology education, career and technical education, and STEM curriculum? How might evaluation improve teaching and learning about the technological world?

 _____________________________________________

1. Appreciation: Develop an appreciation for curriculum evaluation as it applies to informing educational decisions in U.S. schools especially as it relates to curricular programs which enhance technological literacy and technical skills.
2. Understandings: Develop rich understandings of curriculum evaluation concepts, processes, and products by:
   1. Differentiating different purposes and perspectives of evaluation;
   2. Explaining the roles and responsibilities of curriculum stakeholders;
   3. Discussing the political nature of curriculum evaluation;
   4. Analyzing the elements, organization, scope, and sequence of curriculum frameworks and documents; and
   5. Analyzing standards addressing content, professional development, student assessment, and curricular programs.
3. Skills: Develop a usable skill set as it applies to conducting an evaluation by
   1. Selecting, developing, and using data-gathering instruments and protocols,
   2. Analyzing and interpreting quantitative and qualitative data; and
   3. Applying systematic decision-making procedures.
4. Compare and Evaluate Curriculum: Critically compare and evaluate curricular programs, products (standards, guides, assessments) and processes from different levels (state, school, program, course) and theoretical perspectives.
5. Ethics and Appropriate Use: Develop understandings and commitment to act on principles of social responsibility and ethical practice, including areas of integrity, honesty, equity, confidentiality, anonymity, privacy, copyright, and informed consent.
6. Professionalism: Develop an image of oneself as a curriculum evaluator.

 _____________________________________________

I. Overview of Evaluation

A. Definitions

B. Differentiating Measurement, Assessment, & Evaluation

C. Purpose and Goals

1. Planning or Diagnostic: Need

2. Formative: Continuous Improvement

3. Summative: Accreditation

D. Principles and Logic of Evaluation

E. Foundational Resources

II. Approach and Models

A.  Philosophical considerations, purpose, and goals

B.  National and state policies

1. ESEA and NCLB

2. Perkins

C. Evaluation Models

1. Goal Achievement

2. Side-effects or Goal-Free

3. Client-oriented

4. Social Efficiency (Cost-benefit)

5. Others

III.  Evaluation Process

 

A. Plan the evaluation

1. Identify the criteria of merit

2. Establish standards for these criteria

3. Select or develop data-gathering instruments and protocols

4. Identify data-analysis methods

5. Seek approval to conduct the evaluation

B. Implement the evaluation

1. Administer instruments and protocols to collect data

2. Analyze data

3. Apply criteria of merit to judge merit and worth

4. Prepare recommendations

D. Report the evaluation

1. Prepare written report

2. Present report to stakeholders

 

IV.  Design and Methods

A. Evaluation Designs

1. Case Study

2. Synthesis

3. Comparative

B.  Levels of Evaluation (Kirkpatrick)

1. Reactions

2. Learning

3. Application

2. Impact

C. Data Gathering

1. Student Assessment

Performance Assessment

Tests and Measures

2. Survey

3. Interview & Focus Group

4. Existing Documents and Databases

5. Experiments and Quasi-Experiments

7. Observation

D. Data Analysis

1. Quantitative Methods

a. Descriptive

b. Inferential

2. Qualitative Methods

a. Content Analysis

b. Mapping and Alignment

c. Thematic Review

E. Making Judgments

A.  Standards of Responsible Practice

1. Systematic Inquiry

2. Confidentiality and Anonymity

3. Equity and Diversity of Interests

4. Integrity and Honesty

B.  Coherence

C.  Transparency

D.  Utility

 _____________________________________________

Activities and Projects

Highlights of the activities and projects are described below:

1. Activities and Quizzes (30%): Provide evidence of thoughtfully and thoroughly conducting short-range (less than two weeks) learning activities, including:
   1. Comparing program evaluation plans;
   2. A critique of an evaluation report;
   3. An interview with state or local school administrators,
   4. Developing data-gathering instruments and observation or interview protocols.
2. Discussion (20%): Class participation will vary in form and quantity to complement weekly learning goals and activities. When the major learning activity includes ONLY class discussion, your contributions to the asynchronous discussions will be evaluated in terms of the quantity and quality of your messages. The criteria and procedure outlined in the Assessment Rubric for Asynchronous Discussions will be used to determine discussion grades.
3. Major Project (50%): Plan, conduct, and report an original evaluation of a curricular program or needs assessment. Planning will result in the creation of an evaluation proposal which includes: goals and essential questions, description of the educational setting, instruments and protocols for gathering data, evaluation criteria, and analytical methods. After receiving permission to conduct the evaluation, the student will implement the evaluation plan by administering instruments and other data gathering protocols, analyzing this data, and making judgments concerning the worth and merit of the program. Finally, students will compose an evaluation report and present it to interested stakeholders. The criteria and procedure outlined in the Assessment Rubric for Evaluation Report will be used to assess the final report.

Assessment

Assignment Quality: Although the expected quality of student products and contributions is contingent upon the goals and characteristics of the activity, general quality guidelines for all deliverables include:

1. completeness and coherence of work;
2. logically applying evaluation concepts and processes within the assignment;
3. attributing ownership of ideas, maintaining privacy, and conforming to other ethical practices established within the Publication Manual of the American Psychological Association (2001) and the American Evaluation Association's (AEA) Guiding Principles for Evaluators; and
4. applying standard conventions of English composition. Remember to use a spell checker and a grammar checker before submitting documents.

Assignment Schedule: All activities should be submitted as scheduled. A 5% grade reduction may occur for each day the assignment is late. Extensions will be administered on a case-by-case basis at the instructor's discretion.

Grading: Grades will be assigned for each activity and posted in the Blackboard Gradebook. Final course grades will be derived by (1) dividing the points accumulated by the points possible to arrive at a decimal score for each category (activities, participation, and project), and then (2) applying the weights for each category to arrive at a course score. Final percentage scores will be given letter grades according to the following scale:

A = 100 - 94
A- = 93.9 - 90
B+ = 89.9 - 87
B = 86.9 - 84
B- = 83.9 - 80
C+ = 79.9 - 77
C = 76.9 - 74
C- = 73.9 - 70

As noted in the Graduate Catalog (2005) "no course with grades below C (2.0) may be counted toward any degree program" (p. 19).

Participation & Discussion. Unlike a traditional class where participation may be judged by attendance, in a distributed environment participation is documented by communiqués and information exchange (e.g., email messages, conference postings, telephone calls, and document exchanges) among the class participants. The substance, function, and nature of these communiqués (e.g., relevance, clarity, and insightfulness) become indicators of a learner's conceptual and attitudinal growth.

Class discussions can be productive learning activities if you:

• conscientiously share relevant resources and experiences;
• make connections between first-hand observations of educational problems and curriculum issues;
• critically examine your own assumptions, as well as the assumptions of others;
• offer and explain your propositions, ideas, and insights;
• justify your assertions with evidence;
• interweave the ideas of others into your own posts; and
• summarize readings and discussions.

 _____________________________________________

Guidelines for Email correspondence:

• include only one subject per message;

• use a descriptive subject heading;

• when conducting group business, carbon copy (cc) all group members;

• follow acceptable e-mail etiquette; and

• be concise, write short sentences, and use bulleted lists.

 _____________________________________________

The following resources will be employed during the Fall 2008 offering of Curriculum Evaluation. You may purchase these resources through MBS Direct or another vendor, such as Amazon or the International Technology Education Association. In some instances, digital copies are free and readily available. 

Kaufman, R., Guerra, I., & Platt, W.A. (2006). Practical evaluation for educators: Finding what works and what doesn't. Thousand Oaks, CA: Corwin Press. ISBN: 9780761931980

Standards: Acquire the content and program standards from your professional field, school district or state, such as: 

International Technology Education Association. (2000). Standards for technological literacy: Content for the study of technology. Reston, VA: Author. Available electronically from http://www.iteaconnect.org/TAA/PDFs/xstnd.pdf

International Technology Education Association. (2003). Advancing excellence in technological literacy: Student assessment, professional development, and program standards. Reston, VA: Author. Available electronically from http://www.iteaconnect.org/TAA/PDFs/AETL.pdf

 Manuscript Guide:

American Psychological Association. (2001). Publication manual of the American Psychological Association (5th Ed.). Washington, DC: American Psychological Association.  ISBN:1-557987-91-2

 Required Computer Applications

The following computer applications are required tools for ITEDU 600. Please purchase/download and install the following:

• Microsoft Office, including Word, PowerPoint, and Excel. REQUEST: Kindly save Word files created with Office 2007 to an earlier version (i.e., Word ".doc" rather than Word ".docx") prior to sharing or submitting assignments. This simple gesture will eliminate frustrations for others.
• Latest version of Adobe Acrobat Reader
• Email
• Web Browser (Microsoft Internet Explorer is preferred)

Additional Resources: Course resources, including relevant journals, organizations, books, research studies, and WWW links, are located on the course Resources page. You may find the following resources to be especially beneficial, including:

Hoepfl, M. & Lindstrom, M.R. (Eds.). (2007). Assessment of technology education. 56th Yearbook. Council on Technology Teacher Education. NY: Glencoe. ISBN 0-07-877770-4

International Technology Education Association. (2005). Realizing excellence: Structuring technology programs. Reston, VA: Author. ISBN 1-887101-05-5

International Technology Education Association. (2004). Measuring progress: Assessing students for technological literacy. Reston, VA: Author. ISBN 1-887101-04-7

Marsh, C. J., & Willis, G. (2007). Curriculum: Alternative approaches, ongoing issues (4th ed.). Upper Saddle River, NJ: Merrill Prentice Hall. ISBN 0-13-171510-0

Newby, A.C. (1992). Training evaluation handbook. San Diego, CA: Pfeiffer & Company.

Professional Development Modules provided by the  Online Evaluation Resource Library (OERL). Available at http://oerl.sri.com/module/index.html

Oliva, P. (2004). Developing the curriculum (6^th ed.). Allyn & Bacon, Inc.

Posavac, E.J. & Carey, R.G. (2007). Program evaluation: Methods and case studies (7th ed.). Upper Saddle River, NJ: Prentice Hall.

Sanders, M. (Ed.). (1992). [Special Theme Issue]. Journal of Technology Education, 3(2). Retrieved December 18, 2005, from http://scholar.lib.vt.edu/ejournals/JTE/v3n2/pdf/