| Related sites for http://www.ed.gov/news/pressreleases/2002/04/evaluation.html |
| International_Baccalaureate_Prairie_Organization Provides course listings, membership schools and contact details for IB schools in Saskatchewan and Manitoba. | | Lapshin\'s__Succulent_Encyclopedia Photos of succulents of numerous families. | | Yale_University_Art_and_Architecture_Library Established in the late 1860s, it contains approximately 100,000 volumes on architecture, painting, graphic design, urban planning, and the history of art and architecture. | | The_CHRISTIA_Library Documents relating to practical Christian life as submitted by the CHRISTIA mailing list readers, including biographies, sermons, and issues of morality. | | UNICA__Institutional_Network_of_the_Universities_from_the_Capitals_of_Europe Association of universities that are located in the capitals of Europe. General information, statutes, grants, newsletter, forum. | | Reading_By_Ear A systematic, comprehensive, schematic multisensory approach to phonemic awareness. | | The_Petroleum_Museum 40,000 square feet of interactive exhibits cover all aspects from geology, the formation of oil, its exploration, pipelining, marketing, and refining to the economic and political impact of the indust | | GreyMatter_Inc_ Focuses on the structured settlements industry, knowledge work in small firms, collaborative technologies and global teams. Features a wiki and a blog. | | Mahymobiles Private collection in central Belgium, exhibiting a diverse range of vehicles, starting from the beginning of the century. | | The_Country_Homeschool An e-group for Messianic Christian women who homeschool and keep the home. | | Cooperative_Education_-_Wikipedia Encyclopedia article which details the history and application of cooperative education. | | University_of_Kuwait Located in Kuwait City. Offers a wide selection of undergraduate and graduate programs. | | History_of_the_Flag_of_Bangladesh Details of the flag's creation and adoption. | | Pythos A commercial system designed to facilitate productive learning conversations between individuals. It functions as a testing or surveying mechanism between instructors and students. | | Uncommon_Sense___Educators_Reinventing_the_Profession_from_the_Inside_Out There is no set of workshops, exercises, books, CD-ROMs or Web sites that will take us where we need to go, simply because the goal keeps moving. However, there are a host of options with which we can | | The_Royal_Hampshire_Regiment_Museum The history of the regiment through the collection of artifacts and exhibitions. News and events, general information, location, and shop items. Located in Winchester, Hampshire. | | Tufts_University_Campuses_Locations_/_Directions Locations and detailed directions to the three campuses comprising Tufts. | | Department_of_Art_and_Art_History Information on courses, graduate and undergraduate programs, and faculty. | | BLAZEN Performing arts troupe whose members take ideas that emerge from real life situations, then transform them into original, issue based poerty, dramas, song and dance. | | Alpha_Kappa_Alpha_Sorority Mu Delta chapter. |
|
New Directions for Program Evaluation at the U.S. Department of Education /*-- Cross-Browser Layer Access Function --*/function myDOM(id) { //alert(id) if (browser == 'ns4') {return ns4ElementById(id);} if (browser == 'ns6') {return document.getElementById(id);} if (browser == 'ie4') {return eval('document.all.'+ id);} if (browser == 'ie5') {return document.getElementById(id);} if (browser == 'ie6') {return document.getElementById(id);} return false;}/*-- Search Layers Array by ID [Netscape 4 Work-Around] --*/function ns4ElementById(layerId,me) { var foundIt = false; var whereAmI = (me) ? me : document.layers; if (whereAmI[layerId]) { return whereAmI[layerId]; } for (var i=0; i -1)&&(parseInt(navigator.appVersion)==4)&&(navigator.appVersion.indexOf('MSIE 6') -1)&&(navigator.appVersion.indexOf('MSIE 6') -1)&&(navigator.appVersion.indexOf('MSIE 6') >=0) ? 'ie6' : false; ns6 = (document.getElementById)&&(navigator.appName=="Netscape")&&(parseInt(navigator.appVersion)>4) ? 'ns6' : false; return (ns4 || ie5 || ie4 || ns6 || ie6);}var browser = browserDetect();function popWin(theURL,winName,features) { var mywin = window.open(theURL,winName,features); mywin.focus();} if(browser && browser == 'ns4'){ document.write(''); document.write('');}else{ document.write(''); document.write('');}    
Advanced Search
 Students Parents Teachers Administrators
Press Room selected = false;group = 'inf';numberOfItems = 10;alwaysTrue = 0;alwaysTrueSubItem = -1;pn = new Array;pu = new Array;expandedItems = new Array;su = new Array;sn = new Array;pn[0] = 'Press Releases';pu[0] = '/news/pressreleases/2002/04/evaluation.html?exp=0';expandedItems[0] = false;su[0] = new Array;sn[0] = new Array;su[0][0] = '/news/pressreleases/latest/?src=ln';sn[0][0] = 'Latest';su[0][1] = '/news/pressreleases/2008/08/?src=ln';sn[0][1] = '2008';su[0][2] = '/news/pressreleases/2007/12/?src=ln';sn[0][2] = '2007';su[0][3] = '/news/pressreleases/2006/12/?src=ln';sn[0][3] = '2006';su[0][4] = '/news/pressreleases/2005/12/?src=ln';sn[0][4] = '2005';su[0][5] = '/news/pressreleases/archive/?src=ln';sn[0][5] = 'Archive';pn[1] = 'Speeches';pu[1] = '/news/pressreleases/2002/04/evaluation.html?exp=1';expandedItems[1] = false;su[1] = new Array;sn[1] = new Array;su[1][0] = '/news/speeches/latest/?src=ln';sn[1][0] = 'Latest';su[1][1] = '/news/speeches/2008/2008-2.html?src=ln';sn[1][1] = '2008';su[1][2] = '/news/speeches/2007/2007-2.html?src=ln';sn[1][2] = '2007';su[1][3] = '/news/speeches/2006/2006-2.html?src=ln';sn[1][3] = '2006';su[1][4] = '/news/speeches/2005/2005-2.html?src=ln';sn[1][4] = '2005';su[1][5] = '/news/speeches/archive/index.html?src=ln';sn[1][5] = 'Archive';pn[2] = 'Fact Sheets, Op-Eds';pu[2] = '/news/pressreleases/2002/04/evaluation.html?exp=2';expandedItems[2] = false;su[2] = new Array;sn[2] = new Array;su[2][0] = '/news/opeds/factsheets/index.html?src=ln';sn[2][0] = 'Fact Sheets';su[2][1] = '/news/opeds/edit/?src=ln';sn[2][1] = 'Op-Eds & Letters to the Editor';pn[3] = 'Photos';pu[3] = '/news/pressreleases/2002/04/evaluation.html?exp=3';expandedItems[3] = false;su[3] = new Array;sn[3] = new Array;su[3][0] = '/news/photos/latest/?src=ln';sn[3][0] = 'Latest';su[3][1] = '/news/photos/2008/?src=ln';sn[3][1] = '2008';su[3][2] = '/news/photos/2007/?src=ln';sn[3][2] = '2007';su[3][3] = '/news/photos/2006/?src=ln';sn[3][3] = '2006';su[3][4] = '/news/photos/2005/?src=ln';sn[3][4] = '2005';su[3][5] = '/news/photos/2004/?src=ln';sn[3][5] = '2004';su[3][6] = '/news/photos/2003/?src=ln';sn[3][6] = '2003';su[3][7] = '/news/photos/2002/?src=ln';sn[3][7] = '2002';su[3][8] = '/news/photos/2001/index.html?src=ln';sn[3][8] = 'Archive';pn[4] = 'Audio & Video';pu[4] = '/news/pressreleases/2002/04/evaluation.html?exp=4';expandedItems[4] = false;su[4] = new Array;sn[4] = new Array;su[4][0] = '/news/av/audio/?src=ln';sn[4][0] = 'Audio';su[4][1] = '/news/av/video/?src=ln';sn[4][1] = 'Video';pn[5] = 'Events';pu[5] = '/news/pressreleases/2002/04/evaluation.html?exp=5';expandedItems[5] = false;su[5] = new Array;sn[5] = new Array;su[5][0] = '/news/events/calendars/?src=ln';sn[5][0] = 'Calendars';pn[6] = 'Senior Staff';pu[6] = '/news/pressreleases/2002/04/evaluation.html?exp=6';expandedItems[6] = false;su[6] = new Array;sn[6] = new Array;su[6][0] = '/news/staff/bios/?src=ln';sn[6][0] = 'Biographies';pn[7] = 'Newsletters';pu[7] = '/news/pressreleases/2002/04/evaluation.html?exp=7';expandedItems[7] = false;su[7] = new Array;sn[7] = new Array;su[7][0] = '/news/newsletters/index.html?src=ln';sn[7][0] = 'Current';su[7][1] = '/news/newsletters/archived.html?src=ln';sn[7][1] = 'Archive';pn[8] = 'Federal Register';pu[8] = '/news/pressreleases/2002/04/evaluation.html?exp=8';expandedItems[8] = false;su[8] = new Array;sn[8] = new Array;su[8][0] = '/news/fedregister/proprule/?src=ln';sn[8][0] = 'Proposed Regulations';su[8][1] = '/news/fedregister/finrule/?src=ln';sn[8][1] = 'Final Regulations';su[8][2] = '/news/fedregister/announce/?src=ln';sn[8][2] = 'Announcements';su[8][3] = '/news/fedregister/other/?src=ln';sn[8][3] = 'Other';pn[9] = 'New at ED.gov';pu[9] = '/news/pressreleases/2002/04/evaluation.html?exp=9';expandedItems[9] = false;su[9] = new Array;sn[9] = new Array;su[9][0] = '/news/web/latest/whatsnew.jhtml?src=ln';sn[9][0] = 'Latest Updates';function navChange( item ) { serve = false; if( myDOM( 'leftNav' ) ) serve = true; if( browser.indexOf( 'ie' ) >=0 && navigator.userAgent.indexOf( 'Mac' ) >=0 ) serve = false; if( !serve ) { document.location = pu[item]; return; } if( item != null ) expandedItems[ item ] = ! expandedItems[ item ]; text = ''; for( y = 0 ; y < numberOfItems ; y++ ) { if( alwaysTrue == y ) text += ''+pn[y]+''; else if( expandedItems[ y ] ) text += '
About ED
Budget
Press Room
Publications
Teaching Resources
Answers
Contact
Help
Online Services
Recursos en español
State Information
Web Survey
Help Improve ED.gov
PRESS RELEASES New Directions for Program Evaluation at the U.S. Department of EducationBackground What's the Problem? From Compliance to PerformanceFour Types of Program Evaluation Continuous Improvement Program Performance Data Descriptive Studies of Program Implementation Rigorous Field Trials of Specific Interventions Who's Responsible for What?BackgroundWhat's the Problem? The Department spends upwards of $100 million per year on program evaluation and program data collection. Yet key decision makers in the Department, in the Office of Management and Budget, and on Capitol Hill continue to operate without the information they need. At the same time, our evaluation studies are not as helpful as they could be to practitioners at the local level, nor can they answer questions about causation. There must be a better way to conceptualize, organize, and execute program evaluation. We propose a significant shift in program evaluation, away from a compliance model and towards a system of research and evaluation focused on results and the effectiveness of specific educational interventions. From Compliance to Performance The Department has recently developed a business case for OMB for a "Performance-Based Data Management Initiative." This initiative seeks to replace the hundreds of program compliance reports with an integrated electronic system of data collection focused on outcomes. One major goal of the initiative is to reduce data burden on our partners, but we also aim to collect higher quality data, data related to what matters most: results. These data will tell us whether the education system and its components are performing well, and they might help us understand which of our programs are having the greatest impact, but they will not tell us why. How can we supplement this data management system with program evaluations that give decision makers the information they need to allocate funds, make policy changes, and consider new directions? How can we build on the knowledge base so that practitioners know "what works" and can spend their federal dollars wisely? Four Types of Program Evaluation Generally speaking, there are four types of program evaluation, each with their own key audiences, questions and methodologies. We need to develop a balanced evaluation portfolio that embraces each of these four types of studies: Type of Evaluation Audience Key Questions Timeline Method-ologies Continuous Improvement ProgramStaff How can we continuously improve our communication and guidance in order to achieve our objectives? ASAP Market research methods like fast response surveys, focus groups, etc. Performance Data Appro-priators/OMB Which federal programs are working? Are some programs more effective than others? Annual Analysis using Performance-Based Data Management System Impleme-ntationStudies Authorizers How well are programs being implemented? Are the policy changes we made leading to improved outcomes? 5-7 Years Passive, descriptive evaluation studies, using methods like self-reported surveys and case studies Field Trials Practitioners What works? What specific educational interventions lead to increased student achievement? Long-term Random assignment field trials with longitudinal dataEvaluation Type #1: Continuous Improvement The No Child Left Behind Act gives the Department a mandate to make sweeping changes in federal policy. We are currently launching a national campaign to transform the culture of education. Especially in areas like research-based reading, we need to know immediately if our message is having an impact. We need good information, right away, to fine-tune our materials and approaches. We also need to know right away if there are any snags at the state or local levels in the implementation of this complex law. Waiting for an annual performance report or a five-year evaluation study is simply too long. Methods common in market research would be appropriate for this purpose. Fast-response surveys--even via telephone--might solicit quick feedback from target audiences (like reading teachers). Focus groups would allow program staff to probe reactions to communications pieces. Because these data would not be shared externally, but would provide a feedback loop for continuous improvement, speed is more important than rigor. Evaluation Type #2: Program Performance Data Every year, Congressional appropriators and the Office of Management and Budget want to know if individual programs are working so that budget decisions can be based on data. The Government Performance and Results Act requires program-by-program performance information for this very reason. Unfortunately, answering the question, "Is this program working?" is surprisingly hard to do. This is for four reasons: Many Department programs, like Title I, Perkins, and Adult Education, are not programs, but funding streams. In other words, they are not specific educational interventions, but rather sources of flexible funds that can be spent in a myriad of ways. New flexibility provisions encourage a blending of resources. For good reason, federal policy encourages States and localities to view federal funds holistically. New provisions encourage our partners to take funds targeted for one purpose and use them for another. This is good policy (and an acknowledgment of the appropriate federal role in education) but it makes evaluating the impact of individual federal funding streams quite tricky. Linking interventions to impacts takes time. Even for the (relatively few) Department programs that mandate a specific educational intervention, it is very difficult to provide annual impact data. Ideally, studies to track these impacts would be longitudinal and would use field trials with random assignment of treatment and control groups. These methods will not tend to produce annual or nationally representative data. Isolating the impact of Federal policy or funding (versus state or local efforts) is difficult or impossible. Rightly, federal policy increasingly seeks to coordinate federal programs and approaches with state and local efforts. A good example is Title I, which in effect mandates standards-based reform, a strategy already practiced by many states. It is impossible, then, to attribute outcomes to federal policy changes, versus state efforts. So with these caveats in mind, how can we provide good information to Congress and OMB on an annual basis? How can we identify program-by-program GPRA indicators that are linked to student outcomes? It depends on the type of program. Flexible Funding Streams For large formula-based grant programs, like Title I, we will report: National achievement trends, as appropriate for the program (ideally aligned with the Department's strategic plan indicators) Achievement trends for schools receiving program dollars The data for these indicators will come from the Performance-Based Data Management System. Already, through the comprehensive database built for us by AIR, we can track achievement trends for every school in America. As the system becomes more sophisticated, we will know which schools receive which pots of federal funds. Several funding streams will, in effect, be lumped together with specific achievement indicators. For example, Title I, Title II, Title III and Reading First of the ESEA all aim to increase student achievement (through various means). So national achievement trends will be used as indicators for all of them. But the ability to analyze achievement by school will also allow us to find out if certain pots of money appear to be more effective than others. For example, we'll know, over time, if schools receiving both Title I and Reading First funds do or do not outperform schools receiving only Title I funds. There are many methodological concerns with this approach, but it will give appropriators a rough idea of which formula grant programs are having the largest impact. Competitive Programs For the Department's smaller, competitive programs, we will also look at progress in student achievement, but only for the schools or students served by the program. For example, for the American History program we would look at history achievement indicators for schools receiving money under the program. In many cases, our smaller programs will struggle to find relevant achievement data. We might decide that no performance data is better than bad, compliance-based information. This approach to annual performance data is not perfect, but is a step in the right direction. To more fully understand the impact of federal programs, or specific educational interventions funded by them, we must use other, longer-term methods. Evaluation Type #3: Descriptive Studies of Program Implementation Authorizing committees on Capitol Hill rightly want to know how well various federal education programs and policies are being implemented, and what impact they are having. This information would feed into subsequent program reauthorizations, typically on a five-year cycle. The bulk of evaluations produced by the Planning and Evaluation Service (PES) are targeted toward this purpose. Their methodologies include a variety of "passive research" designs, usually including: Nationally representative, self-reported surveys of districts, principals, and/or teachers Case studies of representative sites Co-relational analysis PES is very good at producing these types of descriptive studies. They can provide a wealth of information to Congress about what's happened in the real world since the previous reauthorization, and what role federal program and policy changes have played. For example, descriptive studies of the Title I program have helped us to understand the challenges faced by state and local agencies when implementing the accountability provisions of the law. They also give some insight into the relationship between the law's requirements and the actions taken by Title I schools. However, these descriptive studies have failed to provide solid evidence proving a causal link between specific policies or interventions and changes in student achievement. It is impossible for these studies to make that causal link, since they do not include randomly assigned treatment and control groups. (Plus, as explained above, Title I is not in itself an intervention, making a study of its effectiveness difficult.) Descriptive implementation studies play a crucial role in understanding the impact of policy changes, but they are no substitute for rigorous field trials of specific interventions. Evaluation Type #4: Rigorous Field Trials of Specific Interventions Even with high-quality fast-response surveys, annual performance data, and descriptive studies, we still cannot answer the question on the minds of practitioners: "What works?" To be able to make causal links between interventions and outcomes, we need rigorous field trials, complete with random assignment, value-added analysis of longitudinal achievement data, and distinct interventions to study. This approach might be considered "research" rather than "evaluation." Whatever the name, the Department's evaluation agenda would be incomplete without it. It is a fair use of evaluation dollars because federal program funds are paying for the interventions to be studied. In some cases, these types of field trials will be able to answer the question, "Is the federal program working?" There are a few federal programs that are, in and of themselves, specific interventions. In other cases, these trials will instead answer the question, "Does a specific intervention (funded by federal dollars) produce results?" For example, field trials might examine the effectiveness of specific reading interventions, or whole-school designs, or professional development regimes. All of these are funded by large formula programs (like Title I) but are not the program itself. Who's Responsible for What? While coordination in program evaluation is essential, each type of evaluation will be assigned to a distinct part of the Department: Continuous Improvement Studies will be the responsibility of the Program Assistant Secretaries. For example, the assistant secretary for elementary and secondary education will be responsible for developing and implementing market research studies related to Title I and Reading First. These studies are integral pieces of the day-to-day work of the program offices. Program Performance Data will be the responsibility of the Performance-Based Data Management Team, housed within the Executive Management Team. The Program Assistant Secretaries will play a crucial role, though, in helping to define the performance measures for individual programs. Descriptive Implementation Studies will continue to be the responsibility of the Planning and Evaluation Service, reconstituted as the "Policy and Program Studies Service." Evaluation questions should be developed in cooperation with the Program Offices, but also with the needs of the authorizing committees in mind. This new office will also serve as an incubator of new policy ideas, and will continue to commission timely policy studies. Field Trials of Specific Interventions will be the responsibility of a new evaluation unit within the Office of Educational Research and Improvement. OERI will need to develop the capacity to oversee numerous high quality evaluation studies; a regular funding stream should allow it to do so.Top  Printable view  Share this page Last Modified: 09/15/2004
  Helping America's Youth
FOIA | Privacy | Security | Inspector General | Notices
Whitehouse.gov
ExpectMore.gov
USA.gov
GovBenefits.gov
Pandemic Flu
|
|
