Organizational Process Programs

Cognitive Studies of Interdisciplinary Collaboration

Interdisciplinary teams are increasingly common in industry, government, and society and are often utilized to guide and facilitate decision-making processes addressing complex multidisciplinary problems. Indeed, the NISE comprises a number of interdisciplinary teams working on a variety of issues related to SMET education, an approach promoted by NSF’s Directorate on Education and Human Resources. Interest in understanding and improving this approach to problem solving has led to a substantial body of literature and much "wisdom of practice" on interdisciplinarity in recent years.

The goal of the Cognitive Studies of Interdisciplinary Communication (CSIC) Team of NISE has been to advance the scientific understanding of how social and cognitive processes interact to drive intellectual growth and construction of intellectual products in interdisciplinary groups doing research on important societal problems. This knowledge has informed the design of better procedures and technologies for improving interdisciplinary collaboration and effort, particularly within the Institute.

Accomplishments to Date

The primary focus during Year 3, our final year, was on data analysis, data collection, and developing a methodology for identifying and understanding group knowledge construction. Our decision to focus on knowledge construction was based on a belief that productive collaboration is dependent on high-level discourse that promotes learning within professional communities. We were particularly interested in how group practices and selected classes of technological tools afford or constrain group and individual knowledge construction within interdisciplinary groups. Our aim is to provide guidelines for scaffolding collaborative teamwork and learning. Specific analyses completed this year focused on two primary areas: mental model misalignment as a source of cognitive conflict, and the use of analogy. We also engaged in several outreach activities to present findings and share ideas for improving practice.

Data Collection

Our data collection goals for Year 3 were to continue data collection of two NISE teams and to expand our data collection activities to at least one interdisciplinary group operating outside the NISE. Supplementing our data with data on an outside group working in a different context allowed us to identify those characteristics of format, process, and interaction that were similar and those that were dissimilar, thereby serving as another way to validate our findings and analyses.

The new group we "observed" was a research team working and interacting in a "multiuser virtual environment" (MUVE), the Teacher Professional Development Institute (TAPPED IN) at Stanford Research Institute (SRI). TAPPED IN is focused on enhancing teacher professional development activities by providing an on-line "institute" where teachers can go to discuss issues, create and share resources, hold workshops and seminars, engage in mentor-apprentice interactions, and conduct collaborative inquires. Data collection was made simple since their meetings are held on-line.

During Year 3 over 100 hours of meetings and interaction of two NISE teams and the on-line group were recorded. Related correspondence and products for the three groups were collected and archived.

Research Methodology

The primary research methodology for our studies is discourse analysis of meeting conversations. Teams as wholes, rather than individuals, are the unit of analysis. We have developed methods for coding and analyzing discourse that are based on several theoretical frameworks: situative, sociocognitive, argumentation, and distributed cognition theories.

Even though these frameworks may identify knowledge construction occurring in the same places, the same data might be interpreted differently depending on the theoretical assumptions of the framework, because the mechanisms and processes underlying the analysis are different within each framework. Using multiple frameworks allowed us to develop a multifaceted, multilevel description of knowledge construction.

Our research methodology is described in detail in a forthcoming chapter on knowledge building practices in technology-mediated interactions (Derry, Gance & Gance, forthcoming). It will also provide the basis for evaluating one of the new initiatives proposed for NISE years 4 and 5 entitled "Secondary Teacher Education Project (STEP): A Technology-Based Distributed Professional Learning Community as a Model for Teacher Education."

Research Analyses

Mental Models in Team Work. One of our analyses examined the mental models of members in an interdisciplinary team to identify how model misalignment contributed to team conflict. In group work, each individual brings to the team a unique conceptualization of the task and of the team. These conceptualizations, which develop based on prior experiences, can be represented as "mental models" of the team situation.

In individual problem solving, a person’s mental model of the situation shapes his or her recognition of and approach to a problem. In group work, individuals’ mental models are similarly important, but the group environment introduces a new factor—the interaction of multiple mental models—that also influences the team’s productivity. In particular, the degree of similarity among team members’ models affects team members’ ability to work together and to communicate effectively about the task and team.

In our analysis entitled "Mental Models in an Interdisciplinary Team" (DuRussel & Derry, in progress), conversational and interview data revealed that team members differed significantly in their models of the task, of task distribution, and of their overall view of the conflict. The differences in participants’ mental models were related to a number of factors, including the lack of explicit communication about goals and analytical approach, the distribution of work among team members, and a lack of continuity in attendance at team meetings. In general, team members approached the group work with disparate expectations about how the work should be distributed among team members and what the appropriate approach to the task was. While work distribution is common in working teams, in the case studied here the analyst’s expectations for his or her role were not in alignment with the expectations held by other team members. As a result, an in-group/out-group split was created that caused both cognitive and social conflict to escalate on both sides. It appeared that this model misalignment was exacerbated by the fact that team meetings contained little explicit communication about either team goals or specific analytical approaches. As a result, team members were unable to align their models of how the work should be distributed or of what the appropriate analysis should be.

Mental model misalignment also was influenced by inconsistent attendance at team meetings. Team meetings are one obvious forum for aligning team members’ views of the situation. Even when goals or team process are not explicitly discussed, common experiences at team meetings allow for implicit alignment of team members’ models. In the case studied here, two participants’ radically different models of the overall situation seemed linked to the fact that they were not always present at the same meetings. Because of this lack of temporal continuity, each participant’s model of the team and task were being influenced by different stimuli, and thus the two models did not achieve adequate alignment.

Overall, misalignments among participants’ models of the task and of task distribution contributed to the escalation of a conflict among team members and hampered team productivity. This research suggests that misalignments in team members' models of the teamwork can have significant effects on the team process and productivity.

Analogy. Cognitive research has shown that analogical reasoning can be central to theory building, argumentation, and problem solving. However, few studies have examined analogical reasoning in natural group settings. Our second analysis, "Analogical Reasoning in an Interdisciplinary Working Group" (DuRussel, in progress), supplements the existing body of research by exploring how one working team used an analogy of "education as a pathway" to conceptualize issues of equity and educational progress and analyze a complex data set.

Several dimensions of analogical reasoning were examined, including (1) how the content and structure of the analogy source (pathway) and target (student education) influenced the team's analogical reasoning; (2) how the pathway analogy was evaluated in the course of team work; (3) how the group environment (including participant background, role distribution, and leadership) influenced the reasoning process; and (4) how individuals differed in their view of the analogy. The analysis revealed that ambiguities in both the source and the target domains contributed to team members' difficulty in applying the analogy. Group conversations revealed evidence that the analogy was being evaluated over time, but individuals differed in their opinions of the utility of the analogy. Individual differences in analogy definition, application, and evaluation were linked to both participant background and participant role in the group, with large variance among participants. In general, the pathway metaphor was more successful as an expressive metaphor (used to describe student educational experiences) than as an explanatory analogy (used as a tool in analysis).

Outreach

One objective of our work has been to apply what we have learned from our research to the improvement of communication and better designs for team-based work, particularly within the NISE but also in broader national and international academic communitites. To this end we have undertaken several publication and meeting organization projects as a way to foster greater understanding of interdisciplinarity.

Publications. Along with the chapter and articles presenting our methodology and research findings, we have worked on two publications intended for a broader audience.

Following the two-day conference we held in November 1996 on interdisciplinary teamwork, a synopsis of the issues and ideas that emerged during the conference was published as an NISE report entitled "Understanding Interdisciplinary Teamwork: Challenges for Research and Practice" (Gance, 1998). The report reviews key questions for research, the role of theory in research (particularly theoretical frameworks within the field of cognitive science), methods and methodology, and other general issues. The report also summarizes implications for practice and important considerations for designing effective environments for such collaboration.

In addition, an edited book on interdisciplinarity is in progress (Derry & Gernsbacher, in progress). Titled "The Problems and Promises of Interdisciplinary Scholarship: Perspectives from Cognitive Science," the book will include chapters on strategies and roadblocks to interdisciplinary scholarship by Charles Goodwin, John Bruer, Rogers Hall, Johanna Moore, Leigh Starr, Paul Thagard, Gavriel Salomon, and others.

Conferences. Aside from presenting our findings at several conferences and a workshop (AERA, CogSci98, SRI, and an invited address at APA) and locally (at NISE and the UW Educational Psychology Department), we have been engaged in the planning and co-sponsorship of two meetings:

The CogSci98 annual conference taking place at the University of Wisconsin-Madison, 1-4 August 1998, is being cosponsored by NISE. It will focus on interdisciplinarity. This focus will be manifested in four ways: (1) the plenary session is dedicated to the topic of interdisciplinarity; (2) several invited tutorial symposia are designed to provide breadth of interest; (3) one strand of the conference program has been set aside for presentations from Cognitive Science Society members that address interdisciplinarity as an object of empirical and theoretical research; and (4) all submissions for spoken presentations are being evaluated for their ability to transcend their disciplinary boundaries and represent the breadth of the community of cognitive science (in addition to being evaluated for technical and theoretical merit).

We are also organizing a graduate symposium with the Sesame Street cast on making interdisciplinarity work. The symposium, which will take place at the University of Wisconsin–Madison on September 22, 1998, will be open to all faculty and students and will be titled "Understanding Collaborative Work: The Children's Television Workshop as a Case Study in Successful Interdisciplinary Practice." In a panel discussion led by Sharon Derry, members of the Children’s Television Workshop (CTW) will discuss how their interdisciplinary practice has evolved over thirty years of work.

Interdisciplinarity—the integration of concepts, philosophies, and methodologies from different disciplines—is often considered highly desirable as a working practice. This is especially the case when complex problems and projects cannot be understood or solved within the constraints of a single discipline. Many institutions, such as the University of Wisconsin–Madison and the National Science Foundation, strongly promote interdisciplinary collaboration through their policies and funding strategies. At the same time, true interdisciplinarity is known to be extremely difficult to achieve since the positions and approaches adopted by disparate disciplines are often incompatible. In fact, cases of truly successful interdisciplinary collaboration are hard to find. The CTW visit offers a rare opportunity to learn more about interdisciplinary practice from a highly successful interdisciplinary team.

Staff: Sharon Derry (Team Leader), Lori Adams DuRussel, and Laura Lee Gance. During Year 3, the CSIC Team has worked with Angela O’Donnell, Rutgers University, and Mark Schlager, Stanford Research Institute.

NISE Formative Evaluation

The goal of the Formative Evaluation team is to provide the NISE leadership with real-time information about the development of the Institute as an organization in order to help the NISE better articulate its strategies and make midcourse corrections to maximize its goal achievement.

Accomplishments to Date

The role of the Formative Evaluation team has been different during each of the NISE’s first three years. During Year 1, the team provided the NISE leaders with "third-party" formative feedback on how various NISE constituents were experiencing and viewing the NISE teams and the NISE overall. This feedback was provided both informally and in the form of four reports. Two of these reports were based on interviews with NISE participants; one on one interviews with members of representative collaborating organizations; and one on one surveys completed by participants in the First Annual Forum. Each of these reports was carefully reviewed by the Team Leader and Management Teams. These teams found the reports and the discussions of critical value in making organizational improvements.

In the second year, the Formative Evaluation team continued to provide formative feedback information of the same type offered in Year 1, providing two formative feedback reports based on interviews with members of the Institute, plus an evaluation of the Second Annual Forum, "Research on Systemic Reform: What Have We Learned? What Do We Need to Know?" Again, these reports were discussed and used by the NISE leadership to enhance the effectiveness of the organization. During Year 2, the team also adopted an "insider" role by working closely with the Strategies for Evaluating Systemic Reform, Policy Analysis of Systemic Reform, and Interacting with Professional Audiences Teams to plan and organize the Second Annual Forum. In addition, the Formative Evaluation team took major responsibility for managing the production of the Forum Proceedings, and produced a Forum synthesis based on participant think pieces. The synthesis comprises a key component of the Proceedings and was very positively reviewed.

During Year 3, the Formative Evaluation team provided the NISE Team Leader and Management Teams with one more round of formative feedback information about the organization overall and conducted an evaluation of the Third Annual Forum, "Indicators of Success in Postsecondary Science, Mathematics, Engineering and Technology Education: Shapes of the Future." By the middle of Year 3, the team also had become a full member of the cross-team Third Annual Forum planning group. Because the members of the Formative Evaluation team are researchers at the UW-Madison Learning through Evaluation, Adaptation and Dissemination (LEAD) Center, they not only brought to this planning group knowledge obtained by conducting evaluations of the NISE forums, but also expertise developed through SMET evaluation research at the LEAD Center. After the Forum, the Formative Evaluation team managed the production of the Forum Proceedings and, working closely with a College Level One Fellow Elaine Seymour, produced a forum synthesis similar to that produced for the Second Annual Forum. In addition, throughout Year Three, the Formative Evaluation team participated in the Graduate Education Forum (June 29-30, 1998) planning group, bringing to this group expertise acquired through evaluating and planning the first three Forums, producing syntheses for the Year 2 and Year 3 Forum Proceedings, and conducting evaluation research through the LEAD Center. During the first part of Year Four, the team will continue to work with the Graduate Education Forum group by producing a synthesis document of the same type produced for the second and third Forums, and by helping with the production of the Graduate Education Forum Proceedings.

The change in the Formative Evaluation team’s role over the first three years of the NISE is a function of two factors: (1) the NISE team leaders believe that the NISE has evolved to the point where the benefits of a continued intensive and formal formative evaluation process are no longer commensurate with the cost of this effort; and (2) the topics of the Third NISE Forum (assessment in postsecondary SMET education) and the Graduate Education Forum were in the area of expertise of the Formative Evaluation Team, which is comprised of members of the LEAD Center.

Strategic Plans for Year 4 and Implications for Year 5

As explained above, the Institute evolved during its first three years to the point that it outgrew its need for additional formative evaluation by the middle of Year 3. However, a paper synthesizing the key themes that emerged from the formative evaluation work remains to be completed. This paper will present the NISE as a case study that explores the processes and emerging principles by which cross-disciplinary organizations develop and the importance of formative evaluation for the development of the organization. In addition to presenting key findings from the NISE formative evaluation studies, the proposed paper will seek to answer the question of how the development of the NISE fits within the context of the research literature on other cross-disciplinary organizations.

The primary purpose of the formative evaluation synthesis paper is to serve the NISE, NSF leaders and program officers, and others seeking to better understand these processes and principles. Such understandings are of value as we move with ever increasing momentum into a period when, on the one hand, key problems in science, mathematics, and engineering education can no longer be solved by people from a single field of expertise, yet on the other hand, few experts are trained to work effectively in cross-disciplinary teams.

This synthesis will be undertaken primarily by the leader of the Formative Evaluation team. She will conduct a literature search during the early development stage of this project. While developing the initial outline and first draft of this paper, she will confer with the former members of the Formative Evaluation team to benefit from their knowledge of the NISE. She also will share these early drafts with the Team Leaders Team, (in particular with Sharon Derry, the Cognitive Studies in Interdisciplinary Collaboration team leader) to plumb their knowledge as emerging reflective practitioners of the art of developing a cross-disciplinary organization and to allay any concerns about infringement of confidentiality. The final draft also will be reviewed by the Team Leaders. The formative evaluation synthesis paper is expected to be completed by the end of February 1999.

Primary Deliverable:

The deliverable for this project will be an NISE Occasional Paper, which also will be submitted to an appropriate journal.

Staff: Susan Millar (Team Leader)

Collaborators: Dianne Bowcock, Sarah Pfatteicher, Ramona Gunter, and Sarah Mason.