Dr Patrick Buckley, Department of Management and Marketing, Kemmy Business School, University of Limerick &
Dr Elaine Doyle, Department of Accounting and Finance, Kemmy Business School, University of Limerick

Video

Introduction

A pre-eminent theme in management science and information systems research is exploring how Information Systems (IS) can improve decision making at both an individual and organisational level. Prediction markets are a relatively novel form of Group Decision Support System (GDSS) that seek to leverage collective intelligence. They use a market mechanism to access and leverage information and knowledge from a large and diverse group of participants and stakeholders. There is a small but rapidly growing body of work in the IS literature exploring the challenges and benefits of using prediction markets to improve organisational decision making.

From a pedagogical perspective, many disciplines, including IS, aim to develop students’ decision making and analytical skills in real-world situations characterized by risk, uncertainty and constant change. Our pedagogical innovation is to demonstrate how prediction markets can be used in an educational context. This served two educational ends.

First, it provides students with a practical, real world example showing how IS can be used to capture collective intelligence and thereby improve organisational decision making.

Second, this adaptable pedagogical approach can be used in a variety of other disciplines to create individual and group based forecasting projects that develop discipline specific skills such as:

Asset Price Forecasting in Finance Modules [c.f. 3]
Risk Assessment in Risk Management Modules [c.f. 4]
Policy Forecasting and evaluation in Tax and Accounting Modules [c.f. 2]

The innovative and effective nature of this approach is validated by a range of peer reviewed publications and successful funding applications detailed in the relevant sections below.

Description of Innovation

In their simplest form, prediction markets are information systems that utilise contracts whose value depends on a future uncertain event to forecast outcomes. For example, a project manager may wish to evaluate whether a project will be completed on time. A contract is created which returns €100 if the project is completed on time and €0 otherwise. This contract is offered for sale to project employees at €50 (based on an initial 50:50 probability), typically on an electronic market. Market participants who believe the project will be completed on time will buy the contract, causing its price to rise. If they believe the contrary, they will sell the contract, reducing its price. The price of the contract is used as an estimate of the group’s collective assessment of the probability of the project being completed on schedule.

This simple approach can be easily extended to allow large groups of individuals to make forecasts about any future event. Our innovation is to use this tool in an educational context.

We have integrated prediction markets into numerous modules as continuous assessment projects. Students are provided with an endowment of virtual cash which they use to make forecasts on an online platform. The forecasts are tied to the modules’ specific learning outcomes. For example, in a Finance module, students are asked to forecast what the closing price of financial assets such as gold will be on a weekly basis, whereas in a Politics module, students may be asked to forecast the results of an election. It is important to note that as well as buying or selling contracts, students must provide a text rationale for the trade they made.

When students make successful predictions, they receive additional virtual cash, whereas poor forecasts result in a loss on investment. Students are ranked based on the amount of virtual cash they hold at the close of the market. This introduces a competitive element to the activity, as students can directly compare their performance with their peers. It can also be used to facilitate evaluation and marking.

When appropriate, we design the prediction markets to reset and repeat on a weekly basis. For example, students may be asked to forecast the weekly closing price of the Dow Jones Industrial Average. At the end of every week, the student’s endowment is reset. This avoids students becoming discouraged if they perform poorly in a given week. This structure has the additional advantage of creating a feedback loop that encourages critical thinking. Students receive feedback on their forecasts at the end of the week. If they perform poorly, the reset allows them to critically evaluate their decision making paradigm before starting again. Providing this feedback loop enables the reflection and introspection necessary for improving decision making and critical thinking skills.

A key enabler of learning, which is often difficult to scale, is feedback. Our approach results in a range of scalable feedback channels for students. First, they receive feedback from the operation of the market itself both at the close of the market and throughout its operation by virtue of observing the changing probabilities of forecasts. If at any stage students see that the collective estimate of a particular outcome is different from their own, they are prompted to re-evaluate the decisions they have made. They can also read the rationale provided by others for the forecasts they have made. Crucially, prediction markets allow students to change their decisions at any stage during the operation of the market by buying and selling contracts in response to this feedback and/or newly revealed information.

The prediction market system automatically generates individual emails detailing weekly performance and identifying correct and incorrect outcomes. Lecturers can integrate the project into traditional lectures by presenting and analysing the anonymised decisions of both successful and unsuccessful students. They can use the textual rationale provided by students to illustrate successful and unsuccessful strategies. None of these feedback mechanisms carry a prohibitive administrative burden and they scale easily to very large groups.

In general, student engagement with these projects is excellent. The unique approach captures and holds students’ attention. We have used prediction markets in undergraduate and post graduate modules across a wide range of disciplines, in classes ranging in size from 30 to over 400 students. Furthermore, we have shared the same project across similar modules in different Universities. This scalability speaks to the potential of prediction market utility for modern teaching environments such as MOOC’s.

Learning Outcomes

Improved Discipline Skills

Using our approach, academic leaders and lecturers can design forecasting problems that are tied to specific module learning outcomes. For example, Finance modules will often list improving students’ ability to price assets as a learning outcome. Our methodology allows academics to construct a forecasting problem to interrogate this learning outcome. Moreover, it can be used repeatedly over an academic term to investigate the change in student performance over time. For more information on this, please review the papers listed below, particularly [1] [4].

Improved information literacy

We see improved information literacy in students. These projects prompt them to synthesise information from a wide variety of sources of varying reliability to inform their decision making. As well as demonstrating how academic theories and research can be applied to real world contexts, students are prompted to deal with modern phenomena such as information overload and biased information sources. When running these projects, one phenomenon we invariably observe is misdirection. Students will post a false rationale or spurious web links in order to influence others to make poor decisions. Obviously, this behaviour is ethically dubious, but is also a realistic reflection of the real world, and another driver of improved information literacy. For further information and discussion on this, please review the papers listed below, in particular [1] [2].

Improved Motivation and Engagement

These projects have a positive effect on student engagement. In general, students describe them as unique and interesting. For example, one student commented “I thought it was a brilliant way to learn – Pure genius”. For further validation of this, please see the papers listed below, in particular [4].

Improved General Disciplinary Knowledge

To forecast well in a real world context, students must access a wider range of information sources than is typically required in a traditional module. They must pay attention to the news and consult sources such as government reports and surveys. One student noted “It made you read the newspapers and watch the news on a daily basis.” This has the ultimate effect of improving students’ general disciplinary knowledge, meeting our second objective. For further information, see the papers listed below, in particular [2]

Research Outputs

[1] Buckley, P., & Doyle, E. (2017). Individualising gamification: An investigation of the impact of learning styles and personality traits on the efficacy of gamification using a prediction market. Computers & Education, 106, 43–55. https://doi.org/10.1016/j.compedu.2016.11.009

[2] Buckley, P., & Doyle, E. (2015). Using web-based collaborative forecasting to enhance information literacy and disciplinary knowledge. Interactive Learning Environments, 24(7), 1574–1589. https://doi.org/10.1080/10494820.2015.1041399

[3] Buckley, P., & O’Brien, F. (2015). The effect of malicious manipulations on prediction market accuracy. Information Systems Frontiers, 1–13. https://doi.org/10.1007/s10796-015-9617-7

[4] Buckley, P., Garvey, J., & McGrath, F. (2011). A case study on using prediction markets as a rich environment for active learning. Computers & Education, 56(2), 418–428. https://doi.org/10.1016/j.compedu.2010.09.001

Research Funding and Impact Publications

2009: National Academy for the Integration of Teaching and Learning (NAIRTL), €5,000 for Research Project “Using Prediction Markets as Pedagogical Tools”

2011 – 2016: Grant Thornton Chartered Accountants, €3,000 p.a. for class based project “Forecasting with Prediction Markets”

“Game On: Employing Gamification to Enhance Teaching and Learning in Accountancy Education”, published in Chartered Accountants Ireland Magazine, 09/10/2020