The difference between online access panels and online citizens panels

Maaike Retra, Gerben Huijgen, Paulien Giesbertz



1 - Introduction


Everyone will have noticed that there is a growing number of surveys on the Internet; it has even been claimed that it will completely replace traditional data collection methods (Couper, 2000). According to NOPVO (2006), most data is now collected online (more than 50%). Most online surveys are currently carried out using Online fieldwork panels (Göritz, 2004). In 2006, there were more than 25 Online fieldwork panels in the Netherlands. Besides these research panels, more and more large organizations and councils are discovering the opportunities of Online fieldwork, and start up their own citizens panel to compile and analyze the opinions of their citizens.

With regard to citizens panels, what is most important for respondents is their social commitment and the desire to give their opinion. The most important reason why people participate in an access panel, however, is because they like doing it, to earn money or to give their opinion on specific subjects.

Various studies have shown that Online fieldwork can give valuable results (Göritz & Schumacher 2000, Göritz 2002). This survey investigates the differences and similarities in results between the citizens panel and the access panel. Does ‘self-selection bias’ result in undesirable panel effects, because respondents register for the citizens or access panel with a specific motivation?

In this document, the results are presented of a study carried out by I&O Research. The people approached for this survey are from the PanelClix panel, and those approached for the citizens panel are from the Enschede panel. In the Enschede panel, citizens of Enschede have the opportunity to give their opinion on local issues ten times during a period of one year.


2 - The Study


2.1 Composition of the sample

To test the differences between the online access panel and the citizens panel, an identical questionnaire was presented to both panels. Of the Enschede panel 692 members (356 men and 336 women) participated in the survey. Of the PanelClix panel 444 respondents (231 men and 213 women) participated. To be able to compare the results, PanelClix only invited members who were living in Enschede.

There are some difference in the composition of the sample groups. The citizens panel contained more highly educated respondents (HBO+) who participated in the survey (42% of the sample; 26% of the access panel is of the same HBO+ level). The composition of households in the Enschede panel for this sample also differed from that in the PanelClix panel. The Enschede panel consisted of more than 72,5% people who cohabit (of which around 50% have children). In the PanelClix panel, these two groups formed 60% of the sample, with single persons also being a large group (23%). The percentage of men and women that participated in the survey was similar in both panels (52% men and 48% women). The average age of respondents from the Enschede panel, 44.5 years, was slightly higher than the average age in the PanelClix group, 39.5 years. PanelClix had more younger people in its sample than the Enschede panel (31% in the age group of 16-30, compared to 20%). The age Group 31-50 was almost the same (Panelclix: 43%, Enschede panel: 46%). Finally, the Enschede panel consisted of more older people aged 51 and older in the sample (34%, compared to 26% in the PanelClix panel).

The field work took place over two weeks in the period of 30 May 2008 until 9 June 2008.


2.2 Composition of the questionnaire
    The online questionnaire consisted of 46 questions, dealing with the following three subjects;
  • The use of and preference for 14 digital participation tools (30 questions). This measured whether respondents would use certain new media in the communication between government and citizens. These participation tools could offer new opportunities to discuss political subjects, which can narrow the gap between voters and politicians. The respondents were asked two questions per participation tool (there was an extra question for two participation tools. (1. Would you, if the opportunity existed, use a…, 2. If the answer was yes or maybe yes: I would use … mostly for: *receiving information *submitting information *giving my opinion *giving advice *cooperating on the development of policy *participate in the policy decision making process *other)
  • Correspondence of citizens with government agencies (depending on whether the respondent had been in contact with an institution, eight question were asked).
  • As a control variable, questions were asked on a generic, non-political subject. Several questions were asked on the subject selected, the European Championship (these are the last eight questions, if the respondent did not follow the matches, there were seven questions).
2.3 Hypotheses

In view of the three subjects given, the following hypotheses were tested in this survey:

H0: There is no difference in preference for participation tools between members of the citizens panel and members of the access panel.

H0: here is no difference in correspondence with government agencies between members of the citizens panel and members of the access panel.

H0:here is no difference in interest in the football matches between members of the citizens panel and members of the access panel.


3 - Results


The questions that were part of the three subjects were tested using the T test and chi-square test. The tests show that there is a significant difference between certain questions, but in general there are more similarities between the two panels than there are differences. Of the 44 questions, there were ten significant differences in the results between the panels, and 34 questions where the results correspond. In the following paragraphs the similarities and significant differences will be discussed.


3.1 Digital participation tools

The use of and preference for the fourteen new digital participation tools was measured using 30 questions. These questions tested whether members of the access panels and members of the citizens panel gave differing answers in relation to the use of digital participation tools. The respondents were asked whether they would use the fourteen different participation tools (tested using the T test). If they showed an interest, they were shown a follow-up question on what they would use the new participation tool for (tested with the chi-square test). It can be expected that members of the Enschede panel are more interested in the use of these new media than members of the PanelClix panel. They have specifically registered for this panel to be more involved with the city of Enschede and might be more willing to use these new participation tools, to increase their involvement. By using the new digital tools, it is easier for them to stay in contact with government agencies to discuss political subjects concerning their city. Members of an access panel have not registered for the panel with this specific interest, which means the new digital tools may be of minor importance.


3.1.1 Significant differences relating to the use of certain digital tools

The T test for two independent sample surveys analyzed whether the panels display significant differences with regard to the use of and preference for the various participation tools. For five digital tools, respondents from the citizens panel turned out to judge the opportunities for these tools differently than the respondents from the access panel.There was a significant difference between the results from both panels with regard to the use of a GIS system (0.000 < 0.05). Using the GIS system, respondents can communicate problems in their neighbourhood. A map then shows which problems have been reported and what the status of that problem is. The results show that 86.4% of the members from the citizens panel answered they would use a GIS system, compared to 69.8% of the members of the access panel (will probably or very likely use the service). The F-value of Levene’s test is 1.548 and the significance is 0.214, which means that the significance is higher than α=0,05, or that the variances are not that dissimilar. The value of the T test for equal variances assumed is -6.064. The bilateral p-value is 0.000; which means that the null hypothesis is rejected with a 95% certainty and that there is a difference in both panels with regard to the use of a GIS system.


There is also a significant difference in results found between both panels (0.027 < 0.05) with regard to the use of ‘Wat stemt mijn raad’. At ‘Wat stemt mijn raad’ people can read short summaries of council decisions, including how the councillors voted. It showed that 50.7% of the members from the citizens panel, compared to 40.6% of the members of the access panel, would make use of this new medium. Levene’s test shows that the variances do not differ significantly (0.644 < 0.05). The bilateral significance is 0.027 (< 0.05), which means that there is a difference in both panels with regard to the use of ‘Wat stemt mijn raad’.

A significant deviation was found in the results with regard to the use of electronic voting (0.003). Electronic voting gives people the opportunity to vote via the Internet, for instance in local council elections. In the citizens panel, 87.9% of respondents indicated they would use electronic voting, compared to 82.4% of respondents from the access panel. The variance between both panels is not equal (0.042 < 0.05). The bilateral significance is 0.003 (< 0.05), which implies that there is a difference between the panels with regard to the use of electronic voting.

Finally there are significant differences in the results found with regard to chatting and uploading videos. The chatroom can be used to talk to other citizens and councillors. In the citizens panel, 15.6% of respondents indicated they would use chatting, compared to 25.7% of respondents from the access panel. Uploading videos can be compared to YouTube, with people being offered the opportunity to give their opinion about certain subjects using a video. In the citizens panel, 13.8% of respondents indicated they would use this service, compared to 22.3% of respondents from the access panel.

One explanation is that respondents from the PanelClix panel are more open to chatting and uploading videos because these respondents are relatively a lot younger than the respondents from the Enschede panel. Young people are more used to chatting online and uploading videos than older people and will be more likely to use these services.

The chi-square test was then used to test whether there is a statistical connection between both panels and what the digital tools will be used for. The null hypothesis for a similar test is always that there is no connection between the variables and that both variables (the use of media and both panels) are independent. For two of the fourteen variables, it seems that with 95% certainty, there is a statistic connection between the variables ‘Use of digital tools’ and ‘Panels’. These connections can be found in what the members would use the ‘E-petition’ (0.013 < 0.05) and ‘Wat stemt mijn raad' (0.027 < 0.05) for. For the use of the e-petition, there were significant differences between the panels for two answer categories (‘Receiving information’ and ‘Giving advice’). Respondents gave deviating answers to the answer categories ‘Receiving information’ and ‘Submitting information’ for ‘Wat stemt mijn raad’. It is important to find out how strong these four significant differences are. The strength of the connection was measured based on the correlation coefficient Cramérs V. The association masure shows a very weak relationship for all significant differences. (resp. ‘Receiving information’: V=0.068; ‘Giving advice’: V=0.079; ‘Receiving information’: V=0.125 and ‘Submitting information’: V=0.076. This shows that there is a very weak relationship between the type of panel and what they would use the digital tool for (with relation to the four answer categories which are significantly different). In other words, the members from the PanelClix panel and the members of the Enschede panel answered these two questions in nearly the same manner and there is little to no difference in the results.


3.1.2 Similarities between the panels

With regard to the use of the other nine digital tools no difference is shown in the results on the use of these new media. Both the members of the access panels and the members from the citizens panel did not give deviating answers. It also showed that the members of the access panels are open for the use of new media. They have indicated that they are more like to use chatting and the option to upload movies than the members of the citizens panel. This can imply that members of the access panel are as involved with regard to the use of new digital media as members from the citizens panel. Members from both panels indicated they were going to use twelve of the fourteen new digital media for the same purpose.

As of the 30 questions, seven questions showed significant differences (of which two significant differences showed that there hardly is a connection between the answer given and the panel), we can partly assume the null hypothesis. There is no difference in preference for participation tools between members of the citizens panel and members of the access panel.


3.2 Correspondence with government agencies

Of the eight questions regarding the correspondence with government agencies, one question showed a significant connection. For the other seven question, there were no differences in the results between members from the citizens panel and members from the access panel. Three of the eight questions were tested with the Chi-square test and the other five were tested using the T test.


3.2.1 Significant differences in correspondence with government agencies

The Chi-square test was used to test whether there was a relation between the correspondence with government agencies and both panels. The respondents from both panels were asked whether they had been in contact with any government agencies during the previous six months. The value of the Chi-square test (Pearson Chi-square) is 4.077 and the p-value is 0.043. This implies that with 95% reliability there seems to be a statistical relationship between the variables ‘Correspondence with government agency’ and ‘Type of panel’. It is important to find out how strong this connection is. The strength of the connection was measured based on the correlation coefficient Cramérs V. The correlation coefficient shows a very weak relationship: V=0.060. In other words, there is hardly a connection between both panels with regard to the degree in which they correspond with government agencies.


3.2.2 Similarities between the panels

The chi-square and T tests show no differences in results for the other seven questions. There does not seem to be a difference between members of the citizens panel and the access panel in what government agency they communicated with, whether the correspondence was via e-mail or letter, the speed of the reply by the government agency to the letter/e-mail, whether a confirmation of receipt was sent, the manner in which the request/question was handled, the score members gave and the language used in the letters/emails.

As the significant difference found shows a very weak relationship, the null hypothesis can be It is impossible to assume a difference in correspondence with government agencies between members of the citizens panel and members of the access panel.


3.3 The European Championship

As a control variable, it was decided to ask the respondents several questions about the European Championship. This is an independent variable, which does not assume any social involvement. This variable should not yield a result in differences for members of the Enschede panel and the PanelClix panel.

Of the eight questions, two questions turned out to be significantly different. There was no difference for the other six questions. The significant differences for these questions were tested as follows


3.3.1 Significant differences for the European Championship

The chi-square test shows a significant difference in the location where respondents will watch the games. For this question the answer category ‘At home’ and ‘Elsewhere’ turned out to be significantly different. The value of chi-square for ‘At home’ was 6.699 and the p-value was 0.010. The correlation coefficient showed there was a very weak relationship: V=0.092. For ‘Elsewhere’ the p-value is 0.000 and Cramérs V is 0.155, which implies that there is a very weak relationship here as well.
A significant relationship has also been found for respondents who think that the Netherlands will go through to the second stage. The value of chi-square is 4.744 and the p-value is 0.029. Cramérs V shows there is also a very weak connection here (V:0.065). The conclusion is that the nil relationship shows there is no difference in the answers given by the respondents from the PanelClix or Enschede panels on the questions where they are going to watch the matches or whether they think the Netherlands will go through to the next round.

Daarnaast is er ook een significant verband gevonden bij respondenten die denken dat Nederland door de voorronde heen komt. De waarde van de Chikwadraat is 4,774 en de overschrijdingskans is 0,029. Cramér’s V toont aan dat er ook hier sprake is van een zeer zwak verband (V:0,065). Geconcludeerd kan worden dat er door het nihile verband er geen verschil is in de resultaten die respondenten uit PanelClix of het Enschedepanel hebben gegeven op de vragen waar zij het EK gaan volgen of dat zij denken dat Nederland door de voorronde komt tussen.


3.3.2 Similarities between the panels

The results from both the respondents from the Enschede and the PanelClix panels show that there is no difference in whether or not people will watch the matches, the degree to which they follow the matches, how far the Dutch team will get, whether Boschker was dropped wrongfully or whether Engelaar deserved to be added to the team.
As the significant different found show a very weak relationship, the null hypothesis can be assumed. There is no difference in interest in the matches between members of the citizens panel and members of the access panel.


4 - Conclusions and recommendations


In advance of this survey, the expectation was that there would be big differences in the answers given by the access panel and the citizens panel. The reason is that by their participation to this panel members of the citizens panel could be assumed to be more socially involved than the members of the access panel. They are, after all, part of surveys which specifically apply to the city of Enschede, while the PanelClix surveys are more varied and include different types of subjects.

In contrast to our expectations, the survey showed that there were significant differences between the two panels on only a small number of questions. Most of the questions were answered similarly by respondents from both panels. The differences found for certain questions also do not confirm our expectations. It turned out that members of the PanelClix panel are more likely to make use of the digital tools ‘Chatting’ and ‘Uploading videos’ than members from the Enschede panel. One reason can be that the PanelClix respondents from the sample group are younger than the respondents from the Enschede panel. Younger people will be more positive towards media such as chatting and uploading, because they are more familiar with these media. One possible conclusion can bet hat members from the access team are as involved with regard to the use of new media to communicate with government agencies as members from the citizens panel.

The fact that there are so few differences between the panels and that the results mainly show similarities, explains why members from the access panel are as involved with the city of Enschede as members who registered specifically for the citizens panel. The survey shows that members from the access panel on the whole gave the same answers as members who registered for the citizens panel.

It can also be argued that the survey carried out with the citizens panel will yield the same results as a survey carried out with generic access panels. Panel effects as a result of self selection bias are minimal or totally absent.


Recommendation

In the sample survey, the number of younger and older respondents was not equally distributed over the PanelClix and Enschede panels. This could explain why respondents are more or less likely to make use of certain digital tools. Digital tools such as chatting and uploading photographs are used more often by younger people, which is why they will be more positive about these media and use them more than older people. A recommendation for future surveys is to keep the sample (sizes) of groups to be compared relatively similar.


References


Couper, Mick P. (2000) “Web-based surveys: A Review of Issues and Approaches.” Public Opinion Quarterly 64, 464-494
Göritz, A (2004) “Recruitment for online panels”. International Journal of fieldwork Vol 46 Quarter 4
R. van Ossenbruggen, T. Vonk, P.Willems, Uitkomsten Nederlands Onlinepanel, Vergelijkingsonderzoek (NOVPO). 2006 www.nopv