On this page
• Summary
• Review of various methods of consultation including Carpenter Investments survey of June 2008
• Conclusions
On other pages
• Appendix 1 Carpenter Investments survey question 1
• Appendix 2 Carpenter Investments survey question 3
• Appendix 3 Carpenter Investments survey flaws
• Appendix 4 Carpenter Investments correspondence
• Appendix 5 Carpenter Investments questionnaire
• Appendix 6 Background
• Appendix 7 Carpenter Investments completed questionnaires
Summary
This report summarises the various attempts to gauge public opinion on the West Kirby Sail Project from the perspective of a professional statistician.
Since 2003, there have been at least 10 surveys of local public opinion on the Sail project, via public meetings and questionnaires. They have produced a range of figures from 46% in favour of the project to 98% against. The reasons for the apparent discrepancies are discussed in this report. The Carpenter Investments (CI) questionnaire is discussed in detail and it is shown that it was poorly designed (with ambiguous questions), that the execution of the survey was poor (many people were not included in the survey who should have been), and that the analysis has been poor (claims have been made that cannot be justified from the actual questionnaires).
It is concluded that none of the methods used for assessing public opinion have given support for the Sail proposal.
It is further concluded that all the methods used have been unsatisfactory - a well designed survey by an impartial market research organisation might well have determined public opinion in a way that would be acceptable to all, and a lot of time-consuming controversy might well have been avoided. Other alternatives worth considering are
• a council survey via a booklet delivered to each home (as has been done recently by Wirral PCT and by Merseyside Police Authority)
• registers of email addresses and telephone numbers of interested residents, so that opinions can be asked as project decisions arise (as is done by Wirral PCT)
• online surveys (as done recently by Havant Borough Council).
Introduction
Surveys of public opinion are not easy to carry out and questionnires need to be well designed to avoid bias and invalid conclusions. A question such as "Do you love West Kirby and want it stay the way it is: yes, no or undecided?" is clearly going to cause problems for people who love West Kirby but think that it can be improved. This question will be seen to be flawed by most people - but lesser errors in design will be less easy to spot by the layman. Professional market research organisations have expertise and strict codes of practice to ensure that survey questionnaires and other techniques are well designed to avoid bias and other problems. When surveys are carried out by non-specialists, problems of bias and difficulty of interpretation commonly arise.
Since 2003, There have been at least 10 surveys of public opinion on the Sail project. This report examines them in turn, from the perspective of a professional statistician. Details of the author's background are given in Appendix 6.
2004 Wirral Council consultation
About 300 people attended four workshops, and 48 feedback forms were returned. At that time, the emphasis was on replacing the sailing school. Feedback was summarised in the report www.wirral.gov.uk/minute/public/erccs041109rep5a_14472.pdf as follows.
1. General approval of this project and the inclusion of a Visitor/Information Centre for the beach area and Hilbre Island complex which may include commercial facilities. Most comments were around the scale/size of any development. Concerns expressed about a Budget Hotel around this specific site, however support was expressed for a new hotel facility but on perhaps an alternative site.
2. Most respondents and comments raised at the workshops in West Kirby didn't want car-parking spaces used for development purposes unless the spaces were replaced.
2. Most respondents and comments raised at the workshops in West Kirby didn't want car-parking spaces used for development purposes unless the spaces were replaced.
As the report points out, 48 feedback forms is a small number in comparison with the local population.
2007 public meetings
Public meetings were held in the summer of 2007 in Westbourne Hall, West Kirby. I believe that a show of hands on each of two occasions was against the building of a hotel on the Dee Lane car park site by about 95% to 5%. As with many public meetings, it is hard to know precisely what was asked, nor do we know accurately how many people were there. We also don't know how representative the attendees were of the general opinion in West Kirby.
Area Forum show of hands
The Sail project was discussed at the West Wirral Area Forum of 19 February 2008. A show of hands was against the hotel proposal by about 100 to 2 - see Minutes at www.wirral.gov.uk/minute/viewmins.asp?mtg=2422. There was a similar vote (about 200 to 5 against the hotel proposal) at the West Wirral Area Forum of 28 October 2008. While these are large numbers for public meetings, the people attending are just a small proportion (less than 1%) of the residents of the two relevant wards. The age distribution does not match the general population - younger adults are under-represented, presumably because of work and family commitments. Older residents are probably more likely to favour the status quo in preference to change, and so the Area Forum votes may overestimate the opposition to the hotel.
Carpenter Investments June 2008 questionnaire
This survey was carried out at the time of the West Kirby Concourse exhibition. It contained errors throughout all the aspects of design, implementation, analysis and interpretation. The questionnaire contained three questions.
The first question: This was poorly designed, being in reality two questions, and this caused many respondents difficulty in answering since they held different views on the two questions. There are two different sets of summary figures in the public domain. Neither of these can be supported by an analysis of the copies of the questionnaire forms that have been released. My analysis showed 47% forms against the current hotel proposal, and 41% in favour of development of some kind. There were 54% in favour of redevelopment of the sailing school. In interpreting these figures, it is important to remember that many people did not complete forms (see below). The first question is examined in Appendix 1.
The third question: This was seriously flawed in being ambiguous - different respondents replied in different ways. All three summaries that are in the public domain contain the same serious error of misinterpretation: a majority of respondents ticked "Yes" that "this has been a good opportunity ... to comment", and this was taken to mean approval for the consultation process; whereas my analysis of the 53% of forms with comments about the consultation process showed instead strong dissatisfaction with the process - forms with negative comments outnumber forms with positive comments by a ratio of 2.5 to 1. So forms show that many people felt it was a good opportunity to comment and their comment was that the consultation was poor. It is hard to understand how such a major error of interpretation could be made in the published summaries since perusal of the actual forms soon shows up major dissatisfaction with the process. The third question is examined in Appendix 2.
There were also serious errors in the conduct of the survey. For example, no survey forms were available at the start of the exhibition and at other times, and people wrote comments in a book carried by one of the developers - it is not clear how these comments have been included in analyses. These errors are detailed in Appendix 3.
There is uncertainty around some points, and I have asked Carpenter Investments for clarification. I have received a partial response and a threat of legal action - see correspondence in Appendix 4.
I have suggested to Carpenter Investments (14th Nov.) that in view of the doubts that are being raised over the design, analysis and interpretation of the survey forms, that they submit them for scrutiny and analysis by an impartial major market research organisation such as the Electoral Reform Society (this offers independent scrutiny of third party research - see www.erbs.co.uk/election_services/default.asp). See correspondence in Appendix 4.
Wirral News online poll
This was carried out on 11 July 2008 in conjunction with a "live webchat". At the end of the 60 minute session, the response to "Do you support the Sail project proposal?" was "Yes": 21%; "No": 79%. The newspaper did not publish the number of respondents, nor what measures they took to try to minimise multiple votes from the same person, nor if there was evidence of any such multiple voting.
Surveys by local politicians
Local Councillors have carried out surveys of residents by means of a tear-off slips attached to their political newsletter and delivered to local households. I understand that a survey in March 2008 was 98% against the hotel plan, and a survey in July 2008 was 92% against the plan. I do not have the size of these responses, nor the actual wording used.
The local MP is also currently conducting a survey via his newsletter.
Whilst newsletter surveys have obvious uses in allowing residents to flag up their concerns, they cannot take the place of a professionally designed impartial survey, since response rates are likely to be low, and the wording of questions may not be done effectively. It is a shame that the vacuum caused by the absence of a definitive survey has given rise to a plethora of informal surveys that are more likely to add to the confusion, than resolve the outstanding questions.
Other techniques
Possibilities include:
• "I would like a questionnaire from the local council to every home to see exactly what we ratepayers feel" (responder No. 194 to CI survey). This has been done recently by Wirral PCT and by Merseyside Police Authority.
• registers of email addresses and telephone numbers of interested residents, so that opinions can be asked as project decisions arise (as is done by Wirral PCT)
• online surveys (as done recently by Havant Borough Council - see www2.havant.gov.uk/AF3/an/default.aspx/RenderForm/?F.Name=La7Yy7S6ysx).
Conclusions
There have been many attempts to assess public opinion on the Sail project. They have all shown opposition to the project, but they differ as to the degree of opposition. The lack of agreement between them is due to none of them being a proper unbiased survey, as might have been done by an experienced market research organisation.
It is recommended that none of the existing surveys are adequate to enable an important decision to be made, and that any change to the status quo is deferred until an adequate survey of public opinion has been carried out.