Forums / General Discussion Forums / Citizen Sky in General / What's With the Strange Survey Questions for New Members?

What's With the Strange Survey Questions for New Members?

Citizen Sky is now officially permanent part of the AAVSO. In the coming weeks we will be moving additional content to the AAVSO site and freezing this site as an archive of the 1st three years of the project. Please visit the new landing page for future updates.
  • strict warning: Non-static method view::load() should not be called statically in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/views.module on line 879.
  • strict warning: Declaration of views_handler_filter::options_validate() should be compatible with views_handler::options_validate($form, &$form_state) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/handlers/views_handler_filter.inc on line 0.
  • strict warning: Declaration of views_handler_filter::options_submit() should be compatible with views_handler::options_submit($form, &$form_state) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/handlers/views_handler_filter.inc on line 0.
  • strict warning: Declaration of views_plugin_row::options_validate() should be compatible with views_plugin::options_validate(&$form, &$form_state) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/plugins/views_plugin_row.inc on line 0.
  • strict warning: Declaration of views_plugin_row::options_submit() should be compatible with views_plugin::options_submit(&$form, &$form_state) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/plugins/views_plugin_row.inc on line 0.
  • strict warning: Non-static method view::load() should not be called statically in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/views.module on line 879.
  • strict warning: Declaration of views_handler_field_user::init() should be compatible with views_handler_field::init(&$view, $options) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/modules/user/views_handler_field_user.inc on line 0.
  • strict warning: Declaration of views_handler_argument::init() should be compatible with views_handler::init(&$view, $options) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/handlers/views_handler_argument.inc on line 0.
  • strict warning: Declaration of views_handler_filter_boolean_operator::value_validate() should be compatible with views_handler_filter::value_validate($form, &$form_state) in /media/disk/www/citizensky.org/drupal/sites/all/modules/views/handlers/views_handler_filter_boolean_operator.inc on line 0.

tonyome's picture
tonyome
User offline. Last seen 2 years 32 weeks ago. Offline
Joined: 11/07/2009
Posts: 12
Teams: None

A few days ago I joined Citizen Sky and am looking forward to making my observations and entering the data.  But I did want to comment on the survey that is given to new members.  It was part of the sign up process.  It was rather long and took me a while to complete. A lot of the questions were ambiguious and strange, and almost condescending.  Did anyone else get that impression?  For example, if I remember correctly, a lot of the questions dealt with whether scientists can be "artistic" or not.  And, can scientific data be "artistic".   There were also several questions I didn't understand, where they kept repeating the terms "Physics, Chemistry, and Biology".  Are "Physics, Chemistry, and Biology all the same type of science, or different types of science?".  Weird stuff like that.  Finally, toward the end of the survey, they had fill in the blank questions, and one said "what is the difference between the sun and Epsilon Aurigae?  Be sure to use scientific evidence to back up your statement".  They also asked the difference between "intrinsic" and "extrinsic" brightness.  I don't know how other people feel, but coming here as an enthusiastic volunteer, I don't really need to be asked questions that seem a little condescending. Did some graduate student write these questions or something?


Hi, Tony. Yes, I am a graduate student. :) I'm working on a PhD in Science Education at Tufts University. I have a BS in astronomy and worked for the AAVSO for 11 years. I also wrote the National Science Foundation grant that funds this project. Click here for more of my background. I think you did not notice that the survey is optional. Before you took the survey, you were shown a web page that explained what the survey was for. It then gave you two buttons, one to skip it and one to take it. So far, about 65% of the participants opt to take the survey. Even though I am a graduate student, I did not write the nature of science questions. They are taken from well established questionnaires and assessments in the science education community. Click here for some more background info on the survey. I intend to present early results at the upcoming January meeting of the American Astronomical Society. I will also post a PDF of the poster on the CS web site. When you see the information in context maybe it will make more sense to you. I'm sorry you feel some of the questions are condescending. However, a wide range of questions are needed to get the proper metrics in an assessment. Good assessments will have questions that are far too easy and far too difficult for the audience or else you get ceiling and floor effects in the statistical analysis. Citizen Sky is a project aimed at everyone, not just amateur astronomers. We have plenty of people registered who report that they have no prior background in astronomy. (Although, the majority of participants do.)

Rebecca's picture
Rebecca
User offline. Last seen 2 weeks 2 days ago. Offline
Joined: 06/01/2009
Posts: 125

Hello tonyome,When we applied for the NSF grant to fund Citizen Sky part of the proposal was a science education research project - basically a study of how well the CS project is accomplishing its goals. In order to evaluate how Citizen Sky has effected its participants we have to have a good baseline of participants' knowledge base and attitude toward science when they first arrived. The questions were written by an education researcher and one of our CS Staff, Aaron Price. My sincere apologies if any of the questions seemed condescending to you - that is certainly not our intention. I'd like to note that the survey is optional - we don't require that you complete it in order to join Citizen Sky. Perhaps I should take a look at the page and reevaluate whether we make the purpose of the survey clear to those about to answer the questions. Thanks for your input. We're very happy to have you on board!RebeccaCS Project Manager

Susan Kelly's picture
Susan Kelly
User offline. Last seen 3 years 11 weeks ago. Offline
Joined: 08/08/2009
Posts: 21

I suppose the questions might be perplexing to people who have not participated in education research. Science education research has its unique set of challenges, one of which is the writing of bias-free items. Given the diversity of the Citizen Sky participants on many fronts (i.e. age, education, native language, culture) selection of items was surely an unusually challenging task. Having participated in NSF-supported projects before, I must say that this is one of the most well-conceived projects I have experienced. However time-consuming it may have been, patiently completing the survey is an easy way to support this effort.

tonyome's picture
tonyome
User offline. Last seen 2 years 32 weeks ago. Offline
Joined: 11/07/2009
Posts: 12
Teams: None

Thank you Susan and Jeff for your comments. I honestly had trouble understanding many of the questions, nor did I really know what the purpose of the questions was. I felt like I was back in college 30 years ago taking a test that I didn't know the answers too, lol. But I almost expected to be "kicked out" of the group if I wasn't smart enough to answer the questions correctly. I've been interested in astronomy my whole life and consider my knowledge to be high in the area, but I am not a professional. Aaron and the young lady who responded are professionals and they gave good explanations. But I still do think the survey could intimidate many lay people that want to join and be part of the project.


Hi, Tony. First, let me say that I am happy that it did not scare YOU away! Even though I may have different viewpoints, I'm very glad you brought this up. It's a good discussion that will probably answer quite a few questions other people have had but did not bring up. Of course, as a geek I love any chance to "talk shop" too. :)There is a messy-but-easy way to test your concern. Remember the survey is optional. So we can compare the activity of those who took the survey and those who did not. If it is intimidating people then we would expect that the activity of those who took the survey to be less. Drupal, the content management software behind this web site, keeps a record of the number of times a person logs into their Citizen Sky account. A quick SQL query of the database gives me these numbers:Average logins of people who took the survey (729): 3.44Average logins of people who did not take the survey (534): 3.14This is a very cheap way of testing the hypothesis. :) For a better comparison we'd want to do a t-test (like we did earlier on the epsilon Aurigae light curve) but I think this is good enough for this discussion. I think it shows that there is little difference between the groups. In hindsight, would I do the pre-test differently? Of course. For example, I've found a better Nature of Science assessment. The one I used was established in 1978. (Jeff: It was developed by over two dozen science professors at a major university. I can give you the reference if you want.) For about two decades it was the most used in the field. So it has been heavily vetted. However, in 2002 a new test was created by some others and it has since become the "standard" for measuring people's feelings on the nature of science. I may switch to that survey for future member registrations. I'm not sure as there are some other complicating factors (such as it uses open-ended items, so it actually takes longer to answer!). In my first reply, I described why the items need to be testing a wide range of ability. However, I did not get into the issues you had with the philosophical questions of science. One of the questions on the questionnaire is:A scientific theory is similar to a work of art in that they both express creativity.Note the question isn't asking if science IS art, but rather if science has similar creativity. There are four other questions on the survey that ask about creativity in science. One of the others is:Scientific knowledge expresses the creativity of scientists.Before I explain how answers to these two questions are analyzed, I'd like to have a discussion about the questions themselves. What do you (or anyone reading this thread) think? Is a scientific theory the result of creative work? Does scientific knowledge reflect creative thought?

tonyome's picture
tonyome
User offline. Last seen 2 years 32 weeks ago. Offline
Joined: 11/07/2009
Posts: 12
Teams: None

Regarding the statement "Scientific knowledge expresses the creativity of scientists" - I thought that was more false than true. Scientific knowledge are rational facts. When an astronomer views an object he/she gathers facts. The size, the distance, the spectrum, the nature of the object, etc. These are facts, do they express creativity on the part of the astronomer? Scientists are creative when they develop theories, but is "knowledge" creative? No, knowledge is knowledge. It's like saying "water freezes at 32 F, I'm being creative by telling you this". It doesn't make sense to me.Maybe you could explain more about the "Biology, Chemistry, Physics" questions. I think part of my problem is that I overanalyzed the questions. I'm a computer programmer in my job and I'm always focused on details.


The "biology, chemistry, physics" item is stated like this:The laws, theories, and concepts of biology, chemistry, and physics are related.As with the others, the respondent is asked to answer between "I strongly agree" to "I strongly disagree". This particular question is designed to find out what the person feels about the unified nature of science. The creators of the survey (Rubba and Anderson, 1978) describe it as: "Scientific knowledge is born out of an effort to understand the unity of nature. The knowledge produced by the various specialized sciences contribute to a network of laws, theories, and concepts. This systematized body gives science its explanatory and predictive power."In science ed this is a very important topic. There is tons of research showing that students leave what they learn in the classroom. They are unable to transfer what is learned in the classroom to everyday life, including other classrooms. It's one of the biggest and most fundamental problems in science education (it also applies to mathematics education as well).

KeyofSEE's picture
KeyofSEE
User offline. Last seen 4 years 22 weeks ago. Offline
Joined: 01/27/2010
Posts: 1
Teams: None

Thank-you for that. I thought it was just me. I've too recently signed up. Who said you had to have a degree to love astronomy?

skymanlin's picture
skymanlin
User offline. Last seen 3 years 47 weeks ago. Offline
Joined: 08/06/2010
Posts: 1
Teams: None

I am a graduate student and really havedifficults withthese questions.

Maria Shivers's picture
Maria Shivers
User offline. Last seen 3 years 47 weeks ago. Offline
Joined: 08/06/2010
Posts: 1
Teams: None

Hi, Just joined the forum and stuck with theat survery. I don't think that it is a good idea to force new users to fill the unwanted surverys which is all around the web these days. That could be an option like a banner in the side bard if the user is interested he/she can fill it.

Susan Kelly's picture
Susan Kelly
User offline. Last seen 3 years 11 weeks ago. Offline
Joined: 08/08/2009
Posts: 21

Maria, Please see Aaron Price's posts in this forum link: Explanation and Creativity in Science. Susan

Susan Kelly's picture
Susan Kelly
User offline. Last seen 3 years 11 weeks ago. Offline
Joined: 08/08/2009
Posts: 21

Maria, Please see Aaron Price's posts in this forum link: Explanation and Creativity in Science. Susan

Susan Kelly's picture
Susan Kelly
User offline. Last seen 3 years 11 weeks ago. Offline
Joined: 08/08/2009
Posts: 21

Maria, Please see Aaron Price's posts in this forum link: Explanation and Creativity in Science. Susan


Hi,Maria. The survey is indeed optional.Before you see it, you are given a web page that describes it and asks you to participate in the survey. Approximately 55% of the new registrants opt to take the survey, 45% opt-out. So far we have about 1,350 responses. The data is very interesting and I'll report on it here as soon as my dissertation is completed (likely this winter).

kakucis's picture
kakucis
User offline. Last seen 3 years 47 weeks ago. Offline
Joined: 08/06/2010
Posts: 2
Teams: None

I tried to fill this survey when I registered but after a five or so minute sgave up. It is too long for average person. You have to give them back if they fill the whole survey, othervise it is almost useless becoause people are too lazy in general.


I agree it was originally too long. We reduced its length by a third by removing the content oriented questions. Still, we've only had about 10 complaints out of around 1350 people who filled it out, which isn't that bad by survey standards. :) Usually you get a much higher push back. And it has helped us improve the project immensley.

kakucis's picture
kakucis
User offline. Last seen 3 years 47 weeks ago. Offline
Joined: 08/06/2010
Posts: 2
Teams: None

Well those are really impresive numbers! They I take my words back, people are not that lazy :) And this project is really great, no daubt about it!

mmgateway's picture
mmgateway
User offline. Last seen 3 years 35 weeks ago. Offline
Joined: 10/28/2010
Posts: 1
Teams: None

I am an undergrad. I find the question so intriguingly interesting but difficult. However it would be a pleasure and honor to join the team!

Powered by Drupal