New Spredsheet version with optional extinction correction
Citizen Sky is now officially permanent part of the AAVSO. In the coming weeks we will be moving additional content to the AAVSO site and freezing this site as an archive of the 1st three years of the project. Please visit the new landing page for future updates.
Since the old thread was somewhat framgemented I thought I'd start a new one. For most observers eps aur will now be observable at relatively low air mass, so I thought it might be a good idea to extend the Excel sheet with a simple "toggle switch" that will switch on/off air mass correction. In the attached file, you can do just that in cell F51
1 ==> air mass correction is performed 0 ==> air mass correction is not performed (= extinction coefficient is set to 0. No need to input observation location coordinates and observation time in this case)
Note that for both cases, the error in the comparison star mag fit is shown, so that you can actually see by trying both settings which method gives the better fit.
I'm still working on my backlog of data starting in August (!) when airmass was still ~2 for me, but in theory, with eps aur rising higher, the linear regression with air mass correction should become more and more unstable so that eventually the analysis w/o airmass should give a better fit.
Just for fun I thought I would compare your spreadsheet (using "0", no air mass correction with Brian’s spreadsheet. The data I used had eps Aur almost at zenith (air mass = 1.0145). I was getting close but not exact results between the two systems until I realized some tabulated values for V-Cat and (B-V) Cat were slightly different between the spreadsheets. Once I standardized all the tabulated data the V magnitudes came out the same.
I did notice one difference between the two spreadsheets, however. HB, the formula you use for D Cat is D Cat = V Cat – I Mag. Brian’s formula is D Cat = I Mag – V Cat. Obviously, this makes no difference in the final results but I thought it was worth a mention.
Also, HB, is it possible to include the graph of (B-V) Cat vs D Cat in your spreadsheet? For those of us who are more visual than analytical it really helps.
That adds quite some confidence. Definitely, a Plot would be a good visual check. I thought about two other extensions: 1) Some automatic quality checks, that, if violated, would make some "traffic lights stype" indicator cells go red and result in a warning test to be displayed, e.g. when the coeffiecents resulting from regression have unusual or completely implausible values. 2) an optional 2nd order extinction coefficient that is not calculated by regression ( I think that would only make the calculation quite unstable) but would default to some fixed value. We discussed this here before. For example, in "Handbook of AIP", it's proposed to used a fixed value of 0.03. Clear Skies HB
These sound like excellent (and sophisticated) enhancements. I just hope they'll run on my older ('97) version of Excel. If not, well, this old timer has needed a reason to finally upgrade. 8-) Warm regards, Tom
I just discovered a liitle bug in my sheet: the (only informative) check star magnitude calculation should also depend on the "toggle". I'll add that tomorrow. This won't influence the target star magnitude figure.# CU HB
Hi! I've uploaded a new version of the sheet as attachment to the original post. Changes:
All internal tables are now on a separate sheet. Looks cleaner IMHO
2nd sheet is now reserved for a How-To (will come later)
Changed color scheme: User input is now expected in AAVSO-website-blue fields... I needed yellow for ...
Data Quality indicators: For now only two: fit errors for comp stars and check star. More to come (e.g. airmass value plausibility)
Last but not least: two plots: plotted are the correction terms for transmission coefficient and extinction coefficient vs (B-V) and airmass, respectively. Can also be used to judge if enabling airmass correction did improve the fit.
Tom, do those diagrams do what you wanted? CS Heinz-Bernd
Hello HB, I really like the "New Version". The graphs are great. Thanks, too, for correcting my mistake with my sign for longitude. I've attached my results for a 1 Nov observation (V mag 3.792 for Eps Aur). I noticed in another post you were headed out to observe at about that time. Are our results compatible? Warm regards, Tom p.s. In row 13, what does "LST" represent? I thought it was Local Standard Time at first but the numbers don't seem to match up with that interpretation.
Hi Tom, Very reasonable V-mag. Dr. Bob's digikam (a JPEG device) got 3.76 (Nov. 1/2). Jeff Hopkins got V= 3.7428 +- 0.0034 (that's a standard deviation, not uncertainty) on Oct. 30/31 so you're surely in the ballpark. If you look at the errors in the fit zeta, lambda, and rho Aur don't fit very well. As for "LST" in row 13, it's a measurement of time called theLocal Sidereal Timewhich is required to calculate the air mass. It's fixed to background stars and results in a day that's about 23 hours 56 minutes long instead of 24 hours even. Something is also broken in the spreadsheet you attached. The V-mag isn't updating when you toggle the air mass correction on or off. A couple other notes (mostly for me and Heinz):
We shouldn't put the Imags for the check star and target star up at the top. We use the check star on a per-frame basis and the target object similarly, therefore their imags should be extracted one at a time and included below.
I think we should move the Fit Value/Error table back to the Data sheet. The information contained there is very useful (for instance it can show that you measured the wrong star which resulted in a bad fit) and shouldn't be hidden on a different sheet. The other table can stay, but they need to be labeled.
After we have the final version complete (which I hope this spreadsheet almost is), we need to propagate the errors from the fit. I would also like the errors from the calibration stars to be included... I'll think of an easy way for this to be done.
Hi Brian, Thanks very much for the comparison data. I'm encouraged. It looks like you and HB are rapidly closing in on the final version of the Air Mass Spreadsheet. I congratulate you both on this impressive accomplishment. Tom
Hi Brian, >Something is also broken in the spreadsheet you attached. >The V-mag isn't updating when you toggle the air mass correction on or off Actually it does, you just have to format the cell in question to show more significant digits :-). I think this is really good: it shows that at an airmass this low, extiction doesn't have a big effect anymore, but if you nevertheless try to correct for it, it doesn't mess up your results. >We shouldn't put the Imags for the check star and target star up at the top. We use the check star on a >per-frame basis and the target object similarly, therefore their imags should be extracted one at a time and >included below. You mean the I mags should be near the result table, as for the optional additional images? Yeah, I agree. >I think we should move the Fit Value/Error table back to the Data sheet. Ok, I think logically they would belong in the "data quality" section. Ok ? > The other table can stay, but they need to be labeled. OK. I'll see when I've got time to do all this, but I think we are close to finished now. CU HB
Hi Heinz-Bernd, I really like the new spreadsheet. Moving the intermediate tables to a new spreadsheet does make it look nicer, but I think having that information right there is handy. If we are going to move them we need to make sure the tables are properly labeled. The residual plots are a very nice enhancement, but I'm confused by them. The vs. air mass residual plot doesn't subtract the air mass effect, neither does the vs. color plot. For these plots to be called residuals they really need to account for the mentioned effect instead of just displaying it. In the attached spreadsheet I just plotted up the "error after fit" vs. airmass and "error after fit" vs. color and then looked at the resulting slope and r-squared coefficient. For the vs. color diagram the slope is nearly 0 and the r-squared is 0.31, implying little-to-no correlation so we've solved that problem. The vs. Air mass plot has a slope of 0.04 and an r-squared of 0.71 implying there are second-order effects that need to be considered (which we expect to be present as we don't account for second-order extinction effects). I would suggest we display these two plots instead. I should also point out that the slopes of these lines also show up (in magnitude) in the error cells in the Planar fit table on the data sheet. The data quality indicators are a good idea. We should revise our introduction tutorials to conform to the new coloring scheme you implemented (I'll take care of that once we post up the spreadsheet). I think we should keep the instructions for how to use the sheet on the main website for one major reason: we can fix errors and provide updates. The secondary reason is that people can comment on the page where we post the instructions. I think this helps promote dialog between us and the users of our software. What do you all think? Do we think this is ready to go main-stream?
Hi Brian, Just to explain, what I tried to do with the digrams was to show the color and extinction correction separately (as it's difficult to do that in 3D), so that the slopes in the diagrams would indicate the transmission and extinction coefficients, respectively. Without extinction correction, the first diagram would actually be the same diagram we had in earlier spread sheet versions (color vs (instr mag - catalog mag) ). So the diagrams would show the first order effects, not a remaining second order effect, I guess the "residual" in the caption was quite misleading. But I'm ok with the alternative diagrams, as with your other suggestions. CS HB P.S.: I have to run now, it's a clear night :-)
Hello again Heinz, Ah, okay. For a few minutes I thought that's what you were trying to do, but as you mention the "Residual" thing in the caption was indeed misleading. If you re-label your plots and/or put in the residual plots I suggest it should be more than adequate. I hope your clear night is going well! Cheers, Brian
I moved back the most helper tables related to extinction calculation to the main sheet. This is more than cosmetic: it will allow users to add/remove comparison stars by just inserting a line in the comparision star table and filling in the catalog data, all the helper table formulae should then be auto-created automatically. That would not happen if they were on a separate sheet I guess.
I added a new plot for residual magnitude versus air-mass times color index, to estimate second order effects. Not sure this makes sense.
added a plausibility check for the air mass data ranges. I assumed values > 10 indictate an error. I guess most typing errors that result in significant errors in observer site coordinates (e.g. switching signs) or observation time (e.g. local time instead of UTC) stand a good chance to be discovered this way
Instrument magnitudes for check and target stars are now entered in a line of the result table.
moved the LST calculation to the "Internal" sheet because it doesn't seem to be important enough to be displayed to the user.
Added a hint for the correct sign of geo-coordinates of observation site
Removed the Doku sheet
I also moved around some cells for aestethic reasons, hope that didn't break anything :-) CU HB
Hi Heinz, This looks good. I'll re-write the documentation today and get it posted on/before Monday.. then we can get a blog post out about it at the beginning of the week too! Brian
I gave your new spreadsheet a try and found it performed very well. The "toggle" idea is neat! I’ve attached a copy of my results using data from 30 April 2010. The air mass was 1.97. The average error with air mass correction (1) was 0.0189 and without correction (0) was 0.0451.
Eps Aur magnitude was 3.737 with correction and 3.776 without correction. I compared these results with the Hopkins Campaign’s V magnitude light curve (http://www.hposoft.com/Plots09/VBand.JPG
). The range of magnitudes for30 April is between 3.75 and 3.78. The reason these results are closest to my "without correction" magnitude may be that the majority of Hopkins participants are not performing air mass corrections.
Hi Tom, I had a look at your spreadsheet, it gives much better results if you use a negative sign for your observer location longitude. Of course this is pure convention, or is there an accepted standard in astronomy whether W or E should be negative? (I would not be surprised if there were two opinions on that, one "American", and one "English") :-) I'll include a hint in the next and some checks in the next version of the sheet. CS HB
Greetings HB, East longitudes are positive, West are negative. In conversation we tend to say 107 degrees West whereas we really mean -107 degrees East. Brian
Greetings HB, Very good idea! I'll use this as the default air mass calculator when I get around to posting content online (Tom Pearson and I have been revising the text in the tutorials in preparation for an upcoming article). Would you like to write a how-to tutorial for this spreadsheet? It could be similar in spirit to the non air-mass corrected one. Brian
Hello again HB,
Just for fun I thought I would compare your spreadsheet (using "0", no air mass correction with Brian’s spreadsheet. The data I used had eps Aur almost at zenith (air mass = 1.0145). I was getting close but not exact results between the two systems until I realized some tabulated values for V-Cat and (B-V) Cat were slightly different between the spreadsheets. Once I standardized all the tabulated data the V magnitudes came out the same.
I did notice one difference between the two spreadsheets, however. HB, the formula you use for D Cat is D Cat = V Cat – I Mag. Brian’s formula is D Cat = I Mag – V Cat. Obviously, this makes no difference in the final results but I thought it was worth a mention.
Also, HB, is it possible to include the graph of (B-V) Cat vs D Cat in your spreadsheet? For those of us who are more visual than analytical it really helps.
Thanks and warm regards, Tom
That adds quite some confidence. Definitely, a Plot would be a good visual check. I thought about two other extensions: 1) Some automatic quality checks, that, if violated, would make some "traffic lights stype" indicator cells go red and result in a warning test to be displayed, e.g. when the coeffiecents resulting from regression have unusual or completely implausible values. 2) an optional 2nd order extinction coefficient that is not calculated by regression ( I think that would only make the calculation quite unstable) but would default to some fixed value. We discussed this here before. For example, in "Handbook of AIP", it's proposed to used a fixed value of 0.03. Clear Skies HB
These sound like excellent (and sophisticated) enhancements. I just hope they'll run on my older ('97) version of Excel. If not, well, this old timer has needed a reason to finally upgrade. 8-) Warm regards, Tom
I just discovered a liitle bug in my sheet: the (only informative) check star magnitude calculation should also depend on the "toggle". I'll add that tomorrow. This won't influence the target star magnitude figure.# CU HB
Hi! I've uploaded a new version of the sheet as attachment to the original post. Changes:
Tom, do those diagrams do what you wanted? CS Heinz-Bernd
Hello HB, I really like the "New Version". The graphs are great. Thanks, too, for correcting my mistake with my sign for longitude. I've attached my results for a 1 Nov observation (V mag 3.792 for Eps Aur). I noticed in another post you were headed out to observe at about that time. Are our results compatible? Warm regards, Tom p.s. In row 13, what does "LST" represent? I thought it was Local Standard Time at first but the numbers don't seem to match up with that interpretation.
Hi Tom, Very reasonable V-mag. Dr. Bob's digikam (a JPEG device) got 3.76 (Nov. 1/2). Jeff Hopkins got V= 3.7428 +- 0.0034 (that's a standard deviation, not uncertainty) on Oct. 30/31 so you're surely in the ballpark. If you look at the errors in the fit zeta, lambda, and rho Aur don't fit very well. As for "LST" in row 13, it's a measurement of time called theLocal Sidereal Timewhich is required to calculate the air mass. It's fixed to background stars and results in a day that's about 23 hours 56 minutes long instead of 24 hours even. Something is also broken in the spreadsheet you attached. The V-mag isn't updating when you toggle the air mass correction on or off. A couple other notes (mostly for me and Heinz):
Cheers, Brian
Hi Brian, Thanks very much for the comparison data. I'm encouraged. It looks like you and HB are rapidly closing in on the final version of the Air Mass Spreadsheet. I congratulate you both on this impressive accomplishment. Tom
Hi Brian, >Something is also broken in the spreadsheet you attached. >The V-mag isn't updating when you toggle the air mass correction on or off Actually it does, you just have to format the cell in question to show more significant digits :-). I think this is really good: it shows that at an airmass this low, extiction doesn't have a big effect anymore, but if you nevertheless try to correct for it, it doesn't mess up your results. >We shouldn't put the Imags for the check star and target star up at the top. We use the check star on a >per-frame basis and the target object similarly, therefore their imags should be extracted one at a time and >included below. You mean the I mags should be near the result table, as for the optional additional images? Yeah, I agree. >I think we should move the Fit Value/Error table back to the Data sheet. Ok, I think logically they would belong in the "data quality" section. Ok ? > The other table can stay, but they need to be labeled. OK. I'll see when I've got time to do all this, but I think we are close to finished now. CU HB
Hi Heinz-Bernd, I really like the new spreadsheet. Moving the intermediate tables to a new spreadsheet does make it look nicer, but I think having that information right there is handy. If we are going to move them we need to make sure the tables are properly labeled. The residual plots are a very nice enhancement, but I'm confused by them. The vs. air mass residual plot doesn't subtract the air mass effect, neither does the vs. color plot. For these plots to be called residuals they really need to account for the mentioned effect instead of just displaying it. In the attached spreadsheet I just plotted up the "error after fit" vs. airmass and "error after fit" vs. color and then looked at the resulting slope and r-squared coefficient. For the vs. color diagram the slope is nearly 0 and the r-squared is 0.31, implying little-to-no correlation so we've solved that problem. The vs. Air mass plot has a slope of 0.04 and an r-squared of 0.71 implying there are second-order effects that need to be considered (which we expect to be present as we don't account for second-order extinction effects). I would suggest we display these two plots instead. I should also point out that the slopes of these lines also show up (in magnitude) in the error cells in the Planar fit table on the data sheet. The data quality indicators are a good idea. We should revise our introduction tutorials to conform to the new coloring scheme you implemented (I'll take care of that once we post up the spreadsheet). I think we should keep the instructions for how to use the sheet on the main website for one major reason: we can fix errors and provide updates. The secondary reason is that people can comment on the page where we post the instructions. I think this helps promote dialog between us and the users of our software. What do you all think? Do we think this is ready to go main-stream?
Hi Brian, Just to explain, what I tried to do with the digrams was to show the color and extinction correction separately (as it's difficult to do that in 3D), so that the slopes in the diagrams would indicate the transmission and extinction coefficients, respectively. Without extinction correction, the first diagram would actually be the same diagram we had in earlier spread sheet versions (color vs (instr mag - catalog mag) ). So the diagrams would show the first order effects, not a remaining second order effect, I guess the "residual" in the caption was quite misleading. But I'm ok with the alternative diagrams, as with your other suggestions. CS HB P.S.: I have to run now, it's a clear night :-)
Hello again Heinz, Ah, okay. For a few minutes I thought that's what you were trying to do, but as you mention the "Residual" thing in the caption was indeed misleading. If you re-label your plots and/or put in the residual plots I suggest it should be more than adequate. I hope your clear night is going well! Cheers, Brian
Hi! So I did the following changes:
I also moved around some cells for aestethic reasons, hope that didn't break anything :-) CU HB
yet another helper table went back to the main sheet CU HB
Hi Heinz, This looks good. I'll re-write the documentation today and get it posted on/before Monday.. then we can get a blog post out about it at the beginning of the week too! Brian
Hello Heinz,
I gave your new spreadsheet a try and found it performed very well. The "toggle" idea is neat! I’ve attached a copy of my results using data from 30 April 2010. The air mass was 1.97. The average error with air mass correction (1) was 0.0189 and without correction (0) was 0.0451.
Eps Aur magnitude was 3.737 with correction and 3.776 without correction. I compared these results with the Hopkins Campaign’s V magnitude light curve (http://www.hposoft.com/Plots09/VBand.JPG
). The range of magnitudes for30 April is between 3.75 and 3.78. The reason these results are closest to my "without correction" magnitude may be that the majority of Hopkins participants are not performing air mass corrections.Warm regards, Tom
Hi Tom, I had a look at your spreadsheet, it gives much better results if you use a negative sign for your observer location longitude. Of course this is pure convention, or is there an accepted standard in astronomy whether W or E should be negative? (I would not be surprised if there were two opinions on that, one "American", and one "English") :-) I'll include a hint in the next and some checks in the next version of the sheet. CS HB
Greetings HB, East longitudes are positive, West are negative. In conversation we tend to say 107 degrees West whereas we really mean -107 degrees East. Brian
Greetings HB, Very good idea! I'll use this as the default air mass calculator when I get around to posting content online (Tom Pearson and I have been revising the text in the tutorials in preparation for an upcoming article). Would you like to write a how-to tutorial for this spreadsheet? It could be similar in spirit to the non air-mass corrected one. Brian
Sure, I can do that How-To. CS HB