Help with IRIS tutorial

Hello Chris, In fact with recent DSLR (let say since ~2005) and few seconds exposure (like needed at 100 ISOfor epsilon AUR and surrounding stars- mag 3~6 )the bias process and even the dark process are absolutly not needed. They are even worst than doing nothing as anyhow introducing some residual read noise in the picture ! But you need a flat forthecorrection of thevignetting of the lens(except some DSLR that do it) With recent CMOS dark process starts improving images only at one or more minutes ofexposure depending conditions. Bias is no more needed. The only thing is the systematic ADC coding shift, said offset, being applied by some brands like Canon. This is a constant beingadded to the image valuesto avoid to clip the negative components of the random noise. If I remember well, in IRIS, you have the choice tojust introduce that constant instead the name of the bias image. If you use a Canon the offset is 1024 ADUs for the 14 bits types or 256 for the 12 bits types, thatfor most of their cameras. Nikon normallydon't applyoffset (just put 0). Otherwise you can read one imagewith"file/load raw file",click left, peak a small image section in a dark area, click right,choose the "statistic"function, the average value is the offset.You could alsojustput "0" as offset, the elimination of such offset is not so important as the background is anyhow subtracted at the photometry phase of the process. Yours truly, Roger

Hi Chris, If the dark frames in the IRISdemonstration set are the same exposure time as the image frames(which you check in the fits headers) you do not need bias frames. Just subtract the mean dark frame from all the image frames. Bias framesare onlyreally needed if your darks are a different length exposure to the images and therefore need to scale the darks for the different exposure time. "Why would you ever take darks of different exposure time to the image?" I hear you say. Well it can save time as you can have a standard dark which you can use for any exposure. This only works if you have a camera with good temperature control however where you can relyon the darks being constant overthe observing run Robin

Chris, Thank you for pointing out this error, I thought I packaged bias frames with the distributed data set. Checking through my directories I don't have any bias frames so I think we need to modify our tutorials to reflect that. Roger and Robin: What might you two suggest we state? I've synced the team page with the IRIS tutorial page so you can make changes there. Thanks, Brian

Sorry Brian I have not been following the DSLR photometry thread and jumped in assuming the demo set was a set of CCD fits files from Christian Buil's website, not your set of DSLR images. Standard practiseat pro observertories ofcourse isbias darks and flats butI would say that in the case of DSLR images wherethe camera temperature is not stable enough to allow scaling of darks, thebias framescould beleft out and just darks used. (Ideally lights and darks should be interspersed as the temperature of the CCDin the camera increases as more images are taken unless enough time is allowed between each shot for the heat to disipate) Roger raises an interesting point asto whether darksare even required. My view would be that unless you are sure thatthe darkfor your particular camera does not contain any systematic variations between pixels at a level above the required precision, darks should be used and enough darks should beaveraged to limit the additional stochastic noise to an acceptable level. The reason for this is that systematic errors are far harder to deal with than stochastic errors. I would say generally researchers would trade a bit of random variability to eliminate the possibility of an unidentifiedsystematic error. Robin

Hi Brian, I think there is no need for such bias frame, this is a process that was useful with old DSLR and CCD camerasforcalculating darksbutuseless with today sensors. I have checked the dark frames, the black level is well flat at exactly128 everywhere in the image: no need for bias frame even if this 20D is not a recent camera. 128 ADUs is the systematic offset used by Canon in its 12 bits cameras. It could be removed or left: no issue. If someone as an older camera, as explained by Robin, a possible bias drift will be anyhowincluded in the dark. What we need is just to explain to people how to bypass that point in IRIS. If I remember well there is a way to do it. I am fully booked for the two coming days, I could check this on nextmonday. To be clear, my feeling is that such process is too complex for attracting beginners. In fact it's a full process for astro imaging, notjust for photometry of bright star.Ithink that most of the time, with recent cameras, number of phases of that processare useless for photometry of bright starsinvolved in CS. I will have more time in following weeks, If you like I would be happy to review this and propose something more attractive. Maybe the way is to propose several options, one being very simple just for first trials. Yours truly, Roger

Hi Roger, I amsurprised that your 20D bias frame shows no variability at all.I checked Christian Buils tests of this camera andhis bias frame shows a clear pattern which would difinitely need a bias (or dark) to correct http://astrosurf.com/buil/20d/20dvs10d.htm Are you sure that the raw frame convertion that you are using is not clipping the low bits ?

Hi Robin, It's not my 20D, but the images of the tutorial that have been made with a 20D. These images include the "guard" made from the 128 ADUs offset systematically used by Canon to avoide the clipping of low levels and the negative part of the Gaussian noise. This systematic offset has not been removed in previous manipulation of these images this guaranties no low level data has been clipped. In fact all softwares I know use DCRAW from Dave Coffin (including my own software, IRIS, IAP4WIN..) The use of DCRAW with the "pure raw" option ( dcraw -D -4) ensures that the pgm output file is a pure image of theBayer pattern delivered by the camera without anyfurther processing other than thedecompression of the cr2 file. On recent DSLR/DSCs I have tested the bias shows absolutely no drift, anything is within one ADU. I am not so surprise with this as it seems clearto methe electronics controles it.The last thing I have seen is on a 5 years old D70 that showed stripes at level of about 2 ADUs, but in addition they were somewhat unstable by the way impossible to correct well by subtracting frames. Yours truly, Roger

Hi Roger, Attached is the histogram of a single raw bias frame Ijust took with my Canon 350D and loaded into IRIS. Ican also confirma patterning similar to that seen by Buil in the 20D and other DSLR he has tested. Robin EDIT Ihave nowlooked at the differerence betweentwo bias frames and find that the standard deviation is very close to (single frame sd)/sqrt(2)which is as expected for random variability so it does indeed seem that the pattern is unstable and cannot be removed using bias/darks. Very different from my cooled astro CCDcamera wherea similarpattern is very repeatable from exposure to exposure. Iclearly have more to learnabout the characteristics of these cameras. Robin

Hi all, Still we need to figure out how to handle this in the tutorial. I think the reason that bias frames were discussed there was not so much that we thought it would be necessary but IRISinsisting on something as bias frames in the dialogs used. So....for the short exposure times that we use for bright star DSLR photometry, I wonder whether we might explicitly tell users to use the darks as bias frames as well maybe?? Because the bias frames are subtracted from the darks, flats, and the light frames, this should work out in the end to have the same effect as not using bias frames and only dark frames , right ? CS HB

I don't know enough about how IRIS handles darks and bias frames internally so I'm going to defer to you and Robin on this issue. Perhaps you could verify that darks and bias frames are treated equally? Brian

Hi! Yup, will do that and report here CS Heinz

Hello!! Sorry for the late answer.. Indeed the guys are correct that there is no need for bias (and perhaps dark frames..).. But with the procedure presented for the IRIStutorial you need to "gibe" bias. So either you have a series of bias frames and give them like what you do in the dark frames, or you create a fake one by typing at the terminal (after opening a light image): >fill 0 >save master-bias So you select this as a bias frame and go on. This is the procedure to overcome the problem (the 0 don't affect the rest of the image processing). Grigoris

Provided the darks have the same exposure time as the images, Ibelieve Grigoris' solution of creating a nullmaster biasframe is the best way to go for the processing of the image. (It is the technique recommended by Buil somewhere but I can't find where on his website at the moment) Note however that the DSLR master flat creation routine in IRISassumes that flats are taken using short exposures and therefore does notuse a flat darkand subtracts only the bias. http://www.astrosurf.com/buil/iris/tutorial3/doc13_us.htm This meansan artificial bias will not work correctly for the flat.When processing the flat I suggestthe flat dark frames are used in place of the bias frames Robin

Hi Chris, nice to see you here! I have used Iris for epsilon Aurigae since august last year. Until nowIhave done81 observation days and a total of345 pictures has been processed and analyzed (I usually take 4-5 series each time). For eachimage I meassure epsilon and 20 compairson stars. So it has been 6000+ stars meassured by hand now! I use Excel to calculate airmass and correction coefficients. For each of the compairsion stars I putthem inthe equation: V = Vi +Tv * (B-V) + k' * AM + Zp. I know Vi (instrumental magnitud from the picture), B-V (the color index of the compairson star) and AM (the airmass of the comparsion star from its altitude). I then let excel calculate the best fit for Tv, k' and Zp by regression analysis. By comparing the calculted V and the stars V-magnitude fromthe Tycho2-catalogand then calcultaing standard deviation of all the diffrences I could see how good the qualityof the pictureis or how good the photometry work. At last I use the calculated coefficients and apply them to epsilon to got its V-magnitude. That was a bit of the background... As a beginner of Iris last year I read the tutorials on Iris web-site and has therfore used bias and dark frames. I also use a white frame to correct vignetering. I suspected that the bias and dark really didn't matter that much but I have not tested without them before I read this thread. I did a such test now on 4 images and can confirm that Ipractically got the same result with or without bias/dark. With bias/dark I got the following s.d for the 4 images: 0.025, 0.018, 0.026, 0.018. Without bias/dark I got: 0.027, 0.019, 0.027, 0.017. I possibly got a slighter better result with bias/dark but it could also be a random thing as I only used a small sample of 4 images. When I do the processing I use a script that I run in Iris instead of using the dialogs in the program. It is faster, more reliable and you can do other things during the time the images is processed. Every thing you can do by the menus and dialogs in Iris you can also do by commands so I think that should be the thing to use in the tutorial. I only use Iris for processing of the pictures, not for photometry, there I use an other free program, TeleAuto. I did a lot of testing last year when I begun with photometry and I got better values (smaller s.d between the compairson stars and their catalog magnitude) with TeleAuto than with Iris. There is also a couple of things I don't like in Iris photometry function. - The automatic function seems optimized for doing photometry of a few stars on many images (only 5 stars could be selected). I would do many stars on a few images. -When usingthe PSF-function (point spread function) you draw a rectangel around the star you will meassure. But you got very different values if you use a small or big rectangle or a rectangle of different forms. There is no possibility to got consistant values. In TeleAuto you point on the pixel that you will meassure from. - The apperture function have that way of meaasuring (pointing on a pixel). But to get consistant values I always seek to point at the pixel that have the highest ADU count. It is a hard job manuvering the mouse to the right pixel. It would have been great to have a snap-function so that the cursor had jumped to the highest pixel in vincty of the pointer. Or if it had been possible to use amagnified image, nowyou can only use the1:1 scale. The aperture function in Iris also have to be carfully calibrated. I think that the PSF-function in TeleAuto is more forgiving. As I was in test mode this evening I also did a comparsion between TeleAuto PSF and Iris aperture photometry. I used the same 4 images as before. The FWHM value for them was around 2. I first used Iris default values for the ring radius (8 12 20) but got a s.d of 0.065, almost 3 times higher than the value of 0.025 from TeleAuto so I didn't do any more test with that radius. I then tested values of 2, 4 and 8 times FWHM (4 8 16). That was much better and I got s.d of 0.024, 0.020, 0.026, 0.022, almost as good as TeleAuto. For the first image I also tested ring radius of 5 8 16 and 4 8 12. There I got s.d of 0.035 and 0.031, a bit worse than with 4 8 16. I don't know if anyone else have tested this and have found out the optimal values for the ring radius? Thomas Karlsson Sweden

Hi Thomas, Many thanks for answering- so nice to be able to write to you in English? I am currently working my way through the tutorials for AIP4WIN and IRIS. I have come further in terms of instrumental magnitudes with AIP (see separate thread). The other night I took 4-5 images of beta Lyr which I thought I could have a go at as eps Aur is so low at the moment. Could you just tell me how to stack the images in IRIS and then perhaps I could measure them in AIP4WIN. By the way, what is the transformation coefficient for your camera? Des Loughney has +0.15 for his Canon 350D. It would be great to see you at Sagittarius if you have time! Otherwise I am continually my visual observing. All the best, Chris

Hi Robin, Attached is the histogram of a dark (9 images) of the 450D. To better see the asymetry of the foot of the curve I have applied a 100 gain and a 10000 gain (red curve)to the histogram (blue) That clearly shows a couple of peaks below 10 ADUs and a larger population at 1~2 ADUs. The 450D has a 14 bits ADC and the well depth is around 14800 ADUs at 100 ISO. Considering the FPN are in fact random in our case (moving star position in the pictures) my analysis is the impact of such residues is extremely low in our case (exposure of 4~10 sec) well below millimag. I have another way to analyse it: a simulator that uses the dark and other noise data from the camera. It shows similar results. What your feeling about that ? Yours truly, Roger

Hello Grigoris, Brian, As promised I got back to IRIS to better understand its internal processing, but as usual, that took me more time than expected... Grigoris, your proposal to use " > fill 0 " into the bias would work for DSLR like Nikon that use " unsigned integer " CFA representation but is a problem for others, like Canon, that use a kind of limited " signed integer " CFA coding. In that last case they use a specific coding of the output of the ADC (analog to digital converter). This is mostly to avoid clipping small negative value of Gaussian noise that would induce an error of background level (some cases of low value) The black level (zero signal) is not binary coded "zero" but at some value like 128, 256, 1024 depending the number of bits of the ADC. This is a constant, nothing to see with electronics errors like bias, drift and other bias patterning of CMOS. Leaving that value inDARK and into the photometry process would be no issue as the background is anyhow subtracted. But for all what relates toFLAT (non-linear operation) that constant shall be absolutely removed or an error is inducedin the correction coefficient, and/or next, to its application to the image (that correction has to apply ONLY on signal) This sytematic offset is not managed by IRIS even if such parameter of the camera should be known as the camera is identified. The user shall determine it himself. That could be done on a well dark image area using " > stat " or " click left / draw box / click right / statistics " The average value should be very near that systematic offset, easy to understand if that camera uses 128, (12 bits) 256 or 1024 (14 bits). Next the user shall fill the dummy image with " > fill 1024 " or what he find. If the image shows a consistent DC-bias it should be added. Next the flat correction shall be applied to a star image from which that offset and any DC-bias residual has been removed. The offset shall be removed from the dark too. This is apparently well managed by IRISWHEN the right value is introduced in the bias from start. There is no problem in IRIS as it uses " signed integers " including in files it generates. I am continuing to look at other questions. The fact toREGISTER and STACKimages is somewhat questionnable for photometry (this implies complex interpolations that are not obviously energy conservative, next we lose the opportunity to calculate the SD) I continue to believes it would be good to keep the raw values untouched even if the linear RGB conversion of IRIS seems ok from tests I made. I also wonder the COSMETIC improvement is a good thing for photometry (all these functions have in fact been developed by Buil for imaging not for photometry... he does it another way) Yours truly, Roger

I'm very new at this game and have been working through your IRIS tutorial. Just a comment on your question regarding whether IRIS can process a raw file without a bias frame: I guess there are 2 parts to this question - a) whether IRIS needs to do all the intermediate processing, and b) if not, is the resultant file suitable for useful photometry. I have just done question a) successfully with a single Canon CR2 file. My steps were: 1. After the Step 1 setup steps, I used Digital Photo >Decode RAW to produce a single .PIC file. This nearly doubled the size of the original .CR2 file. 2. Ithen jumped to Step 4 and used Digial Photo > Sequence CFAConversion, inputting the single pic file's generic name. This produced another .PIC file of 3x the size. After this processing step it allowed the RGBSeparation option under Digital Photo to become active. 3. Finally, I did Step 5 to separate out the 3 channels. Each file was back to approx. the same size as the original CFA file. Presumably one could stack a series of converted images before doing 3) above. Unfortunately, we have been experiencing cloudy weather here for the past 3 weeks, so I haven't had a chance to generate a useful series to try this on. Also, I can't vouch for the usefulness of the final colour-separated files - that would require someone with a lot more experience than me to assess. Finally, being a rank novice, I'd sure love to alter some of your instructions for IRIS as I found them quite confusing at various points. Just my tuppence worth, Paul. PSHow does one produce a dark-flat and where is it used in IRIS, or is it required only for the other 2 programs?

Hi Paul,

The "flat process" is mandatory to achieve accurate photometry with a DSLR and photo-lens. With recent DSLR the "dark process" is needed only in case of long exposure (let say > 20 s and > 800 ISO) Recent DSLR do not need at all a "bias process". Anyhow, in the past, both dark and bias were in general combined, they were separated only in case of specific usage of astro cameras.

BUT the "flat process" of IRIS is made such it imposes to have a bias subtracted from the master flat image. It's in the IRIS code and we can't by-pass it. In addition Canon cameras have a system offset, the data zero-point is shifted by a fixed constant (in general 128 for the 12 bits cameras and 1024 for the 14 bits cameras) This is to avoid the clipping of the negative part of the gaussian noise and make sure the dark level is not clipped. That means we need to subtract that system offset (or bias constant) from the master flat in IRIS. If not the division of the images by the flat is erroneous (non-linear math) This is the reason for the solution proposed in 3.1.2 (In old cameras the bias level - the zero-point - was affected by drift, patterns, and other defects, this was the reason for a master bias frame subtraction)

The "dark procees" is used to eliminate (subtraction) the noise-like fixed pattern due to the dispersion of the leakage current of the photodiodes (or photogates) of the sensor. A said "dark image" is also affected by Gaussian noise (random, space and time). The way to reduce this random noise and isolate the fixed pattern noise is to stack (average) several images (6~12..) That stack is the master dark to be subtracted from sky images. Dark images are taken under the conditions of observation (exposure time, ISO, temperature... any processing of the camera shall be disabled: working pure raw) in absolute dark (caps on lens, viewfinder, dim surrounding).

The "flat process" is used to compensate the sensitivity variations of the camera across the image. In DSLR and photo-lens the issue is mostly the vignetting of the lens (~50% at large aperture !) and the shadows of possible dust on the low-pass filter (at few mm of the sensor) A "flat" is the image of a perfectly uniform white target and it is divided from the sky images to compensate the sensitivity variations (after some normalization) Like for the dark, the master flat is made from a stack to reduce the Gaussian and the shot noises.

Making DSLR flat is not easy ! It's difficult to avoid any illumination gradient on the target, the target shall be NON GLOSSY and very uniform (0.01 mag error is 1% illumination difference !) The illumination of it shall be very uniform too. The distance between lens and target is not important, one meter or so is fine. The lamp shall be far (~10 meters) or a large diffusing surface shall be used (several meters at a couple of meters) without direct illumination of the target. The symmetry of the arrangement shall be as good as possible. At end it's advisable to stack to series of shots the camera being rotated on the lens axis by 180 deg. The camera shall be set at the observation conditions: aperture, focus, long enough exposure time... With a lot of care it's possible to achieve flat with a 1% accuracy... The flat process included in some cameras is probably not accurate enough for photometry.

In IRIS tutorial all this is under step 3 and application at step 3.5

I hope this makes the subject clear enough !

Yours truly,

Roger