Dynamic Shadow Elimination for Multi-Projector Displays

Yes I am posting very frequently nowadays. The reason is I am reading a lot of papers for my BTP. I have to give a presentation soon on Applications of Smart Camera-Projector Systems. A lot of cool things could be developed by combining a camera and a projector. I have already blogged about Keystone Correction and Presentaion Control using Laser Pointer. Today I am writing my notes on what I understood about Shadow Elimination using 2 or more projectors by reading this paper:-
Dynamic Shadow Elimination for Multi-Projector Displays - Compaq Tech Report CRLTR 2001/4.
This video(courtesy this site) explains the concept nicely:-

A major problem with interactive displays based on front-projection is that users cast undesirable shadows on the display surface. This situation is only partially-addressed by mounting a single projector at an extreme angle and pre-warping the projected image to undo keystoning distortions. But shadows can be muted by redundantly illuminating the display surface using multiple projectors, all mounted at different locations, and with the aid of a low resolution camera. The camera could be mounted at extreme angle to the screen.

This task is done in three steps:-
1. Automatic Multi-Projector alignment: First of all, all the projectors need to be aligned accurately so that they don't form multiple images on the screen (note that 2 projectors are enough for dynamic shadow elimination). For this, the transform for each projector to camera has to be determined. A few reference images are first displayed and their images are taken by camera. Using these, the transforms for each projector is determined by the method described in my earlier post on keystone correction. Note that, in keystone correction, there was only 1 projector. But here we have to align more than 1. For this the scaling+translation matrix S is choosen appropriately.

2. Occlusion detection: Note that we can only remove penumbral shadows. Umbral shadows i.e. the shadows formed when light from no projector is reaching at a spot could not be removed. Also note that a single occlusion can cause multiple shadows because we have multiple source of lights(projectors).

The technique used is very simple - negative feedback.
First a reference image is created for each slide. For this, each slide is presented and camera takes a few images of it. This is done as an intialization step and no occlusion should be present during this process. Note that this is kind of bad. Before each presentation, the system will create reference images of all the slides, which may take around may be 1 mintue. Also this technique will work only for static slides. Won't work for slides containing animations or videos. I think this concept of reference image is the only glitch in this whole technique of shadow elimination.
So after the initialization step is over, user can start the presentation any time. The camera takes images and calculates the pixel by pixel difference of the current image and its corresponding reference image taken earlier i.e. I_t(x,y) - I₀(x,y). The difference come out to be negative for pixels correspondig to shadow regions and almost 0 for normal regions. The system then adjust the intensities for the new time step as:-
Alpha is the intensity multiplication factor(0-255). Intially it is set at 256/(no. of projectors). For a 2 projector system, this is 128. Then this alpha is adjusted everytime based on the difference image - negative feedback. Note that this alpha is calculated for all the pixels. For a 2 projector system the value of alpha will change gradually from 128 to 256 for the pixles corresponding to the shadow regions. This perfectly eliminates the shadows. Note that the same alpha is valid for all the projectors and need not be calculated separately for each one. This is because the increase in alpha of the projector being occluded has no effect on the slide, because that projector is already occluded! The paper author used a value of 0.25 for gamma. He claims this leads to stabilization in around 3 steps for his experiments.

3. Shadow elimination: Now this alpha mask i.e the set of pixel differences calculated above needs to be applied to all the projectors. But we calculated this mask in camera frame/domain. We transform this in the original domain using the inverse of T, calculation of which I explained in my Keystone correction post. Now apply this mask and bingo! shadow is eliminated!

Next I am reading a paper from the same author on Dynamic Shadow Elimination with Occluder light suppression. Note that when a presenter comes in front of a projector, not only he casts a shadow but he/she is also annoyed with the bright light falling on his/her eyes. Occluder light suppression algorithm identifies which projector is being occluded and its corresponding pixels are turned off. So no light falls on the presentor and the shadow is still eliminated using the help of other projector(s). Note that the projector as a whole is not turned off but only the pixels that are under the shadow are turned off.
Watch this video to see it in action(courtesy this site):

I am not quite impressed with the results in above video. I would prefer simple shadow elimination without the occluded light suppression because the annoying to the presenter is not that important as compared to the artifacts that we see in the above video.