Research Blog

More text from an email:

-------------------------------------------

i just finished running some more shape completion experiments with single
partial contours and unknown correspondences. results were good for partial
contours very similar to the mean shape, as the algorithm tries to
find correspondences from the partial contour to the mean shape. results
for partial contours sufficiently different from the mean shape were not
good.

it seems to me that what we have here is a classic hidden variable problem:
we assume the completed shape should be a mix of principle components in our
shape model (i.e. the completed shape lies in eigenspace), and the
difficulty in finding correspondences is that we don't know a priori the
mixing coefficients (principle component coordinates) of the completed
shape. if we knew the mixing coefficients, we could use the resulting shape
(rather than the mean shape, as we are currently doing) to find
correspondences to the partial contour.

i don't know if it's worth the time to write this out as an EM problem, but
i just thought i'd point it out that the problem exists. for now, we could
try a sampling technique, for example:

1) sample 50 full shapes from the tangent space shape distribution.
2) for each sample, find the best correspondences to the partial shape.
3) use these correspondences and the sample full shape to sample several
orientations and scales.
4) complete the shapes for each (full shape sample, orientation, scale)
combination, and take the best one.

-------------------------------------------



Here is the data I was referring to; once again we use the occluded tools contour:




We again hand-segment out the following two partial contours:



For the partial contour on the right, results were almost identical (even slightly improved) to the previous experiment where correspondences from the partial shape to the mean shape were estimated by hand. Here is the best completion (in green) shown against the mean shape (in blue):



For the partial contour on the left, however, the situation was much worse. As is explained in the text from the email above, this is likely due to the fact that the partial contour was generated from a full contour which is significantly different from the mean shape, so finding correspondences to the mean shape is the wrong thing to do. Here is the best completion (the rest were just as bad):

Here are the effects of the first 5 eigenvectors on the mean shape for a model of a wire-cutter tool:







Here is a contour of two tools occluding eachother:



Segmenting this contour by hand yields the following two partial contours (shown in dashed-red):


With hand-labelled correspondences between partial contours and the mean shape, 50 (scale, orientation) pairs were sampled. Here are some of the resulting sample completed shapes (in green) for the first partial contour (on the left):






Clearly, either of the first two completions is to be preferred over the latter two; however, the maximum likelihood shape completion was closer to the bottom two:



The situation is slightly better for the second partial shape (on the right). Here are a few sample completions (in green):






and here is the best completion:



Here is a brief discussion (from an email to my advisor) of the challenges of shape completion as pertains to this experiment:

-------------------------------------------

i've just completed some experiments with known correspondences, and one
issue that stuck out as being problematic is that the shape completion is
highly sensitive to the eigenvalues of the gaussian eigenspace model. for
example, if the model thinks that opening and closing of a tool (a
wire-cutter) is much more likely than changes in perspective, the completed
shape will incorporate very little perspective change, even if the resulting
completed shape is quite ridiculous (e.g. makes the tool look like one of
the handles is crooked).

clearly this is a modelling problem, and not a flaw in the shape completion
algorithm. one way this issue is dealt with in some of the active contour
literature is to use the original dataset of complete contours from which
the model is generated to simply discover modes of transformation
(eigenvectors) while ignoring the relative eigenvalues associated with the
transformations. thus, such an discovery algorithm for the wire-cutter
should discover the modes of opening/closing, and one or two perspective
transformations. this approach makes sense when the changes in contour
appearance have more to do with external factors, such as camera positioning
or deformation by a human (opening/closing), rather than to internal changes
such as the change in shape between two different wire-cutters (as was the
case for the fish classification problem).

--------------------------------------------