How to sync a drawing with video.

In my last post I talked about some of the issues that arise when syncing a drawing with video. I also mentioned a couple of possible ways around the issues.

After working on it for a few days I believe i've charted out my path around these issues.

The first problem i tackled was the laggy drawing after creating a long drawing with the flash graphics (previously Drawing API).

The way i got around the laggy drawing. When you draw there is a pattern to the process of drawing. Even when using a pen or pencil on paper you follow the same process each time you draw something.

//_________________________________________________
//                             T H E  P R O C E S S

On paper, to draw a line you go through a process.
1) pick up a pencil,
2) put the pencil on the paper at some arbitrary point,
3) move the pencil around until you're satisfied with the length of the line,
4) then lastly you remove the pencil from the paper.

In flash, it works exactly the same way. The basic difference here is the pencil no longer exists, replaced by the mouse, and the pressed pencil is now a held click of the mouse.


//_________________________________________________
//                                 T H E  M A G I C

When it comes to Syncing a drawing with-- well, anything really, the magic happens between the click of the mouse and the release. Which in other terms is the time the pencil touches the paper until the pencil is removed from the paper.

A pencil line on a 2D piece of paper can be measured in 4 Dimensions 2 Different ways point by point, or vector. I'm going to be using point by point so i can allow the flash graphics API to take over.

Point by Point
1D) X coordinate
2D) Y coordinate
3D) Z coordinate (in this case the 3rd dimension is not important)
4D) Time (the smaller the measurement of time, the more accurate the drawing can be)

While the pencil is on the paper it goes through an almost infinite series of records that define the makeup of the line. For each millisecond (or less) that goes by, a point is created. If you were to define these series of coordinates and time in XML and call it a line it would look something similar to the XML below.


<line>
     <point x="50" y="50" time="12:28:50:10"/>
     <point x="55" y="50" time="12:28:50:15"/>
     <point x="60" y="50" time="12:28:50:20"/>
     <point x="65" y="50" time="12:28:50:25"/>
     <point x="70" y="50" time="12:28:50:30"/>
     <point x="75" y="50" time="12:28:50:35"/>
</line>


X and Y will look familiar, but the timestamp may not. In this example i'm using the time of day as the timestamp. In an actual project the timestamp would be the progress of a video or animation, whatever is being synced.


//_________________________________________________
//                             T H E  P R O J E C T

Where all this comes together is the video. To sync the drawing with the video i've formatted my xml file a little differently and created some functions to handle these new xml attributes.

<drawing>
     <segment id="1">
         <point id="1" x="50" y="50" time="00:00:00:05" type="start"/>
         <point id="2" x="55" y="50" time="00:00:00:10" type="drag"/>
         <point id="3" x="60" y="50" time="00:00:00:15" type="drag"/>
         <point id="4" x="65" y="50" time="00:00:00:20" type="drag"/>
         <point id="5" x="70" y="50" time="00:00:00:25" type="drag"/>
         <point id="6" x="75" y="50" time="00:00:00:30" type="stop"/>
     </segment>
     <segment id="2">
         <point id="7" x="75" y="55" time="00:00:00:90" type="start"/>
         <point id="8" x="75" y="60" time="00:00:00:95" type="drag"/>
         <point id="9" x="75" y="65" time="00:00:01:00" type="drag"/>
         <point id="10" x="75" y="70" time="00:00:01:05" type="drag"/>
         <point id="11" x="75" y="75" time="00:00:01:10" type="drag"/>
         <point id="12" x="75" y="80" time="00:00:01:15" type="stop"/>
     </segment>
</drawing>

The style is a bit different, its targetted toward drawing based on video timecode.

The <drawing> tag tells me that its starting a new drawing.

The <segment> tag tells me that its the mouse has been clicked and everything contained within the segment tag is still a part of that click (this is important -- i'll explain in a minute).
id="1" tells me that its the first segment thats been drawn.

The <point> tag tells me that theres a new point within the current segment thats part of the overall drawing.
each attribute of the point plays an important role.

ID: tells me which point it is. (like a social security #)
X: tells the X position
Y: tells the Y position
time: tells when the point was created based on the progress of the video at the time the point was drawn.
type: tells whether the point is the first in the series or not. This lets me make sure the lines start and stop when they're supposed to.

Back to the segment tag. This is how i fix the drawing API lag issue. When the mouse is pressed it creates a new segment. When the mouse is released, it ends the segment and waits for a new one.

When the drawing is actually created, the mouse press creates a new Sprite to hold the points and drawing to come. During the playback its necessary to distinguish between which segments are together in order to allow the flash player to cache the drawing as a bitmap sprite and render it quickly without lag.