Timecoding a streamed video

This post is as much for me as for anyone who reads this. Its to help gather my thoughts and try to predict any bugs or logical problems that might arise.

In some previous posts i mentioned a timecode that will be put into the xml file along with several other properties that help describe the point in space and time.

That part is fairly easy, all that needs to be done is to send off a timecode right?
Well, its not quite as easy as it might seem. Sending it off is the easy part, but getting the timecode itself to refer to a specific time during the video playback is bit harder.

The reason its hard. In flash, as far as i know, you cant just use the video timecode as a variable. The only access to the timecode seems to be through the video caption component, which could possibly be used, but i'll have to look into it later.

Anyhow, even if i could get that to work, a problem with the video arises. When a video loads it sometimes skips, or takes a little extra time to load and start playing the rest.

To get the current time and allow the drawing to work in sync yet independent of the video, we need to first figure out exactly what time the video begins playing. This is because no matter how fast or slow the connection, the starting time of the video will likely never be the same.

The flash movie broadcasts a time in milliseconds that represents the amount of time that has elapsed since the swf was initiated. This lets us find the time offset of the movie, which is not always the same, due to different connection speeds.

For example, a test i just did compares time offsets (time from swf initiation to video ready state) between 3 different connection types.
Local: 620 MS average offset (0.62 seconds)
56K : 11016 MS average offset (11.016 seconds)
DSL : 2159 MS (2.159 seconds)
T1 : 797 MS (0.797 seconds)

So according to my results, if i started recording mouseEvents for drawing as soon as the movie starts, the playback of the drawing could be anywhere from .62 seconds to 11 seconds out of sync with the video (assuming the drawing was recorded locally and played back via 56K modem (which isnt likely to happen), but the out of sync drawing will still be noticeable.

Currently, what i plan to do is calculate the offset for the recorded drawing. Then when the timecode is created and sent to xml it will be flexible..

For example;
A user records their drawing with a 500 MS offset. The offset is recorded as soon as the video starts playing. Once a drawing is initiated is grabs the current time and subtracts the offset.
if the first point were drawn at 00:00:00:550 it would be recorded into xml as 00:00:00:050. Then on playback, if the playback had an offset of 850 MS, the xml would start playing at 00:00:00:900 instead of 00:00:00:050.

This is one way around it. One issue i can foresee is the video buffer time, which may also have to be calculated and preserved.

Technical Update

Here are some updates on the current status of the parts of Synthea.

1. XML saved to server.
I've adopted the use of AMFPHP to allow flash to communicate with PHP directly.
This allowed me to create a class in flash called SaveXML. The SaveXML class takes 2 parameters, Directory, and XML. All this does is uploads any XML file that i send to it to the server for later use.

2. XML drawing playback.
I added the ability to playback a drawing based on XML tags and attributes. The tags include and the attributes (ID, X, Y, Time, Type) and any others that i may add in later.

3. Video Synchronization
This part is the heart of the project, and because of this, it takes longer to get working well. I am working on tests where the video is broadcasting a timecode and tests where xml is playing back via timecode. This leads me to believe that when its finally time to merge them together, we'll see some really cool results.


Those are the biggest hurdles so far. The way i have it, i'm doing the work of an entire development team, which I dont mind actually because its a great experience builder.

I have to approach the project from 4 different viewpoints, which constitute the 4 main areas of a development team. I have to approach it from 3 different programming angles on top of that.

1. Designer
2. Programmer
a. Interactivity Programmer (front end stuff)
b. Extensibility Programmer (making things easily extensible)
c. Server-side Programmer (back end php/database stuff)
3. Content Developer (video editor)
4. Project Manager

Because I'm doing this is Actionscript 3, its been very easy to separate development into stages like this. I can work on each part by itself, make sure the classes recieve what they're supposed to receive and return what they're supposed to return. I can also work on video on its own, and plug it in as i go. I can design externally and plug that in later too. The project management allows me to coordinate it all and make sure each part is doing its own thing. So I guess you could call me a one-man development team.

How to sync a drawing with video.

In my last post I talked about some of the issues that arise when syncing a drawing with video. I also mentioned a couple of possible ways around the issues.

After working on it for a few days I believe i've charted out my path around these issues.

The first problem i tackled was the laggy drawing after creating a long drawing with the flash graphics (previously Drawing API).

The way i got around the laggy drawing. When you draw there is a pattern to the process of drawing. Even when using a pen or pencil on paper you follow the same process each time you draw something.

//_________________________________________________
//                             T H E  P R O C E S S

On paper, to draw a line you go through a process.
1) pick up a pencil,
2) put the pencil on the paper at some arbitrary point,
3) move the pencil around until you're satisfied with the length of the line,
4) then lastly you remove the pencil from the paper.

In flash, it works exactly the same way. The basic difference here is the pencil no longer exists, replaced by the mouse, and the pressed pencil is now a held click of the mouse.


//_________________________________________________
//                                 T H E  M A G I C

When it comes to Syncing a drawing with-- well, anything really, the magic happens between the click of the mouse and the release. Which in other terms is the time the pencil touches the paper until the pencil is removed from the paper.

A pencil line on a 2D piece of paper can be measured in 4 Dimensions 2 Different ways point by point, or vector. I'm going to be using point by point so i can allow the flash graphics API to take over.

Point by Point
1D) X coordinate
2D) Y coordinate
3D) Z coordinate (in this case the 3rd dimension is not important)
4D) Time (the smaller the measurement of time, the more accurate the drawing can be)

While the pencil is on the paper it goes through an almost infinite series of records that define the makeup of the line. For each millisecond (or less) that goes by, a point is created. If you were to define these series of coordinates and time in XML and call it a line it would look something similar to the XML below.


<line>
     <point x="50" y="50" time="12:28:50:10"/>
     <point x="55" y="50" time="12:28:50:15"/>
     <point x="60" y="50" time="12:28:50:20"/>
     <point x="65" y="50" time="12:28:50:25"/>
     <point x="70" y="50" time="12:28:50:30"/>
     <point x="75" y="50" time="12:28:50:35"/>
</line>


X and Y will look familiar, but the timestamp may not. In this example i'm using the time of day as the timestamp. In an actual project the timestamp would be the progress of a video or animation, whatever is being synced.


//_________________________________________________
//                             T H E  P R O J E C T

Where all this comes together is the video. To sync the drawing with the video i've formatted my xml file a little differently and created some functions to handle these new xml attributes.

<drawing>
     <segment id="1">
         <point id="1" x="50" y="50" time="00:00:00:05" type="start"/>
         <point id="2" x="55" y="50" time="00:00:00:10" type="drag"/>
         <point id="3" x="60" y="50" time="00:00:00:15" type="drag"/>
         <point id="4" x="65" y="50" time="00:00:00:20" type="drag"/>
         <point id="5" x="70" y="50" time="00:00:00:25" type="drag"/>
         <point id="6" x="75" y="50" time="00:00:00:30" type="stop"/>
     </segment>
     <segment id="2">
         <point id="7" x="75" y="55" time="00:00:00:90" type="start"/>
         <point id="8" x="75" y="60" time="00:00:00:95" type="drag"/>
         <point id="9" x="75" y="65" time="00:00:01:00" type="drag"/>
         <point id="10" x="75" y="70" time="00:00:01:05" type="drag"/>
         <point id="11" x="75" y="75" time="00:00:01:10" type="drag"/>
         <point id="12" x="75" y="80" time="00:00:01:15" type="stop"/>
     </segment>
</drawing>

The style is a bit different, its targetted toward drawing based on video timecode.

The <drawing> tag tells me that its starting a new drawing.

The <segment> tag tells me that its the mouse has been clicked and everything contained within the segment tag is still a part of that click (this is important -- i'll explain in a minute).
id="1" tells me that its the first segment thats been drawn.

The <point> tag tells me that theres a new point within the current segment thats part of the overall drawing.
each attribute of the point plays an important role.

ID: tells me which point it is. (like a social security #)
X: tells the X position
Y: tells the Y position
time: tells when the point was created based on the progress of the video at the time the point was drawn.
type: tells whether the point is the first in the series or not. This lets me make sure the lines start and stop when they're supposed to.

Back to the segment tag. This is how i fix the drawing API lag issue. When the mouse is pressed it creates a new segment. When the mouse is released, it ends the segment and waits for a new one.

When the drawing is actually created, the mouse press creates a new Sprite to hold the points and drawing to come. During the playback its necessary to distinguish between which segments are together in order to allow the flash player to cache the drawing as a bitmap sprite and render it quickly without lag.

Drawing Optimization

The flash drawing API is very simple to use, but problems arise when drawing lots of things on the same plane. What happens, as far as i know, is the Flash Player tries to render the line that's been drawn continuously, whether the line is actually contiguous or not.

Since the lines that are drawn are actually vectors the player has to calculate each and every vector in the line, every single time a new vector is drawn.

Naturally, if you have to calculate and recalculate the entire line before you can make the line any longer its going to take quite a bit of processing power. This results in a VERY VERY slow drawing API as the drawing grows and becomes more and more complex.

=========

My Workaround

Actionscript 3 has 3 different container objects, 2 more than AS2 had. Movieclips (same as before), Sprites, and Shapes. Now, Movieclips are the same old usual movieclips, they're objects with a timeline that can be animated.

Sprites become plain old flat static images when they're rendered, so they use very little power to be rendered. Movieclips try to loop through the timeline when they're rendered, adding a little to the processing time. Shapes are the same as Sprites but for speeds sake they have disabled most interaction with shapes.

The way i get around the slowing down of the drawing api is to split up the drawing as its being drawn. For example, when the mouse is pressed down it automatically creates a new container object, when the mouse is moved the drawing api starts drawing inside of the new container object.

When the mouse is released it stops drawing and caches the container as a flat image. Once the mouse is pressed down again, it creates a new container and because of this, a new drawing. This relieves the API of having to recalculate the entire drawing each time a new line is created.

There are a few advantages from doing it this way. The biggest advantage being it speeds everything up. It also opens new doors for extra interaction. Drawing things then scaling, moving, rotating, and deleting segments of that drawing.


Right now im not entirely sure which container I'm going to use, but of the 3 types that I can choose from I will definitely not use MovieClip and will probably use Shape if the interactive limitations don't get in the way.

Here's Keith Peters Input on the Shape vs Sprite debate: http://www.bit-101.com/blog/?p=899

Synthea Status

This is the first detailed status update.


1. Coming Soon =0)


1. Drawing Application
This is just using the flash drawing API. I experimented with some features that add a little spice to the api itself. It still needs to be optimized ALOT, at this point, the way the drawing app is setup, the flash player will use up too much RAM and cause it to render at a very low FPS if the drawing is too complex. (this shouldn't be too difficult to fix)

2. Video Player
Actionscript 3 is quite different from Actionscript 2. There are several ways to add video to a project through AS3. I've been experimenting with the different methods to see which method would be the best to use with Synthea. The classes being experimented with are: Video Class, Video Player Class, Video Error Class, Video Event Class, Video Progress Event Class, Video Scale Mode Class, and Video State Class.

One thing to note: the Video Player Class and the Video Class are very similar, but the Video Player Class seems to be the more powerful version. All are being used with NetConnection Class, and NetSream class to allow the videos to interact with a server interface.

3. XML
XML in AS3 is fun. As I write this, i'm actually taking a break from working on a new class that will handle recorded drawings. The way the drawings will work is...

As the video plays, it has a timestamp thats formatted like 00:00:00:00, Hh:Mm:Ss:MSms (hours:minutes:seconds:miliseconds). As the video plays, the timestamp increases like so; 00:00:00:01, 00:00:00:02 ... 00:01:12:65, etc...

When the Mouse is pressed down and moved simultaneously the timestamp is recorded in an xml object in flash (i may record it in an array, then add it to the xml object later, not sure yet). Anyhow, along with the timestamp, a few other attributes are stored. The X and Y position of the mouse, the Size, and the Color of the drawing.

The one big problem i've run into with this method is duplicate timestamps. If not programmed correctly, the application will simply skip over all the duplicate timestamps and will not redraw the points that it recorded.

My workaround for this is to modify the framerate of the flash SWF dyanmically to always match the framerate of the flv. When the video is paused, the framerate will continue on the same as it was before the video is paused. At this point all the points in the xml node will be nested within that node. Once the video is played again it will continue to add new xml nodes along with their properties.

Im sure there are some issues with this method, but they'll pop up later and I can worry about them then.


1. Integrate the drawing app into a video and record the drawing in xml.
2. Create a database to hold the xml and videonames. (im debating whether or not i should store the XML itself in the database or create an XML file and load it in during playback).


Theres much more and I'll post on it all as it comes to mind.

Synthea Development Blog

I've been working on the Synthea video editing application for almost 4 weeks now. Quite a bit of progress has been made on it so felt its time I share the development process with the world. I'm still in the very early stages at the moment where I am still in the process of building the individual parts that will be molded, cast, and refined into the final project.

To give a completion percentage right now i would give it a 10% completion status where this consists of progress made on the application during and after completion of the research required.

The status of research I would give a 70%. Because the research phase is arguably the most important in the development process I would consider 70% incredibly productive.

Again to reiterate, at this point in the process Completion percentages are below:
Development: 10%
Research: 70%
Overall: 15%