This release is a beta version, some real-world testing will be needed to ensure no bugs remain. However, I am pretty confident this version can be used in a production environment, because it is currently used in such a way on a daily basis where I work.

Source code and binaries can be downloaded at Github or Google Code.

Here is a summary of the most interesting features:

• In-place updating: in addition to writing new files with inserted metadata, FLVmeta can now also update an existing FLV file, in order to simplify workflows where several encoding steps take place.
• Exhaustive dumps: flvdump has been removed, as its functionality has been integrated and expanded into the main flvmeta executable.
  FLVmeta can now dump the onMetadata tag, the whole file, or any chosen metadata tag. A few interesting output formats are available. In addition to the old “debug” format, XML, JSON and YAML are now supported, allowing integration most software.
• File checking: FLVmeta can perform a large range of verifications on a FLV file, from the most begnin issues to fatal errors rendering the file impossible to play. A textual report will be printed, optionally as XML to allow further processing.
• File fixing: currently experimental, this is an option that will attempt to fix invalid FLV files when updating them.
• User documentation: there is a man page now, that explains in detail how every command and option works.

I have successfully implemented support for the CMake build system in addition to autotools, initially in order to facilitate Windows builds, and now as the primary build system.

I intend to get rid of autotools, for a few reasons. First, CMake exists on even more platforms that the autotools. Furthermore it is much easier to support than autoconf to implement non-trivial features. And finally, I now feel that the number of users used to the “./configure && make && make install” way of installing software on Unix derivatives is decreasing due to the availability of binary package installation systems, such as Aptitude or RPM. FLVmeta can already be found as a binary package on a number of platforms, including Debian and derivatives such as Ubuntu, or FreeBSD, NetBSD.

I also discovered a great documentation management tool named Pandoc. I needed a tool to author or maintain FLVmeta’s man page, because avoiding editing raw NRoff files seemed obvious.
At first I tried the Ronn tool, but I had trouble obtaining satisfying results, and its Markdown-derivated dialect was too far from the standard everyone seems to be used to.
Using Pandoc gives me a great level of control over the output formats, and is easy to use and integrate into my build chain. This is especially true when using CMake, as the man page can even be automatically compiled from Visual Studio, assuming the tool has been installed on your Windows box, and can be found via the PATH environment variable.

All in all, I am satisfied by the experience of developing open-source software, and I am happy to have created a tool that is useful to at least a handful of people. I know that the number of downloads is probably not a very representative figure, but there have been thousands of downloads since its first release in 2007 (approximatively 10889 as of 2012-05-13).
There have been some bug reports and patches that have been proposed (and accepted), and some people are willing to help me maintain the program, for example by providing Debian packaging.
I have learned a lot about video formats in general, FLV, AVC (H.264), and proper development in C which I had never really used in its pure form before, since some IT schools in France think it is a great idea to teach an hybrid form of C and C++ as first coding experience for many students.

So, until now, FLVmeta has indeed been developed in pure C. I first chose this language in order to achieve the greatest possible performance, especially because I needed to replace FLVtool2, which did not fit my needs (back then, I needed to update FLV files hundreds of megabytes large). I was not disappointed by the result I could obtain, very fast execution, and great control over the amount of memory used by the program. However, I am feeling kind of frustrated by the shortcomings of the languages, the more features I add.
The pure algorithmic parts of the code are fine as they are, it is mainly about handling structures and making choices depending on the contents of the files, lots of ifs and control structures.
What bothers me is the fact that when I need to use a non-trivial data structure (lists, maps…) there is no readily-available standard solutions, and I need to reinvent the wheel and write a container adapted to my needs, for example as seen in the amf.c file.
So I am pondering switching to C++. Many parts of the code will not be modified, however I’ll be able to make some parts more modular, especially AMF parsing, or the FLV reader/writer.
Do not expect heavy development soon, I first need to make sure the current iteration is stable enough.

The current todolist for FLVmeta has a few items:

• metadata insertion via XML command files: This feature would greatly benefit from a switch to C++, since I’ll need a robust XML parser and the ability to store the described commands as in-memory structures, which might be a little it complex.
• batch file processing: for the moment, FLVmeta can only handle one file for each invocation. The ability to work on a set of files would be a neat improvement, since a lot of video editing workflows that I have met involve working on a lot of files.
• FLV encryption support: I am not sure this recently added feature is widely implemented, but I need at least to detect whether a file is encrypted or not, since it must certainly have an impact on its playability on a variety of systems.
• improving command line processing: simply put, FLVmeta is a program that is invoked to run a specific command. However, right now, a command is specified using a command line switch, such as -C / –check. The more options I add, the more limited choices I have to find available letters for options. So basically, I want flvmeta to use the pattern of programs such as Git, where the program is just a driver for sub-commands specified right after the program name.
  For example, I want to have flvmeta update –no-last-second file.flv and for example install a script such as flvmeta-update that automatically invokes the update command.
  This would allow me to offer better online contextual help, like flvmeta help update, invocation that would print detailed help only for the update command.

Now that FLVmeta 1.1 has been formally released, I am going to shift my focus to other kinds of projects. I am currently learning how to program with QT, and I am certainly going to make some interesting cross-platform applications with it.
One project that would probably be nice to develop is flvmeta-gui, as the name indicates, a GUI for flvmeta. I need to conceptualize the interface of such a program, but it would certainly appeal to people allergic to the CLI, and I know a few of them, even though I disapprove their way of doing things .

I will finish this (again) big post by promising I am going to try to post more at Zenhaven.org. I love writing, I love writing about software development, and I hope some people will enjoy reading me and react by writing (non-spam) comments !
The few next posts will be about FLVmeta and a few use cases that might interest potential users.

Then came the DCVS.

Early discovery

I first learned about Mercurial. I was impressed by its great merging and branches abilities, more than the whole “distributed” concept, however I found it a bit tricky to install and make it work in Windows, especially with TortoiseHg, which was definitly not yet as good as its older brother, TortoiseSVN. So I didn’t look further, and stuck with SVN for the time being.

A few years later

I’m now working in an environment where rapid website release cycles mean that a lot of developments/commits make it directly to production, the production environment being directly based on the current trunk, probably a wrong approach, but most developers in the team don’t care about branches.
This means that if someone is working on a feature that hasn’t been completed and spans several commits on the trunk, any urgent bugfix will cause these features to go online, and inevitably cause bugs on the production website.

The obvious solution to this problem is feature branches, branches that only live for as long as the feature is not complete, and get reintegrated into the trunk when functional and tested.
However, in SVN, every branch is server-side, which means that the central repository might become really encumbered with trash in the /branches directory, unless each developer deletes such a branch when done with it. Guess what ? In practice, a lot of orphan branches remain, and noone cares about deleting them.
Of course, this has an interest because SVN also works as a code backup platform, so at the end of the day, each developer should put his code on the well-protected server, instead of keeping it on their potentially dead-the-next-day desktop computer.
Also consider the fact that developers can work from home as well as from the office, and potentially need their code from anywhere.

SVN has also another flaw, it’s really getting slow when working on large codebase, especially with binary files like images, quite common in websites.
This is caused by the fact that everything happens between the client and the server, even getting file or repository-wide history. Trying to work out a problematic trail of changes is basically impossible if the internet connection is down for some reason.

Here comes a new challenger

That’s why I got very interested when I tried Git, which I had been following since its introduction by Linus Torvald, after the infamous issue with Bitkeeper.
Git’s Windows support has evolved quite a bit since the early days, and I must say that TortoiseGit (a TortoiseSVN clone) has become a quite good piece of software, allowing people to more easily bypass the initial steep learning curve of the many many git commands and options, by using a simple yet potent and familiar UI.

Git solves practically every issue I have encountered so far, especially about branches. Feature branches are a core part of Git’s workflow, and they’re freaking fast.
It also makes working locally very practical, as apart from the initial cloning, and eventual pull/pushes, everything is done on the client side.
And at least branches and tags are properly implemented without feeling “half-baked”.

Now you might certainly ask me: “how can it be used in a server-is-the-backup environment if all is done in a distributed way ?”
Very simply of course. As long as a developer has access to a machine via SSH and that has git installed, saving your work is just a matter of creating a bare repository in your account, and add it locally as a remote into which you can push and pull at will, without ever needing to communicate with the “central” repository.

Goodbye, really ?

At work, no, we’re not quite ready for the switch yet, but it will come, eventually.

I’m talking about flvmeta, instead.

When I decided to host flvmeta publicly back in 2007, I decided to choose Google Code project hosting, because of its simplicity, and its support of SVN. It’s been a good choice, and has worked flawlessly since then, with an active community, bug reports, and a now established codebase (says ohloh.net).

These days, I’m working on the trunk, to complete the 1.1 release of the software. It’s much more advanced than the 1.0.x branch, and given its good stability, I encourage people to use it instead, even though some planned features are still being developed.
So there’s no formal release yet, and people are encouraged to use the latest trunk revision.

This approach gives me, the main developer, a moral obligation: the trunk must NOT be broken. I’ve been very tempted to use feature branches to work around that issue, and so I created a repository at Github based on a conversion from the flvmeta SVN.
I must admit that I like Github a lot, and so I decided to maintain both the SVN and Git repository in parallel, opting not to use git-svn facilities, just to compare Git to SVN for daily tasks.

After a while I’ve started to wonder about closing the google code project in favour to Github entirely, because this double-repository thing is just an annoyance, and using feature branches in the Git version adds the risk to desync it from the SVN repository, which could confuse users.

And then, yes, they did it, Google added support for Git in Google Code ! So I uploaded my local flvmeta repository to Google Code, converted the Wiki SVN to Git, and that’s all there was to it. Now Github and Google Code are both mirrors for my local repository, and keeping them in sync is just a matter of pushing correctly.

Last words

I’m now a happy gitter, I will not come back to SVN for flvmeta. I just need to fix a few things in the build scripts, because I relied on SVN revision numbers to tag the executables during compilation, if just to make support easier by having people telling me the exact revision of the program they’re using.
It will just be a matter of using git hashes instead, there’s a handy command that I can use for that: git rev-parse --short HEAD, or even git describe --always.

The only thing that I don’t like with Google’s support of Git is that they decided to stick to the most used and abused protocol ever, HTTP, instead of ssh: or git:, therefore forcing me to enter my password for every interaction with their servers, since TortoiseGit won’t let me save authentification data.
I really prefer how Github handles authentication, by using cryptographic key pairs, that can be revoked individually, so if a developer machine is ever compromised, risks can be mitigated.

Nothing to complain of with the speed of delivery, it took me no more than 10 minutes to order the new box, be notified of its availability, and install an Ubuntu Lucid distribution (signaled as beta).

As I was very curious about the performance of the famous Nano processor, I pitted my older Dedibox (the server powering this very website) against the new one using nbench, here are the results:

First, the original dedibox:

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :             450  :      11.54  :       3.79
STRING SORT         :          71.765  :      32.07  :       4.96
BITFIELD            :      2.5932e+08  :      44.48  :       9.29
FP EMULATION        :          88.285  :      42.36  :       9.78
FOURIER             :           13993  :      15.91  :       8.94
ASSIGNMENT          :          18.371  :      69.90  :      18.13
IDEA                :          3042.6  :      46.54  :      13.82
HUFFMAN             :          1113.4  :      30.87  :       9.86
NEURAL NET          :          21.175  :      34.02  :      14.31
LU DECOMPOSITION    :          762.56  :      39.50  :      28.53
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 35.427
FLOATING-POINT INDEX: 27.756
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : GenuineIntel Intel(R) Celeron(R) CPU          220  @ 1.20GHz 1200MHz
L2 Cache            : 512 KB
OS                  : Linux 2.6.24.2dedibox-r8-1-smp-x32
C compiler          : gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
libc                : libc-2.7.so
MEMORY INDEX        : 9.421
INTEGER INDEX       : 8.429
FLOATING-POINT INDEX: 15.394
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Now the new server:

BYTEmark* Native Mode Benchmark ver. 2 (10/95)
Index-split by Andrew D. Balsa (11/97)
Linux/Unix* port by Uwe F. Mayer (12/96,11/97)

TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          625.84  :      16.05  :       5.27
STRING SORT         :          72.811  :      32.53  :       5.04
BITFIELD            :      2.9437e+08  :      50.50  :      10.55
FP EMULATION        :            88.6  :      42.51  :       9.81
FOURIER             :          8060.1  :       9.17  :       5.15
ASSIGNMENT          :          13.373  :      50.89  :      13.20
IDEA                :            3123  :      47.77  :      14.18
HUFFMAN             :          1478.7  :      41.00  :      13.09
NEURAL NET          :          13.524  :      21.73  :       9.14
LU DECOMPOSITION    :           772.2  :      40.00  :      28.89
==========================ORIGINAL BYTEMARK RESULTS==========================
INTEGER INDEX       : 37.871
FLOATING-POINT INDEX: 19.972
Baseline (MSDOS*)   : Pentium* 90, 256 KB L2-cache, Watcom* compiler 10.0
==============================LINUX DATA BELOW===============================
CPU                 : CentaurHauls VIA Nano processor U2250 (1.6GHz Capable) 1600MHz
L2 Cache            : 1024 KB
OS                  : Linux 2.6.32-22-generic-pae
C compiler          : gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
libc                :
MEMORY INDEX        : 8.883
INTEGER INDEX       : 9.899
FLOATING-POINT INDEX: 11.077
Baseline (LINUX)    : AMD K6/233*, 512 KB L2-cache, gcc 2.7.2.3, libc-5.4.38
* Trademarks are property of their respective holder.

Snapshots can be downloaded on the Google Code project page at: http://code.google.com/p/flvmeta/downloads/list

A lot of new features have been added, here is an exhaustive list:

A few things are still missing for the next stable release, for example some proper user documentation, now that a complete set of command line options are available, as well as the implementation of the check command.

Everyone is welcome to test the software and report bugs on the issue tracker.

I’m also still pondering the benefits of using gettext in order to fully internationalize the output of the program. On one hand, there are many people who don’t speak english, and adding the potential to reach a wider audience is tempting. It is also a must-have in order to be featured in distros like Debian.

On the other hand, I still have issues with building flvmeta with gettext enabled. First of all the cmake/windows combination would be kind of experimental at best, and using autotools, I still need to fully understand which files must and must not be added to the repository.
There’s also the question of performance, since flvmeta is mostly used as an automated tool for update purpose. Dynamically loading strings in that setting is kind of pointless since speed and memory can be affected and there’s noone to ever read those messages anyways.

After version 1.1, which is transitional in the sense that it really represents the maturation of flvmeta into a real CLI tool, a 1.2 version will be in development.

Features expected for 1.2 include:

To add a quick follow-up to my previous post, I might add that… no, FLV is not quite dead yet, since H.264 streams in FLV containers are working quite well in the Flash Player, and major sites have adopted that combination.
I think we also might even expect support for On2 VP8 in FLV files before the end of the year. If Google really open-sources its code, alternative flash implementations will certainly jump the wagon before maybe even Adobe.
Of course, due to Adobe being a major On2 licensee, as well as thanks to the rather good relationship between them and Google, the official Flash Player might support this codec fast enough for it to become a viable alternative to the increasingly controversial H.264.

That can seem like a rather bold statement, but let’s face the truth. After Adobe’s recent publication of updated and open specifications for the FLV/F4V and SWF formats, that seems to be about the most logical conclusion.

If someone reads the aforementioned specification in detail, there’s one thing that will immediatly be noticed. The specification is not about FLV, but FLV and F4V.
What is F4V ? In a nutshell, it is a subset of the MPEG-4 standard destined to be a container for video, audio, and various metadata. Exactly what FLV is supposed to be too, you’ll tell me. That’s only partly true. FLV has a few limitations as a media container.

First of all, FLV fails as a streaming format. Ok, it’s widely used on major sites like Youtube, dailymotion, and many others, but it’s not true streaming, because all it does is playing an already existing file, not a stream generated on-the-fly, and playing continuously, like a TV channel.
Even for random access seeking, an encoder producing FLV files has to add some specific metadata, that tell the player where it can seek.
Some tricks are needed on the server side, like using specific modules for the web server (eg: mod_flv_streaming for Lighttpd), or using some scripting, in PHP or other server languages.
Unhappily, free encoders like ffmpeg or mencoder are unable to add those precious metadata while encoding the files.
That problem drove me to write the FLVMeta tool, that precisely does this, taking an existing FLV file to update with the needed metadata. To be fair, there are several existing tools doing the same thing, but none of them worked well enough for my taste

Another problem with the FLV format is that each frame belongs to its own “packet”. Every frame is accessible in a standalone way, even when a frame is not a keyframe, a frame containing enough information to be fully rendered, without relying on information from previous frames.
That means that a FLV file can only contain video data that can adapt to that layout. Basically, a FLV file can contain video data that belongs only to a hard-coded list of codec families, like Sorenson Spark and ON2 True Motion VP6. Unfortunately, Sorenson Spark, derived from H.263, is outdated and is not on par with the modern codecs in terms of image quality. On2 VP6 is much better as a codec, but is a proprietary format, and it’s known that On2 is not exactly friends with people who try to implement their format.

On the other hand, H.264 has been implemented as Free Software, like the encoder x264, even though the standard is encumbered by many patents that can be enforced in countries like the USA. H.264 has already superseded Divx as a highly compressed movie format, and is expected to be used more and more as a streaming format.

Therefore, realizing the limitations of the existing video formats, Adobe chose to implement H.264 in its Flash Player software.
However, as the FLV format is unable to contain H.264 video, Adobe pushed a variation of the MPEG-4 multimedia container format, and heralded it as its new champion, the F4V format !

This “new” format is able to contain any video and audio format, leaving more space for Adobe if they wish to widen their offer. It is also able to contain more diverse metadata, for instance in the recently introduced AMF3 format, as FLV was limited in being only able to contain AMF0 encoded data.

So, the most recent Flash Player (version 9.0.124.0) supports both H.264 playback as well as F4V. It’s only a matter of time before developers will upgrade their Flash authoring tools to versions able to output Flash9 files, using ActionScript 3. And then, will there be a reason to continue to support FLV ?
I’m pretty sure that the major players in the league will switch to H.264, to avoid paying license fees to On2, and to provide better High Definition contents.

So what will be the point of keeping on using FLV ? Pretty much none, in my opinion. Maybe the F4V format is somewhat harder to implement than FLV, but most of the existing encoding software will just have to adapt their already mature MPEG-4 code, making the F4V format even more well supported than FLV is today.

So yes, FLV is dead, long live F4V.

PS: Fear not for FLVMeta, I’ll continue supporting it and developing the latest 1.1 version, FLV will be around for a few months at least, but I will focus my efforts on developing my AMF0/3 parser/serializer libamf which should allow C and C++ code at least to fully support Adobe’s metadata.

Update: Adobe has since then added support for AVC (H.264) in the FLV format. It must be noted that there are some limitations, including the fact that only the first frame contains the necessary information to decode the whole stream, namely the Sequence Parameter Set NALU, so random access streaming via pseudo-streaming does not work properly.

For those who don’t know her, Marie is the singer extraordinaire of Katzenjammer Kabarett, a nice Electro/Gothic/Punk/Cabaret… errr well, an interesting band anyways

This is going to be Marie’s first solo concert, so she needs encouragements and a loooot of people screaming her name.

Oh I forgot, the entrance is free !

Links:
http://www.myspace.com/mariedarling

http://www.katzenjammer-kabarett.com/

http://www.myspace.com/aloysuke

Ok, ok, no digression for a first post. This one marks the opening of my first blog, and website, so I’ll talk a bit about me and blogs.

Why did I wait so long before doing my website ? It’s been like ten years I’ve been in the IT business, the Internet, new technologies, etc, and I have never found the motivation to invest time and energy into this. There are several explanations, I guess.

Maybe I’ve just been lazy, or I didn’t feel like finding a decent web host. I was a student for six years, and it’s true that I was pretty busy with my studies. I really prefered improving my programming skills rather than talking about those, and I couldn’t really afford a dedicated server.
I must also admit that I suck at web design (see http://noirotm.free.fr for an early example of what kind of web design I can come up with). I’m really happy that there are now very intuitive solutions such as WordPress, or Drupal, that have thousands of themes, and can be almost be used as-is, as publishing platforms.

Another explanation is that maybe I didn’t have anything to say. Yeah, a lot of things have already been said on the Internet, and I didn’t think I could contribute with my limited knowledge about stuff. And I don’t like talking about myself either

But fear not, after all those years, I now have my blog, and it will be as random as possible !