It’s been a long time… how have you been?

This is going to be another Python/Cinema 4D related post.

While python is great for quickly scripting workflow optimizing tools, prototype stuff or write generators and tags, is it suited for big, complex plugin development?

It is!

You get ‘nearly’ full access to the C++ API of Cinema 4D, don’t have to compile or worry much about platform dependencies, can easily find a ton of neat recipes and libraries out there…

So it’s not only possible to code full-fledged python plugins but it even has its advantages over classical C++ development.

Imagine you wrote a nifty script, decided to make it a simple plugin, and now want to take it a step further – add more features, a complex GUI with some dialogs etc…

If you are still working with the script editor by now, things will most likely start to get out of hand as you introduce more modules and classes, deal with more code and more functionality.

This is the point where you might want to look out for a little help.

We are going to set up an IDE (Integrated Development Environment), talk a bit about modules/packages and finally I will try to give some general hints and tips.

Coding Environment/IDE:

Prepare Cinema 4D:

You may skip this part, but I highly recommend it:

• Make a copy of your Cinema 4D installation – name it “Cinema 4D R12 Dev” for example.
• Get rid of all Cinema 4D plugins you don’t need for your development
• Place an empty text file named “c4d_debug.txt” into the root folder of your Cinema 4D copy.

This will slow Cinema 4D down a bit, but enables you to monitor memory leaks.

We will add some useful things on Cinema 4D-side later – for now let’s continue with the Eclipse setup…

Eclipse:

If you never used one, think of an IDE as your workshop, it’s a collection of tools as well as the place where you do and organize your coding work.

Eclipse is one of the most popular – its Java based, open source, free to use and most importantly has excellent support for python.

So let’s start by downloading and installing Eclipse Classic from http://www.eclipse.org/downloads/
If you don’t have the Java Runtime Environment installed jet, you need to do this first.

PyDev:

As said before, Eclipse has excellent python support – but not out of the box, so we are going to install PyDev.

PyDev will help you coding by adding syntax highlighting, code completion and much more.
Should you encounter any trouble with the following steps, please refer to the PyDev installation manual.

• After installation, startup Eclipse, click “Help”, and chose “Install New Software…”.
• Next click the “Add…” button, type in “PyDev” as name and “http://pydev.org/updates” as location – then confirm.
• Now enter “PyDev” in the “Work with:” field, you should see an entry “PyDev for Eclipse” - check it and click “Next”.

After confirming again and accepting the terms, it will download and install.

• Click “Windows” and open the “Preferences” dialog.
• Select PyDev and the child “Interpreter – Python”
• Click “New…”, input “Python Cinema 4D” as name and browse to your Cinema 4D installation.
• In “/resource/modules/python/res/Python.win64.framework/” select the python.exe.
• Confirm your selection and the paths PyDev wants to add to your syspath.

So much for the PyDev installation itself, now we have to associate the Cinema 4D Python plugin extension *.pyp with PyDev to enable syntax highlighting and code completion for those files as well:

• Open ‘Window’ within the file-menu and click ‘Preferences’
• Select ‘General’ -> ‘Content Types’ from the list on the left.
• In the ‘Content types:’ view on the right, unfold ‘Text’ and click ‘Python File’
• Hit the ‘Add…’ button of the ‘File associations:’ list on the lower right, type “*.pyp” into the popup and confirm.

Its helpful to add a file containing all symbols of the Cinema 4D Python API’s “c4d” package to the ‘site-packages folder to spice up the auto-completion – grab it here.

Place the file within “/resource/modules/python/res/Python.win64.framework/Lib/site-packages“

Note:

You will have to update this file if you want to keep up to date with future versions of Cinema4D, it can be easily printed to the console by executing:

for itm in dir(c4d):
    s = getattr(c4d, itm)
    print "%s = %s" % (itm, s)

Project setup

We take your IDE for a spin and create the first project:

• Click “File” – “New” and chose “PyDev Project”
• Input a Name for your Project and uncheck the “Use default” box below.
• Browse to your Cinema 4D installations plugin folder, create a new directory with your plugins name and select it.
• Select “2.6″ from the “Grammar Version”- dropdown and leave the Interpreter set to default.
• Last but not least uncheck the “Create default ‘src’ folder…” checkbox and hit “Finish”.

To setup the needed folder structure, header- resource- and string files for your plugin, please read the Cinema 4D Python documentation “Plugin structure” –  especially “Directory structure” – and take a look at the Python plugin examples within the documentation – or the experimental plugin from this post:

Just remember this part is vital, so do not rush through it – at least read the section mentioned above and take a good look at the SDK examples.

Only if you have built C++ plugins before and are already familiar with the structure, you may…

Hint:

You can also add your existing project files by just placing them within the plugin-directory we just created, or dragging them into the project-folder within Eclipse.

When organizing the file structure within your project, you are working with your file system – changes you do here are reflected on your harddisk.
This also works the other way round, move something on the file system and Eclipse will update those changes to your project – while not always instantly.

You can press F5 within Eclipse to update a folder and its subfolders or the contents of a file.

Don’t worry, if the project-view is out of sync with the file system – Eclipse will tell you.

A coders Cinema 4D:

We are all set with the IDE part – let’s tune the Cinema 4D installation a bit more to improve the workflow and finalize your coding environment.

If you didn’t add/import your existing code to the project we just created, write a simple hello world and fire up Cinema.

It’s a good idea to setup a developer friendly layout, I like adding the console window to the tabs on the right, next to the object manager – or even better to a side monitor.
Should you be writing a CommandData Plugin, consider opening the Command Manager and dragging its icon into the layout, so you can quickly re-open it.

You will be launching and closing Cinema 4D a lot during development – it’s helpful to have a strategy to quickly start and close it.
Place a link to Cinema 4D on your taskbar/dock, close it using ALT+F4/Command+Q or create a little python script containing “exit()” and add it to the layout/create a shortcut.

Finally, open your Plugin as well as everything you need to test its functionality and save the layout as startup layout.

If you have a test document that you need to check your plugin, press CTRL+E, click the “Open Preferences Folder…” button and place it as “default.c4d” within that folder so it will load on startup.

You long got the idea – try to prepare everything so you don’t have to spend time doing recurring tasks on every unit-test you are going to make during development.

Some additional tools:

It’s often helpful to keep an eye on memory consumption and or active threads so I always have the task manager/activity monitor in reach when coding.

And while not really necessary, it’s convenient to have a good, fast text editor with Python syntax highlighting so you are able to read a file or snippet without having to launch the IDE first.
I can recommend Notepad++ and TextMate or Emacs for this job.

Working with eclipse:

Eclipse in combination with PyDev is a mighty! tool – Aside from the obvious killer features – syntax highlighting and code completion, there is a ton of others.

Often you will discover some that will make you feel bad for not having used them right from the start, so I’ll try to point out the most useful, while I encourage you to read up on Eclipse and PyDev.

The Find/Replace Dialog

Press CTRL+F for all your search and replace needs – just listing it to make you aware of the direction-, selection- and regex support.
If you haven’t used regular expressions jet, you should directly head over to Wikipedia - you won’t regret it.

Even more powerful is the…

Search Dialog

CTRL+H opens the global or project wide search – here you can also replace over all files.

Just keep in mind that the “PyDev Search” tab does only look within *.py files while the “File Search” tab by default searches over a *.* pattern.

Compare feature and history

Just select two files, right click and chose “Compare with” -> “Each other”.

You can also compare a file with an earlier state of itself by choosing “Local history”.

This is another huge plus of using an Eclipse – don’t worry anymore about changing huge sections of code or commenting it – just go ahead and delete it – you can always return to an earlier state using the “History”- tab.

But use this with care, as it’s possible to lose the history by moving or re-creating a project and it’s not infinite.

Tabbed views

You can drag a tab to one side of the code view until a small arrow appears – the tab will then snap to that side, splitting your view so you can work with multiple source files at the same time.

This is very useful when working with the header-, resource- or string files of your project as you can code on the left and copy ‘n paste ID’s from the right for example.

Coding python for Cinema 4D:

We have setup the coding environment – how about doing some programming now?

This will be a loose collection of tips that are all more or less Cinema 4D related, but mostly  good practice in general.

Modules and namespace:

First – this is OOP (Object Oriented Programming) and your project’s structure should reflect that.

In general, classes and modules are a good thing, it’s a far more common fault to have less abstraction than too much.

Packages/Modules are also the key to keep your code organized, manageable – AND reusable.

So if you are going to write something that you might want to use again later, consider spending the extra time to make it a bit more user/coder-friendly and build a class or module.

When working with python and modules you have to keep a close eye on naming and namespace, especially in context with the Cinema 4D SDK.

Be creative and use unique module names – this will keep things easier when you are using other modules – and you will – at least the ones from the Cinema 4D API.
Also be careful when importing from modules – it’s rarely a good idea to just import everything (from foo import *) – instead carefully select the members you really need and give everyone its own import line rather than stacking them with commas in one.

It’s often better to just import the module into your namespace and reference the members via module.foo – you don’t pollute your namespace and it’s more readable.

In any case you should avoid the API’s module AND member names like the plaque.

For example if you extend the GeDialog Class and have a method “def close(self):” – but make the slight mistake to call it as “Close()”, you might end up with a similar behavior but wonder for minutes while your dialog just closes without saving it’s settings first (or whatever “close()“ should have done).

I know for some this is basic stuff – many have their own patterns for this and that’s ok!

Whatever you do – be consistent and aware of your namespace.

Symbols:

As you need to keep track of the enums within the c4d_symbols.h, it is best to define every integer by yourself and not auto-enumerate them.

Then mirror those symbols to a *.py file called “ids.py” and be careful to update both should you change them or introduce new ones.

Instead of hardcoding ID’s, you should always use the symbol within the c4d-package.
Should you ever need to identify a symbol by it’s integer value (for example if you want to check for a certain flag that is returned) – this little script will help you out:

import c4d;
from c4d import gui; 

def main():
    wanted = gui.RenameDialog("ID to search for...")
    outp = ""

    for itm in dir(c4d):
        s = getattr(c4d, itm)
        if s == int(wanted):
            outp += "%s = %s\n" % (itm, s)

    gui.MessageDialog("Sorry - none found..." if outp == "" else outp)

if __name__=='__main__':
    main()

API calls / performance:

Don’t get me wrong – the Python API is blazing fast – so fast you often can’t tell the difference to C++ – but still, API calls take time.
When working with thousands of objects, this quickly adds up no matter how fast each individual call is.

Try to limit those calls in sections of your code, where you really need performance – look for ways to keep states within your own plugin logic rather than querying them form Cinema 4D whenever possible.

If you are looking for bottlenecks, be sure to check out cProfile – it’s great.

Prototyping:

Using an IDE doesn’t forbid you to use Cinema 4D’s script editor and console to quickly whip up and test something – you can even write and directly execute a simple test script within eclipse if you are working on parts of your plugin that don’t need the Cinema SDK (some kind of parser for example) and take advantage of the PyDev debugger.

Debugging:

The lack of a real debugger can be compensated by using the PyDev remote debugger, pythons inspect module and the logging package.

Remember that there is a limit to the amount of characters the Cinema 4D console can output for a single print command.

Also take a look at sys.exc_info()

Garbage Collection:

When introducing circular references you should use weak references to give the garbage collector a hand.

Compilation:

How about shipping your plugin as precompiled byte code?

This will make Cinema 4D start up faster – as the compilation of your plugin can be skipped, obfuse your source and still be platform interoperable.

Just copy your plugin, open the console/terminal and execute “python.exe -mcompileall C:\PathToMyPlugin” and delete all *.py files afterwards.

Note that this might not work in some cases, so be sure to test your plugins functionality again.

Platform dependency:

While Python is platform-independent, there are still some things to keep in mind.

Don’t use backslashes in paths but slashes – they are understood by windows AND mac OS.

You can also use platform.system() to determine on what OS your plugin runs on and import modules after a conditional check – for example:

if platform.system().lower() == "windows":

    import winsound
    winsound.PlaySound(os.path.join(os.path.dirname(__file__), "../res/snd", "DingSound.wav"),
winsound.SND_FILENAME)

GUI – Threading:

It’s the old story, if the GUI thread has to wait for your plugin, you lock up Cinema 4D – unacceptable for the user.

Doing time intensive stuff within your own threads is the simple solution to this, but keep in mind that you can’t have them call the GUI thread – instead use messages and the GeDialog’s SetTimer().

Resource reference:

Classes that use the “plugins” module need a reference to the plugins resource pointer.

This can be handled by importing those classes within the plugins *.pyp file and setting the class variable “__res__” or using a singleton (see below).

Singletons and invading Borgs:

This is something for the advanced developer – and kind of a hot topic too.

The use of globally accessible objects or ‘globals’ is often considered bad practice but also unavoidable in some cases (logger) and python even somewhat encourages it.

So if you know what you are doing and think about using a singleton, you might want to take a look at the “borg”-(anti)pattern – a monostate proxy with a twist.

class Borg:

__we_are_one = {}

    def __init__(self):       
    self.__dict__ = self.__we_are_one

Still alive

This was a lot of compressed stuff to digest, I know, but please do me the favor and don’t let any of this scare you off, it’s easier and more fun than you would expect.

I am sorry some topics where not covered in the depth they deserve -  it wasn’t easy to keep this a readable post rather than a book…

I hope this still was helpful to someone, if you have any questions, thoughts or would like me to write a more detailed post on a certain topic – let me know.

Humanity has presumably gazed at the stars long before the invention of speech and the concept of numbers…

But it took until the middle of the last millennia till we really understood our own place within the cosmic clockwork.

Checkout the Kopernikus Planetarium!

In 1704, George Graham and Thomas Tompion build a mechanical, heliocentric illustration of the solar system – a copy was made for the Earl of Orrery – hence the name “Orrery“.

Recently my fellow worker Philipp Althoff introduced me to those fascinating devices and we decided to build one.

As orrerys naturally show a scaled version of the solar system, high precision is vital for the device will quickly start to become inaccurate.

Philipp did an outstanding job measuring, modeling and texturing a model that also is completely parametric, so we could freely scale its polygon count for use in Unity 3D.

Klaus Hünig from Astromedia.de was so kind to support us with plans and data of his “Kopernikus-Planetarium“, a beautiful orrery construction kit.

While I did my best to research and calculate the exact orbits, directions and pivots, Mr. Hünig pointed out my errors and helped to refine the model.

Conclusion

The result is a 5MB heavy, animated Unity 3D version of the Kopernikus orrery by Astromedia.de.
Orbits and scales are accurate, the time scale able.

Finally we also produced a screensaver installer (WIN) and Philipp did the finishing touch on the browser version by composing the celestial background music.

Hope you find it relaxing.

There are those popular structures we all know and love because they simply look fascinating from every angle – like the Sierpinski triangle, the Pythagoras tree or the Menger sponge.
Generative geometry is one more proof that math is beautiful no matter how basic.

Recently I stumbled upon a new pattern that is also phenomenally simple but produces rather complex shapes – the “Binary Kite”.

I first saw it in Mikael Hvidtfeldt Christensen’s ‘Structure Synth‘ – an ingenious tool for exploration of generative 3D structures.

Basically it’s a 3D array of randomly black/white colored cubes that slightly scale(linear) and rotate(noise) per iteration on one axis.
What makes it look so interesting is that each items transformation matrix is subordinated to its predecessor on that axis.

Initially, I played around with the kite in Unity 3D but quickly discovered that it can´t be animated at high iterations.


So I fired up C4D and wrote a simple Py4D generator.

The results

I did end up with this gorgeous picture of a 30*30*500 cubes kite (5400000 polygons) (fullsize):

Here is a closeup from the same viewpoint (fullsize):

Last but not least I rendered this short animation of a 5*5*150 kite growing:

Want to render your own kite with even more cubes? Grab the scene file here.

Update: The file has been updated to work with the R12 version of Cinema 4D.

Hint:
Remember to uncheck “Optimize cache” in the Python Generator´s properties if you want to animate the parameters.

Conclusion

While Structure Synth is awesome, Cinema 4D and Py4D are a great way to visualize generated geometry too as Cinema’s render instances enable you to work with millions of objects.


Cheers

Surely you have already heard of Conway‘s cellular automaton – the Game of Life.

For those who haven’t, it’s a simulation of cells interacting with their neighbors by following a simple rule set on an infinite, two dimensional grid.

So let’s add a dimension -> view the game of life in 3D!

The Game of Life has become famous not only in theoretical computer science, for it is Turing Complete, generates the most interesting patterns and is easy to implement.
In flash it can be done in ‘one line’ of code by means of a convolution filter.

So since 1970 the Game of Life has been explored by almost every programmer, it’s no surprise its “Glider” pattern became the official “Hacker Emblem” in 2003.

Customarily it’s visualized by a 2D representation of the board/grid – however, it can be applied to 1D and 3D space as well.

While I find 3D versions/rule sets to be interesting too, there is a much simpler way to lift the game into three dimensions that also generates more beautiful patterns:
Simply plotting the (visible) game grid´s state over time on the third axis perfectly visualizes the process and generates amazing shapes… or at least very interesting greebles

Why Unity?

The only issue is the amount of cells to draw – visualizing a small 10*10 field over 100 generations already means handling up to 10k cubes or sprites.

And I had an even higher amount in mind, so flash wouldn’t get me anywhere this time.

First I thought about building a standalone app, then using Processing or Py4D…

There it hit me – Unity! As long as OpenGL isn’t native to the web, Unity 3D is definitely the best way to deliver intensive 3D games/apps via the browser.

As Unity offers C# as scripting language I immediately feel at home. Unity 3D reviewed in one word: Awesome!

The draw call issue

But there is always a stumbling block hidden somewhere.

Since I am generating a lot of cubes (cells) on every frame, unity has to do a huge amount of draw calls – one for each object.
The frame rate on my machine (Q6600, GF8600) dropped right into the basement.

The simple solution to this issue was to generate the complete generation of cubes and to fuse them to one single mesh using Mesh.CombineMeshes.
Normally this is done on startup – for the exact purpose of reducing draw calls – and I thought it could also be used at runtime for this scenario.

As objects are limited to 65k vertices, it wasn’t possible to merge the complete scene into one mesh, but that might have quickly become too time intensive as well.

So finally I was able to animate/display more than a million vertices.

Conclusion

The app now draws a 50*50 grid over a time of 200 generations – press space to reset the game and seed a new field of life cells.

Admittedly, it takes a lot of resources – a graphic adapter with at least 128MB is essential as the simulation might end up with 1.5 million vertices.

Hope you get to see some interesting patterns – have fun!

I haven´t written about Py4D for quite a while now…

In the meantime, Maxon has first announced a cooperation with – and now the acquisition of Sebastian Rath´s praised python implementation for Cinema4D.

So finally – Cinema4D meets Python!

Let´s celebrate with some cool experiments!

If you are new to this blog or Py4D – a short introduction:

Py4D enhances C4D with a standard Python interpreter that provides access to the C4D SDK and allows to easily code platform indepentent scripts and plugins that control or extend C4D.

Basicly the same as COFFEE, C4D´s own scripting language, but Py4D has two major advantages:

• It follows the C++ SDK more closely and implements it further than COFFEE does.
• It is a easy to pickup, widely known language with a huge community and tons of existing code and librarys.

While C4D is a mighty tool in itself, this powerful scripting interface enables the user to work with external data, automate processes or build his own tools.

Being more the C++ type when it comes to plugin development and C4D, I enjoy being able to solve small problems just by hacking a few lines of code into the script manager.
I also prototype most of my plugins in Py4D before I start developing in C++ – it´s just more convenient.
The step of compiling for every test takes much more time in the long run.

So here´s the meat:

Four simple examples how to have a good time with Py4D and some particles…

Everyone requires C4D R11.5, the Thinking Particles module and the actual version of Py4D.

Update: All files have been updated to work with the R12 version of Cinema 4D.

Py4D has been through some changes since the open beta started, so I took the time to update the scripts I published in the boids, and strange attractor posts and included them here.

Click the images for a preview.

1. Stars4Py4D

This is a Py4D port of the Sagan experiment I did in flash <- check out this link, it´s fun

A good and simple first step with particles - position vectors get parsed from a CVS database and from a galaxy of suns… “My God. It’s full of stars!”

Hint: this example might be the best to start with, as it includes comments – the others don´t, because they use pretty much the same pattern.

Just download the package, load the scene and change the path to the CSV file within the “file” user data field of the Py4D Tag.

2. Phy4D

A very simple adaptation of a 2D “physic engine” – works only with spheres.
The direction of gravitation is animated to make this a little more interesting.

Have a look at the preview video and scene file.

3. Boids

This is simply a revival of the Py4D Boids post – same beast, but updated to the Py4D release version.
The script creates a flocking boids swarm, that behaves according to the rules outlined by Craig Reynolds.

Checkout this video from Sebastian and get the scenefile.

4. Strange Attractor

Again just the script from the original strange attractor post, but ported to the final SDK.
View the final rendering and the original post – you can use the Strange Attractor Finder to locate your very own, chaotic attractor!

Here´s the scene file.

Conclusion

Python and Cinema4D are a great team and Py4D is THE alternative to COFFEE.

I hope this gives everybody who is interested in Py4D and controlling particles, a small head start, some inspiration or at least something fun to play with


Thanks to Sebastian for suggestions, support – and of course Py4D.

Cheers!

In his TV series “Cosmos”, Carl Sagan travels the universe in a “ship of the imagination”.

Uninfluenced by the laws of physics, “perfect as a snowflake, organic as a dandelion seed”, this vessel carries him between galaxies, stars, and planets.

And now you can cruise the cosmos too!



Recently I watched cosmos again and remembered how that sequence fascinated me as kid.
The sheer scale of the distances involved blew my mind. It still does today.

The concept has been revisited by many popular movies and media (Man in Black, The Simpsons…) and is always the same – the camera enters the universe, our galaxy, solar system and finally stops on earth – sometimes entering the microcosm, like the DNA of a cell.

Years ago I experimented with something like a pyramidial image to archive this effect in flash, unfortunately the result didn´t create the feeling of depth that I had hoped for.

But flash has advanced in those years… as I wrote before, in the strange attractor post, Ralph Hauwert and Joa Ebert have published some astonishing code that renders nearly half a million particles in real-time!

First I decided that I would work with real data – otherwise this would become just another “random stars / noise flying at you” screensaver.

But after some research I learned that the actual distance of nearby stars (called “stellar parallax”) is not always known, so that most models and databases project stars onto the inside of a virtual sphere.

Finally I found the HYG database – “a subset of the data in three major catalogs: the Hipparcos Catalog,the Yale Bright Star Catalog (5th Edition), and the Gliese Catalog of Nearby Stars (3rd Edition)”.
It contains about 120.000 stars, many including their parallaxes and conveniently cartesian coordinates, so I saved the step of calculating xyz from right ascension, declination and distance – like I had to with other databases I tested.

The CSV file got stripped of all additional data aside from the positions with a simple parser – that got it down from 20 to 4MB – still too big for my taste.

So I added the Fzip library to Joa´s AS3 only version of Ralph´s experiment, parsed the position vectors, scaled them and got my first view on our galaxy – breathtaking.

Scaling, controls and a simple clipping where swiftly implemented and everything was set for the voyage to the stars…

Tribute – “A Glorious Dawn” displaced

While watching the result, an idea struck my mind:
As the z-axis movement of the camera or the zooming into the szene is done by simply scaling the particle positions, its obviously possible to change their parameters in realtime…

I remembered my displacement experiment with papervision and quickly added a color parameter to the particle class, loaded an flv video and compiled this:

“A Glorious Dawn” live rendered to a displaced point cloud

Conclusion

It´s possible to handle an awful lot of particles with flash, even more if haxe/alchemy – inlining gets involved.
As the displacement experiment shows, the dataset can even be dynamic!

And hey – these are real stars, so this is science!

Hint: turn up your gamma adjustment.

Smart PRS is a Cinema4D tag that allows to record position, rotation and scale of one or multiple objects to key frames.

This is especially useful for all, who want to bake Thinking Particel setups and don’t own the MOCCA module.

Take a look at the demonstration Video.



Recently I tried to use the AfterEffects export Option from Cinema4D to pass a simple particle animation to AE.

Unfortunately this didn´t work out – the issue is that all animation capable export formats/plugins rely on key frames, and objects animated via Xpresso/Thinking Particles are most times not even present in the timeline as they get generated as “Particle Geometry”.

While this can be easily fixed by creating them manually and linking the particle positions to them, it´s still impossible to export that animation without key frames.

Cinema4D offers the option to “bake” certain motions to key frames, but not from simulations.
Actually there is a plug-in called “Cappuccino” within the MOCCA module that allows this, but if you are not into character animation, chances are you don´t have it handy.


SmartPRS makes those animations export-and editable by recording them as key frames while the simulation is running.
After that, the Xpresso setup can be removed from the scene and the animation edited or exported.

The plug-in even allows to capture the position of an object that gets moved in real-time by the mouse.

In the series GITS:SAC, a hacker called “the laughing man” defaces himself and other persons by digitally placing an animated logo over their face.

Now he is back and defaces the entire internet!

Look for yourself and check if your brain has already been hacked!

This experiment is really just an excuse to play around with hair cascades.

In 2001, Michael Jones and Paul Viola published their object detection framework, that introduced the integral image and cascade classifiers combined with a learning algorithm to achieve fast detection of facial features.

This method is still the quasi standard in face detection and an implementation of their paper is available within the Open Computer Vision library.

That code and even the whole library has been ported to flash and there are many great experiments out there – one by mr.doob even featuring the laughing man.
So this has already been done and explored with flash, but after my multiple object detection adventure I just had to do something “interesting” with the Viola-Jones detection.

However the idea to analyze and manipulate every image on any page is problematic.

Initially I was tempted to try this with JavaScript, even if I had some concerns regarding the performance – it should have been possible with modern browsers/engines … but sadly it isn’t.

The issue is the browser sandbox.

Sure, you can replace images within an html document on the same domain, but you can’t just load a page from a foreign domain into a frame and mess around with its contents. So in essence it´s the same cross domain policy that we know from the flash runtime and thus the solution to this is also the same: pull the desired content to your domain through a proxy.

Doing this with just streams or images is fairly simple and can be coded within a few lines, but fetching a complete page is a different story.
I did end up with a proxy that does this quite well but it struggles with redirects, certain JavaScripts and naturally flash, Silverlight or other plugin related content.

Still I find it quite reassuring to know that defacing a website isn´t an easy task.
After all, this topic is quite close to phishing and other black-hat stuff…

The detection part, on the other hand was quite simple – the actual OpenCV source was built quickly and works like a charm. Intel and other people where kind enough to contribute some of their research to the library – in form of xml lists with capable classifiers.

Because the server has to download first and then to compute all the images at once, you will notice a delay when loading a page with many images or with huge ones, although, the detection runs very fast.
Also I have it check every image twice – once with a frontal and then with a portrait classifier list.
The detection rate is quite satisfying – somewhat around 60-70% I would guess.

It would be possible to further improve that rate if one could adjust it to a fixed environment.

Conclusion

Sandboxes and cross domain policies are a good thing even if they are known to cause headaches to developers.
They protect us from frauds and make sure that others have to ask permission before they can use foreign content within their web apps.

Yet, given some time and motivation, they can be bypassed. So, always keep an open eye…

As for face detection: this isn´t just great fun to play around with but also has many interesting applications, spanning from camera tracking over to privacy (Google uses it to blur out faces in street view or to filter content for the image search) to security (face recognition).

I recommend OpenCV to anybody who wants to fool around with face/object detection or argumented reality
- and there´s much more…

About three decades ago, pixels where as big as toasters – and mostly green.

Today we have millions of them to deal with, and the terms anti-/aliasing are part of our daily live and every time an image is scaled, it gets interpolated to keep those bad “stairs” away from our eyes.

But what if the source image is aliased in the first place?

Checkout the demo of SmartAA!



While upsaling is the most common cause of aliasing, there are images that are naturally aliased – for example images from a 3D render engine.

Of course there are many ways to antialias cg images when generating them – for example by rendering to a higher resolution than targeted and downscaling the result (FSAA)- or by rendering and merging four slightly offset variations.

While this is as computational intensive as it sounds, modern GPU´s can handle this task rather easily as they have their buffers and hardwired magic to do so, but still it’s not uncommon to come along aliased images in the world of computer graphics.

So is all this material “useless” or can an image that has never seen any antialiasing, or has been upscaled with the nearest neighbour method, be reconstructed?

In fact all that modern imaging software does when rescaling an image is reconstructing its pixels to fit the new dimensions – most times with a bicubic filter.

SmartAA is a bicubic BC-spline reconstruction filter for PixelBender.

Its 3×3 kernel gathers all neighboring pixels of the one to reconstruct and weights them with the Mitchell-Netravali(german link) filter.

In essence, this means that each pixel is modified by the average color of its surrounding friends.
While “average” being the wrong term here as the influence of each pixel is determined by the spline curve of the Mitchell-Netravali filter.

This eliminates those hard, “stair-like” artifacts of the aliasing by visually “softening” or unsharping the image.

However this is not simply a destructive blurring effect but a reconstructive analysis of the source “signal”.

There are many known filters that do this hard job – it´s not easy to reconstruct information that was partly lost (or never there in the first place).

Thus all of them suffer of the same issues – the result always shows some artifacts.
Mostly it’s a visible unsharping, anisotropy or the so-called “ringing“.

In general bicubic reconstruction methods create the best results.

Among the various possible filter methods the windowed Sinc(Lanczos)- and Mitchell-Netravali filter are chosen the most as their output is considered to be the best mix of the inevitable unsharping, anisotropy and ringing.

Just for the record: there are “better” solutions, the so called “adaptive” or “smart” filters that try separate methods on edges, fills and other structures.
They are particularly useful when working with sequences of moving images (video) but again slow to compute.

SmartAA on the other hand is a quite simple implementation of the M.-N. filter and runs blazingly fast when run on the GPU – producing about 70 redraws per sec on a 1024*768 image.

Unfortunately it´s not nearly as fast on the CPU.
If you have a rather slow machine, you will have seen the demo struggling to keep a 24fps framerate, so its just! that bit too slow to to antialias the pixelated output of a flash based C64 emulator.

That would have been cool… would love to see something like that.

I kept it flash compatible anyway – still there might be applications.

Its a matter of personal reception, but to me the M.-N. filter looks better on videos as the inbuild “smooth”-method.
Anyway – flash video has been criticized enaugh for high cpu usage so lets not make things worse

Also: like any reconstruction filter it won´t work in any given scenario as there are some natural and some specific limitations.

So what can SmartAA do and what not? Here is a feature/limitation list:

Features:

- Works with Photoshop, After Effects, Flash
- Supports alpha channel
- Is real-time

Limitations:

- Does no up/down scaling – just reconstruction of anti aliasing
- Is scenario specific – for example text will often look strange
- works only with artifacts in the same pixel space.

Let me elaborate on the last point:

As SmartAA takes only neighbouring pixels into account it isn´t able to reconstruct aliasing effects that span over greater distances than 2 pixels.
For example if an image exists of 4*4 pixel blocks of the same color, those blocks would get rounded on the edges but will still be visible as rounded blocks.

But it works ideal on sources like aliased output from a render engine, or images that have been upscaled x2 with the nearest neighbour method.


Conclusion:

Aliasing should be fought before it ends up within an image as there are more sophisticated methods to keep it from emerging than to reconstruct it after the damage is done.

But sometimes it´s inevitable and if you are looking for a simple but effective way to get rid of that ugly blocks within an image or video – SmartAA will do the trick.


Download:

Grab the Shader here and use it with Photoshop (Pixel Bender Plugin needed), After Effects or Flash!


Hint:

Try to add a sharpen filter or better an unsharpen mask to the result to counter the blur effect.
Don´t forget to use the width/hight properties – SmartAA will work without but there will be artifacts on the upper- and lower right corner.

I hope this is helpful to anyone out there and would love to hear some comments or ideas.

Cheers!

The flame wars are on!

CEOs, CTOs, platform evangelists and users are fighting the war of the worlds out there.
It´s HTML5 vs. Flash, open vs. closed, H264 vs. OGG, desktop vs. mobile, truth vs. lie.

Lets first calm down with this nice audio visualization before we dive in…



Since Steve´s presentation of the iPad and recent statements from him and Apple it´s clear that they are not going to support flash on their mobile devices.

This has triggered an ongoing debate on Flash, HTML5, browser plugins and web technology in general.

Unfortunately the whole discussion is rather uncivilized as there are a lot of people involved who make their living from those technologies and standards – and even more who use them on a daily basis.

So much has been written on this topic during the last two weeks… much was impulsive, exaggerated or plain wrong misinformation while few statements showed greater insight or were just watered down by irony and harsh language.





So I am not going to take part in this discussion with a detailed view on the technology involved or why one standard is superior to another etc.. sorry, but there will be no rant

Why?

Because this has already been done beautifully by others and because – it doesn´t matter.

What really matters is to get things done! To create content and to deliver it to the widest possible range of audience.

As a developer I don´t have the luxury of choosing my tools based on personal favor. I have to work with what’s best suited to achive the desired result – be it HTML, Flash or Silverlight or whatever may come along.

As a user however I have that luxury – I vote with every click and my wallet for and against technology that I consider to be worth my time or money.

So there is no point for these two groups of people to lash out against each other or certain technologies.
Of course this doesn´t apply to CEOs, CTOs or platform evangelists – it´s their job to influence the aforesaid groups on their opinion.

So what will all this mean for the web? Almost nothing.

The web has always been based on open standards while innovation was often driven by the “big boys” like IBM, MS, Apple and many others.
Finally this innovation gets adapted to open standards – as well as the economy assimilates ideas from the open source world.

And all this is healthy!

Sure – like in nature, it sometimes seems cruel and unforgiving, but whatever survives in the end may stay… just to be replaced by its successor if it doesn´t evolve.

And in the end everybody in this constantly changing ecosystem wins:

Companies get to sell their products and to rant about their competitors, developers get jobs instead of finally being replaced by generators, and users get more and more for their money.

So I am glad we live in a world with more than one operating system, programming language, browser, or pair of socks to choose from.

It´s the freedom “to choose” that makes the web/live so exciting.

Cheers!



PS: About the audio visualization experiment in the opener:

I wanted to do something with Five3D for a long time now and well – here it is.
You can load your own mp3´s via the right click menu.

Of course everybody remembers the scene from 2001 when Dave Bowman removes HAL´s memory banks.

While being shut down HAL sings the song ‘daisy’ and his voice continuously pitches down while his ‘mind fades’.

With this little test app you can sing like HAL!

…provided you installed the Flash Player 10.1 beta and have a microphone handy.

Yes 10.1 – as the new year is coming, so is the next subversion step of the flash player, bringing among other features video playback and p2p/streaming enhancements.
The beta2 has been released at the 17th December and brought also the long-awaited debug version.

But aside from the global error handling, the new feature I was looking forward to the most is the access to binary data coming from the microphone.

This will allow to develop a bunch of pretty interesting applications that make use of voice input – like voice recognition for example.

This little demo records data from the mic and plays it with an adjustable speed offset, pitching the sound.

You can download the source here.

Note: to be able to compile the project you need to download the updated “playerglobal.swc“.

Relighting is an interesting method that is often used in video post production pipelines.

Given the vector of its surface normal, it is possible to calculate the brightness of each pixel by a pseudo 3D lightning system.

Checkout the demo of SmartReLight!



As the ‘object space’ – or ‘material’ normal’s have to be known, relighting is mostly used within the post process of CG generated images or videos.

It’s a convenient way to archive great lighting and gives the artist most control as he can tune it in real-time without having to re- render the footage for every adjustment.

As you can see from this example, the complete lighting is done by the shader using only a color pass and the object normal’s:

This implementation is fairly simple as it doesn´t support reflections and refractions like Normality , but its fast and does the job – and what better way to get through the last cold days of the year than playing half an hour with Pixel Bender?

It’s also possible to relight already lighted sources to some extend, by not using the color- , but an already phong- shaded pass:

You can download SmartReLight here and the source of the demo including the shader here.

Compiling PB shaders at runtime

Last but not least a short but very interesting story:

Recently I was asked if it was possible to compile a PB shader at runtime.

Given it some thought – why not?

And as it turned out the hell of a chap Nicolas Canasse already took care of that matter.
There is also already an AS3 port by James Ward.

All the best for 2010 !

Ever dreamed of having a chat with you own personal AI?

Well, we are still a bit behind Arthur C. Clarke´s dream of a self-aware AI like the HAL9000 but with AIML, Festival and Flash you can build your own talking chatbot in no time.

Have a chat with HAL first.

The Brain room:

So we are going to build ourselves the poor man’s version of a “Heuristically programmed ALgorithmic Computer” and of course we start with its brain.

The Program E – AIML Manager will function as the brain room and during the setup we install a set of data crystals (aka an AIML set).
Program E runs on PHP and parses and stores the actual “brain” within a mysql database.

So create a new database, install Program E to “mydir/programe” and upload the “talk2me.php” from the project package at the end of this article into “programe/src”.

Then upload and parse an AIML brain file into the db – call the new bot “HAL”.

The speech module:

We will use the Festival Text To Speech package to make HAL talk.
This needs a Unix based system and the installation is quite simple – refer to this tutorial if you encounter problems.

Festival comes with a “text2wave” script and a suitable default language.

We will also need the LAME mp3 encoder in order to convert those created wav files to a transferable mp3 because speed is critical.

Within the package you will find some PHP files including a “config.php”.
Upload and chmod them and modify the config to fit your servers directory structure.

The eye:

Now we setup an interface to Program E and our TTS system – HAL´s famous “eye”.

Open the “src/hal9000.fla” and the “src/Main.as” and change the variable “path” in line 31.

Upload the recompiled “index.swf” along with the HTML and JS file and have a first small talk with your chatbot.


Download the project here !



Note: This is the poor – AND the lazy man´s way to build a talking chatbot.
This could be implemented in many more elegant ways – it´s even possible to have a bidirectional voice communication with HAL by using an rmtp server and a speach recognition sdk.

Screen Space Ambient Occlusion or SSAO is a method to compute an approximated AO pass in real-time!

This is done by using the depth information of a 3D object or scene that has been stored from the Z-(depth) buffer of the renderer.

Go ahead and try the interactive demo or see a video example!

The technique was introduced 2007 at Siggraph by Crytek Studios and is becoming an often used feature within next generation live-3d engines or cg/video post applications.

After playing with the ray traced inside-out AO and having to realize that it´s not quite doable with Pixel Blender, I turned to SSAO and quickly got some nice results.

In short it´s all about importance sampling.

SmartSSAO looks up the depth information of 12 surrounding samples and checks if they get occluded to average the AO value.

The disadvantages lie in the nature of SSAO being “screen spaced” which means that it can´t take objects from the off into account, and the result has to be blurred to reduce the inevitable noise.
Also there is no ray bouncing, so classic AO is visually superior – but hey… who can argue with “real-time”?

View another video of the buffer and the solo SSAO pass without/with blurring.
Please note that those strange looking anomalies on the buddahs arms originate from the badly ranged zmap I produced in a hurry.


To sum it all up:

SSAO makes it possible to have AO in places where it normally isn´t possible – e.g. real-time engines (yes – even for Flash/Papervision), or within a video post process.

All that’s needed is the Zbuffer of a 3D rendering.

You can download the Shader here or get it packed with an AFX example.


Hint: As said before, the SSAO pass needs to be blurred in order to clear out the emerging noise – but don´t overdo it, as the blurring will “bleed” the result over to unwanted areas.
This could be targeted by a special SSAO blur filter that would again have to take the depth map into account.


Click here to see this screenshot as video

Update:

I feel like I should adress the X,Y-offset params of the filter which can be used to tweak the “shadow direction”.
“Shadow direction” in quotes as (SS)AO is all about indirect lightning so this is a real “tweak” or an unrealistic param – which I thought might come in handy in some cases.

I was asked if the shader was performant enaugh for flash – and even if it looks a bit ugly codewise as I had to get rid of a loop – it surely is.

So I updated the package and made this small interactive flex demo :

Be sure to try unchecking “color SSAO?” and “show SSAO?” to get an impression.

You can grab the source here.

Update 2:

The filter now features an additional sampling radius parameter which controls the dispersion and allows to adjust it to the desired level of detail – this should also make it more useable for papervision.

I hope this little experiment will be of use to somebody out there and would love to hear any suggestions.

Since some years the idea of harnessing the power of modern graphic cards for 2D/3D applications has become more and more popular.

Today we find GPU support in many CG applications like Photoshop or After Effects.

But first the fun part: Watch a video raytraced with PB and AFX!

The chaos group is even working on a GPU based real time version of Vray – a widely used modern GI raytracer.

In fact GPU shading is often more performant plus it quasi ‘adds’ an unused/inactive processor.

One could argue that Pixel Bender doesn´t quite fit the term as its mostly run in software mode (on the CPU) – for
example when executing a PB shader in Flash or rendering it with AFX… for now.

But PB is still young and the prospect of being able to run it within multiple adobe platforms (Flash,PS,AFX) is very compelling.

When I saw Tom Beddard´s PB port of Keenan Crane’s genius julia set tracer, I remembered that I had planned to get into PB GPU raytracing once I got around Mike Welsh´s PB raytracer.

So I spend some time with it, created a scene similar to ‘The Eye’ and added viewport controls as well as anti aliasing.

The result can be viewed here.

Rendertime was 3:10 on a Q6600 for 4700 frames.

And here is the modified shader.

I also fiddled around with adding an inside-out AO (ambient occlusion) pass but wasn´t able to get satisfying results.
It even almost fried my GPU in the process so in conclusion it’s doable but sadly not practical.