AI in the film industry
Artificial intelligence has now arrived in many areas of audio and image processing, even editing is already partially supported with AI. What differentiates the AI boom of the 90s from today is simply the performance of the computers that are available to us today. As early as the 90s, people believed that AI would change and make everything easier, but the limitations imposed by the computing speed were simply too great.
Today we find AI in numerous tools and applications, and more and more in the field of sound and image processing, and thus also in the film workflow. The developers of the NLEs are very quick and we can look forward to automatic masking functions, but also face recognition and AI-supported Sky-Replacemnt.
AI-based tools are now springing up like mushrooms. Frameworks such as TensorFlow or PyTorch make it much easier to get started with AI-based development!
On The Edge
One of the biggest challenges will definitely be the development of new applications "on the edge", which means that the functions can run locally on dedicated devices. Here, the industry has been developing extremely powerful new processors and chips for several years, which have already implemented the necessary AI - for example for image processing - such as the Kneron Edge AI chips. There will be big leaps in image processing applications, especially in the automotive and medical sectors.
Film and post-production will also benefit from this. Thanks to such chips in the cameras, it will already be possible to carry out functions that we still dream of today. The trend towards "on the edge" solutions also has another reason. The cloud-based solutions are far too traffic-intensive, especially in critical applications such as image recognition in the automotive industry or in medical applications, cloud solutions will be used less and less in some cases.
Completely new tools will be created for AI-based and supported image processing, which will not only use NN, but will also use much more complex structures and models, while Midjourney is child's play.
Automation through AI?
In my opinion, there will soon be automated color adjustments that hardly require manual adjustment. In a few years, color correction will no longer be part of a colorist's job, yes, the entire image processing industry will be shaken up and reorganized, this process has long since begun. There will be new players and the current stakeholders are well advised not to "oversleep" this trend.
The AI-based and automated color adjustment in scenes is already possible today in good quality, although manual post-processing is still required here. The speed of image processing will also increase significantly thanks to AI, so that manual image processing will gradually be pushed aside for cost reasons.
AI will not replace the role of the colorist in the foreseeable future, but rather complement it. But it's only a matter of time before tools that can fully assume the role of a colorist see the light of day. Anyway, I'm surprised that certain functions haven't been implemented yet, because purely technologically, functions such as automatic and safe skin tone correction and retouching are already possible today, I'm already looking forward to the next DaVinci Resolve version!
The area of responsibility of a colorist will change drastically in the next few years, even though I'm often alone with this statement at the moment. I've seen too many new technologies come and go as part of research projects, I'm quite sure that AI in image processing will achieve a high degree of automation.
What matters in the future?
This brings me to a very different but related point. Even today, anyone who wants to can create very good images with affordable cameras, it's no longer an art. The technology is getting cheaper and the differences in quality are getting smaller. Of course there will always be gold standards like the Arri cameras, but in order to produce good images, a compact camera with a log format and a possible 15+ stop dynamic range is often sufficient. In short, the differences in the images are no longer determined so much by technical differences.
In the future, a clear demarcation will increasingly only be possible via content. I think that a good story and quality content will be king again and more and more in demand, I would really appreciate it!
This is already reflected in the industry. In conversations with cinematographers from larger production houses, I experience more and more often that the topics that are discussed in the industry are hardly ever technology-based, this was completely different just a few years ago.
Today more good content is in demand again, good storytelling is an issue again. Anyone who now thinks that I put the AI above everything else is wrong. Analogue technologies are just as dear and expensive to me, it's not for nothing that I still like analogue photography, which of course also has something to do with my background. The preference for a "good picture" is more important than the love for new technologies. And of course we should all take a critical look at AI technology in general and not blindly entrust everything to it.
One thing is for sure though. AI will transform the film industry from within. And Blackmagick will most likely be at the forefront of NLEs with DaVinci Resolve when it comes to AI-based image and sound processing. Of course, someone still has to operate and feed all these tools, but that is increasingly becoming a question of interfaces. There will still be colorists in the future, but they will have to take on more and more other functions and tasks. I can totally understand if people demonize me after this blog post, but you will see it, there is no way around it.
AI integration in the film workflow
AI is already finding its way into our workflows. At the moment the results are not that mature and some important aspects are left out. This is due to the tools themselves (e.g. ChatGPT) as well as the necessary depth of learning.
Simply put, it's a matter of "training" the AI. The more and the better you feed the AI, the better it gets. What would happen, for example, if Walt Disney took all his scripts, "threw them" to an AI and then, given further parameters, made them write scripts based on their own specifications? You don't have to be an expert to realize that it's all about the quality of the training, but it's already conceivable and feasible today. Stories and poems are no longer a problem. The question remains whether it will be "touched up" manually afterwards, which would certainly be the case with Walt Disney. But maybe it is even conceivable that this - let's call it "refinement" - is saved in perspective and completely taken over by the AI.
Believable dialogue and really nuanced characters still need human help, and I like that. After all, writers have experience and know how to write and develop dialogue that captivates audiences. The AI cannot (yet) look inside the heads of the authors and producers, but even that is already possible today to some extent. Likewise, the relationships between the credible personalities would still be open, the AI is not yet able to really develop relationships based on layers of the individual actors, but I'm pretty sure it will come.
AI in pre-production
Of course, AI plays a big role in pre-production, that's for sure. The numerous tools that we already use today in pre-production alone are being developed more and more in the direction of AI. Tools that analyze the costs of crew, locations and equipment and create optimal production plans and budgets are quite conceivable in the medium future. It is "only" about data, but every computer scientist knows that you can generate added value and new knowledge from such data.
If you only consider the research for potential actors for the individual roles, it quickly becomes clear that AI can already be of great service here. If you then factor in nominations for awards, previous awards and audience demographics, the AI would be in a very good position to determine the ideal casts for the roles with a high level of accuracy.
Filming locations can also be found and evaluated even faster and more specifically with the help of AI. Again, this would be a question of data quality.
And of course it is also a question of costs whether you want to continue doing everything manually or at least let an AI make the pre-selection and selection.
AI makes films
Now it's getting scary! AI makes movies? Yes, of course, this is already happening today, at least in places! The AI is already heavily involved. The AI can already control cameras and drones, do zooming, panning, tilting and focus dragging, although for the most part only as a support.
A world in which the AI tracks people and cleans up shake using customized stabilization in camera and "on the fly" is quite conceivable. The question remains as to what else she can take on. Let's think about the analysis of a scene and its contrast ratio, coupled with the light setup, it would be able to set the desired contrast ratio and illuminate the image exactly to 18% gray. Some people get sick at such visions, but we will not be able to avert this development.
The AI could even take over the right lens selection and make real-time decisions about the composition and the best camera angles. Well, are you already afraid?
Cinematography is one of the most complex arts and we all have a great deal of respect for it and actually adore cinematographers of great successful productions, I don't exclude myself in places. We are well advised not to demonize a development that is difficult to accept, but to recognize it and the associated potential and to deal with these technologies and possibilities in good time.
Cinematography is one of the most inclusive crafts when it comes to human creativity - something we revere and adore. It would be a difficult pill to swallow if all of this was automated.
AI in post production
Another bitter pill is post-production. It is sacred to us and colorists, cutters and VFX artists are reluctant to hand over their work to an AI. But it is "manageable" and most likely controllable by an AI. Especially in editing and color grading, the development is already well advanced. Color management is already being partially taken over by AI and we read about new developments and breakthroughs every day. It won't be long before the first tools deliver really high-quality results.
Imagine controlling an AI with voice: a little more orange in the highlights, but leaving the Specualr highlights white, desaturating the shadows a little bit more and the sky more blue with a nuance of cyan please and a little more texture in the skin tones. Such a scenario could be possible in a few years. Then what will become of all the colorists?
It is also conceivable and not at all far-fetched that AI smooths faces, ideally adapts skin tones to the environment, retouches highlights and refreshes make-up, all while specifying a few reference images. Well, dear colorists, are you feeling warm already?
Let's go one step further. Transcriptions and dubbing are no longer art and could do without a speaker if the AI takes over. It is already possible to replace speech in high quality. For example, one can very well imagine a tool that analyzes film material and automatically selects the best shots and sequences. Then improve transitions and add other elements to improve the story. Even sound design could then be taken over by an AI tool.
Is there anything else?
Of course, the field of action of AI goes much further. Let's just think about the evaluation of viewing habits, audience monitoring and evaluation of trends, audience preferences and the intersection with demographic data, which help to optimize the distribution of a film. In some cases, this is already being implemented with the help of AI, and here too it is only a matter of time before tools will support us almost completely or can complete this task in a quality that "only" requires a subsequent selection, if at all.
From my point of view, there are two central questions that absolutely must be considered. Is the quality right and how much time is saved, because only then will it make economic sense to hand over tasks to the AI.
In my opinion, the AI will shake up the existing professional fields, at least in editing and color correction. It remains to be seen whether there will still be colorists in the modern sense in a few years. My prognosis in this regard is very clear. The area of responsibility of today's colorist will no longer exist in the foreseeable future, as will that of the cutter, with the exception of certain disciplines.
ConclusionAn important final question is: Do we want all that?
To stay with the example, this would mean that we would have a new episode of Walt Disney every month - for example in the form of a daytime soap. Walt Disney until the doctor comes!
In my opinion, the entire film industry will rely more and more on AI and use it as an aid, maybe even to test their own ideas. The ethical aspects of such a development are a completely different matter.
Nobody comes around anymore. The question remains as to how quickly the individual job descriptions will change. So far it has always been a social process with a gap of approx. 5 to 10 years until a completely new technology found broad acceptance and application. Will it be faster this time?