Update: ARRIRAW files from the ALEXA 35 can now be processed into HDE files in both the ARRI HDE Transcoder application, as well as traditional workflows utilising third-party data-management software in conjunction with CODEX Device Manager. More information on CODEX Device Manager is available here: codex.online/products/device-manager
This presentation was so helpful! Thank you! Love all these Tech talks. I'm still wrapping my head around this. So if I'm hearing correctly: HDE is completely Lossless, when it is decoded it is a bit for bit perfect match to the original file. It’s less data with no loss in image quality but a file reduction size. It’s not compression, not a new file formate, it’s a reordering scheme where the data is rearranged in order to reduce its size. So for example: If I piled my van with all my equipment. Just tossed it in there without organizing, and there’s so much equipment it reaches the roof. But then someone came over and said “hey let me organize this for you” and started shifting things around according to a pattern and organized it so thoroughly that I actually had extra space now! All of the equipment is still in the back of the van but there’s more room. Is that what the HDE encoder is doing with the RAW files? All the data is still there it's just rearranged in a space saving manner?
Hey there, thanks for your comment. There are several parts to it, but the most common analogy we use is that HDE works in a similar way to creating a zip file of a folder of photos. When you zip the folder, the filesize shrinks, and when you unzip it, the photos are the same as they were before hand. There are many factors that can go into what happens during the "zipping" process, but some of that is that any identical data across lots of photos needs to only be stored once, instead of multiple times. The efficiency of your friend's tetris skills can also play a part ;)
At 14:10 I have a question. What is the verify step doing in the process ENCODE during CLONE? Isn't it directory generate HDE files to drives and it has no other backups? So we only have one HDE backup, which is the destination. As I thought "verify" needs to generate the checksum of source and destination files. So without a HDE source, how can it be verified?
I get that it's probably a bit of a different beast with the massive data rates of an Alexa35 but I still think it's worth mentioning that lossless RAW compression with all metadata intact (compression ratios between 1.5:1 to 2.8:1) has been around since 2015 for cinemaDNG-files through slimRAW, a software created by one single guy.
I'm also struggling with this... The ARRIRAW files work perfectly, but the HDE won't import into the project. Is there an Adobe Extension I should download?
Hey there, at the moment Adobe does not support HDE files, but we believe that is in their roadmap. The full list of software tools that do support HDE can be found here: www.arri.com/en/learn-help/learn-help-camera-system/alexa-35-workflows - and then in the Postproduction Compatibility section.
Hi there - it would be possible for third party companies, such as Pomfort, to license the HDE compression and incorporate similar features. However, the design of HDE is such that it is not intended for users to offload regular ARRIRAW as well as HDE ARRIRAW. HDE is not just visually the same as an uncompressed ARRIRAW file, HDE uses a mathematically lossless data reduction. The format is used by virtually every ARRIRAW production that is supported by ARRI Rental, it is supported by most postproduction software and there is a SMTPE RDD that allows anyone to read the bitstream, which makes it the ideal format for archiving. Keeping a backup of the uncomressed ARRIRAW material alongside the ARRIRAW HDE material would greatly increase the data storage requirements instead of reducing them.
i am a DIT curewntly using this incredible camera and trying to figure out the most efficient and time management way to offload the files to two destinations as HDE. Post only wants the HDE files. I am using silverstack...any suggestions will be greatly appreciated
i just want to tranfer the arri raw files from the card to the transfer drives but in HDE only. As well, I am required to use silverstack to generate the reports etc
When you are offloading the ARRIRAW data with the offload manager and have arri transcode simultaneously writing the HDE version, do you end up with both versions on your target drive? If so that seems to defeat the purpose
Hey Cory, in this example Oliver is offloading the original ARRIRAW files to one drive, and then encoding HDE files to a second drive using the watch folder functionality.
@@ARRIChannel thank you for that. i am a DIT curewntly using this incredible camera and trying to figure out the most efficient and time management way to offload the files to two destinations as HDE. Post only wants the HDE files. I am using silverstack...any suggestions will be greatly appreciated
Thank you Oliver for this very clear and efficient presentation. Regarding the watch folder, how does the encoding work when the file is not completely silverstack-copied in the first place? In the video, it looks like the encoding is simultaneous and does not need the file copying to be fully achieved to start encoding. In this case, you will need to have at one moment enough room to get the original arriraw mxf files and additionally the arriraw HDE encoded file. Therefore you will need to have an 50% extension of room on your initial drive. This seems to be very safe but how can we have so much room? Moreover, the whole purpose of the HDE is to reduce the size of the file although keeping the quality of the native file. Thus you will need to delete the initial files after verifying it. I guess you can produce the silverstack xxHash MHL files to the insurance company when required but what is the point if you can not deliver the native files? At the end of the talk, you have mentioned the hash strings that could be used for further comparisons. Any tools to be suggested for performing that? Great work in any case, cheers
Hi Brice, the original file itself must be a completely written and closed file. This means the process for the High Density Encoding will not starte as long as the original file is not completely written. We agree that this will result in higher demand for storage you will need. To minimize the storage it would also be possible to do the High Density Transcode directly from the original compact drive. The "high density encoding" itself is a very safe process. If the whole on-set to post-production workflow has been tested successfully it should be OK to delete the original files. The xxHash (ASC MHL) or MD5 file that is created by the ARRIRAW HDE Transcoder during the job contains the hash for encoded HDE files. This xxHash (ASC MHL) or MD5 side car file should be passed forward to the post-production facility or should be used for further steps of post-production for verification. More information about ASC MHL can be found here on github: github.com/ascmitc/mhl At time of writting here it looks like the ASC MHL is in an early stage of development and I’m sure the will be offical updates in copy and verify tools e.g. Pomfort Silverstack for a better support of ASC MHL.
Hi there, thanks for letting us know, here is the link to download the software: www.arri.com/en/learn-help/learn-help-camera-system/tools/arriraw-hde-transcoder
Hey there, the idea is not necessarily to convert back to regular ARRIRAW as most finishing tools already support HDE. It's possible to transcode from HDE ARRIRAW files to OpenEXR through the ARRI Reference Tool if you need to send files to a VFX company, for example. If you need to convert HDE files to regular ARRIRAW files for a particular purpose we'd like to hear from you: digitalworkflow@arri.de
Update: ARRIRAW files from the ALEXA 35 can now be processed into HDE files in both the ARRI HDE Transcoder application, as well as traditional workflows utilising third-party data-management software in conjunction with CODEX Device Manager. More information on CODEX Device Manager is available here: codex.online/products/device-manager
This presentation was so helpful! Thank you! Love all these Tech talks. I'm still wrapping my head around this. So if I'm hearing correctly: HDE is completely Lossless, when it is decoded it is a bit for bit perfect match to the original file. It’s less data with no loss in image quality but a file reduction size. It’s not compression, not a new file formate, it’s a reordering scheme where the data is rearranged in order to reduce its size. So for example: If I piled my van with all my equipment. Just tossed it in there without organizing, and there’s so much equipment it reaches the roof. But then someone came over and said “hey let me organize this for you” and started shifting things around according to a pattern and organized it so thoroughly that I actually had extra space now! All of the equipment is still in the back of the van but there’s more room. Is that what the HDE encoder is doing with the RAW files? All the data is still there it's just rearranged in a space saving manner?
Hey there, thanks for your comment. There are several parts to it, but the most common analogy we use is that HDE works in a similar way to creating a zip file of a folder of photos. When you zip the folder, the filesize shrinks, and when you unzip it, the photos are the same as they were before hand. There are many factors that can go into what happens during the "zipping" process, but some of that is that any identical data across lots of photos needs to only be stored once, instead of multiple times. The efficiency of your friend's tetris skills can also play a part ;)
Thank you so much for the reply! Helps a ton! Can’t believe I’m just now implementing this into my workflow. Works amazing. Thank you very much!
At 14:10 I have a question. What is the verify step doing in the process ENCODE during CLONE? Isn't it directory generate HDE files to drives and it has no other backups? So we only have one HDE backup, which is the destination. As I thought "verify" needs to generate the checksum of source and destination files. So without a HDE source, how can it be verified?
I get that it's probably a bit of a different beast with the massive data rates of an Alexa35 but I still think it's worth mentioning that lossless RAW compression with all metadata intact (compression ratios between 1.5:1 to 2.8:1) has been around since 2015 for cinemaDNG-files through slimRAW, a software created by one single guy.
How do I import HDE file in Premiere Pro? Do I need to convert to Proxy?
I'm also struggling with this... The ARRIRAW files work perfectly, but the HDE won't import into the project. Is there an Adobe Extension I should download?
Hey there, at the moment Adobe does not support HDE files, but we believe that is in their roadmap. The full list of software tools that do support HDE can be found here: www.arri.com/en/learn-help/learn-help-camera-system/alexa-35-workflows - and then in the Postproduction Compatibility section.
why not add this feature directly to offload manager, as a copy running in second thread with compression option to other destination?
Hi there - it would be possible for third party companies, such as Pomfort, to license the HDE compression and incorporate similar features. However, the design of HDE is such that it is not intended for users to offload regular ARRIRAW as well as HDE ARRIRAW. HDE is not just visually the same as an uncompressed ARRIRAW file, HDE uses a mathematically lossless data reduction. The format is used by virtually every ARRIRAW production that is supported by ARRI Rental, it is supported by most postproduction software and there is a SMTPE RDD that allows anyone to read the bitstream, which makes it the ideal format for archiving. Keeping a backup of the uncomressed ARRIRAW material alongside the ARRIRAW HDE material would greatly increase the data storage requirements instead of reducing them.
i am a DIT curewntly using this incredible camera and trying to figure out the most efficient and time management way to offload the files to two destinations as HDE. Post only wants the HDE files. I am using silverstack...any suggestions will be greatly appreciated
i just want to tranfer the arri raw files from the card to the transfer drives but in HDE only. As well, I am required to use silverstack to generate the reports etc
When you are offloading the ARRIRAW data with the offload manager and have arri transcode simultaneously writing the HDE version, do you end up with both versions on your target drive? If so that seems to defeat the purpose
Hey Cory, in this example Oliver is offloading the original ARRIRAW files to one drive, and then encoding HDE files to a second drive using the watch folder functionality.
@@ARRIChannel thank you for that. i am a DIT curewntly using this incredible camera and trying to figure out the most efficient and time management way to offload the files to two destinations as HDE. Post only wants the HDE files. I am using silverstack...any suggestions will be greatly appreciated
Thank you Oliver for this very clear and efficient presentation.
Regarding the watch folder, how does the encoding work when the file is not completely silverstack-copied in the first place? In the video, it looks like the encoding is simultaneous and does not need the file copying to be fully achieved to start encoding.
In this case, you will need to have at one moment enough room to get the original arriraw mxf files and additionally the arriraw HDE encoded file. Therefore you will need to have an 50% extension of room on your initial drive. This seems to be very safe but how can we have so much room?
Moreover, the whole purpose of the HDE is to reduce the size of the file although keeping the quality of the native file. Thus you will need to delete the initial files after verifying it. I guess you can produce the silverstack xxHash MHL files to the insurance company when required but what is the point if you can not deliver the native files?
At the end of the talk, you have mentioned the hash strings that could be used for further comparisons. Any tools to be suggested for performing that?
Great work in any case, cheers
Hi Brice, the original file itself must be a completely written and closed file.
This means the process for the High Density Encoding will not starte as long as the original file is not completely written.
We agree that this will result in higher demand for storage you will need.
To minimize the storage it would also be possible to do the High Density Transcode directly from the original compact drive.
The "high density encoding" itself is a very safe process. If the whole on-set to post-production workflow has been tested successfully it should be OK to delete the original files.
The xxHash (ASC MHL) or MD5 file that is created by the ARRIRAW HDE Transcoder during the job contains the hash for encoded HDE files.
This xxHash (ASC MHL) or MD5 side car file should be passed forward to the post-production facility or should be used for further steps of post-production for verification.
More information about ASC MHL can be found here on github: github.com/ascmitc/mhl
At time of writting here it looks like the ASC MHL is in an early stage of development and I’m sure the will be offical updates in copy and verify tools e.g. Pomfort Silverstack for a better support of ASC MHL.
Th link on your page to download this software isnt working.
Hi there, thanks for letting us know, here is the link to download the software: www.arri.com/en/learn-help/learn-help-camera-system/tools/arriraw-hde-transcoder
but how do I convert HDE back?
Hey there, the idea is not necessarily to convert back to regular ARRIRAW as most finishing tools already support HDE. It's possible to transcode from HDE ARRIRAW files to OpenEXR through the ARRI Reference Tool if you need to send files to a VFX company, for example. If you need to convert HDE files to regular ARRIRAW files for a particular purpose we'd like to hear from you: digitalworkflow@arri.de