Avid HD Offline Workflow for Television

Are full HD workflows possible? Iron Man’s offline workflow was at HD resolution. All their material was shot 24fps which was a great start. Twenty Four is the necessary frame rate for DNxHD 36, which is a great quality compression for working in HD due to it’s small file size and easy processing for Avid. If Iron Man had to deliver at 30fps, DNxHD 36 wouldn’t have been available in Avid. Also, Iron Man had enough drive space to keep two copies of the media, the highest quality HD, and lowest quality HD.

I’m currently working on a televison show where we’re shooting with the Sony PDW700, the EX3, the 7D and 5D (XDCAM disks and tapeless media). So I asked if we could get a shitload of drive space. I had permission to go up to a 32 Terabyte Unity, an assistant editor’s dream for a television show! I danced around and sang the no digitizing song. Could it be possible to copy the XDCAM disks and tapeless media to the Unity, AMA everything, work at full quality on five edit bays, consolidate to a hard drive and send it to online at the end!? No.

1. I was told by multiple people not to use AMA in a shared environment. Everyone said AMA is for quick and easy projects, such as one editor on a local station. So, our media would need to be transcoded before editing could take place, causing a lot of duplicate media or deletion of the original source media.

2. We are shooting a lot of footage at 23.98fps as well as 29.97fps. So either the 23.98 footage needs to be converted to 29.97 HD, also causing duplicate media or deletion of the original source media, or, the editors would need to work with mixed frame rates in their timeline, causing lots of rendering and possible other issues such as lag and crashes.

3. Avid does not work well with the H.264 codec of the 7D and 5D, so that media would need to be transcoded for sure, regardless of frame rate. I know this from first-hand experience, it is very very slow at processing the codec.

4. God forbid the editors have problems actually working at a full HD resolution, such as DNxHD 220.

While 32 Terabytes is a lot of space, after mirroring the drives it goes down to 16 Terabytes, and I cannot safely say that having so much duplicate HD media would fit on it. And I cannot in good conscience delete the original media after it’s been transcoded, especially if it was shot at a different frame rate. And I can’t risk having editors work at different frame rates in their sequence… at an HD resolution. Oh, I forgot to mention one thing.

5. If a second season came along, you’d be fucked. Sequences couldn’t be recaptured if they were edited with transcoded tapeless material that was blown out. Any media from the first season that was tapeless couldn’t come back online without re-transcoding, and I give it a 99% chance that it won’t relink after the transcode, so every shot would have to be re-cut into sequences and media re-grouped and re-labeled.

So… we’re digitizing the XDCAMs in real-time at low resolution… laying off the tapeless material to tape and digitizing that as well… and going with a simple 8 Terabyte Unity Over the course of the show I will push and test Avid to see how much it can handle as far as using AMA in a shared environment, mixing resolutions, and working at DNxHD 220, to confirm or deny the advice I’ve been given, but mostly to understand the system’s capabilities better.

-Matt Rittorno

Sony EX3 & Avid Workflow

Hi Marty!

In a perfect world, we would shoot with the EX3, 30fps, and we’d edit on Mac’s with Avid 5, and the assistant editor computer would be the fastest motherfucker on the planet. I recommend the EX3 over the F800 because you don’t have to buy XDCAM stock, you don’t have to rent decks, upressing takes 1/10th the amount of time, and we probably don’t need a night assistant editor. All we need to buy are Raid drives, and we’ll have to convince the network to let us deliver on hard drives.

If we’re shooting 600 hours of footage, and the SxS cards are 32 GB’s / card, then that would be just under 12 Terabytes of Disk Space needed for storage. If this is the case, I would recommend a 16 Terabyte Unity, and 24 Terabytes worth of Raid storage for backing up the memory cards. There are different types of Raid, and I would recommend Raid-1. Raid-1 is very important because if one of the drives dies, we won’t lose our original footage! These cost about $800 / 8 Terabytes, so $2600 total. http://www.bhphotovideo.com/c/product/608031-REG/Western_Digital_WDA4NC80000N_8TB_ShareSpace_Network_Storage.html

I recommend we rent Mac computers, OS X version 10.6.3, with Avid 5.0.3.9. It’s pretty stable. Couple minor crashes here and there, but nothing that will slow down editorial. Avid 5 is a necessity for tapeless cameras, and a big time saver for the F800. The newer versions of Avid have a feature called AMA which allows you to read footage straight from the memory card into Avid, immediately.

The workflow I recommend would go as follows:
-After filling the memory cards, we copy them onto Raid-1 storage and wipe the cards clean.
-In Avid, we load the cards with the “Link to AMA Volume” feature, and transcode the media to a lower resolution, while keeping the 48khz audio.
-We can group, edit, and do outputs.
-When we go to online, we reload the AMA volumes. Then relink the low resolution material to the HD files, and consolidate to an external hard drive.

A major bottleneck with the new tapeless cameras and F800 is importing, exporting, rendering, consolidating and transcoding. So the faster the assistant editor computer goes, the faster we go. A 12-core mac computer with 2.93 GHZ processors and 64 Gigabytes of RAM would be the fastest. I recommend having Final Cut Pro X installed with Compressor and DVD Studio Pro and Motion, it gives us more options to save time. And Adobe Creative Suite, with After Effects and Photoshop… more options as well. I also recommend having a gigabit ethernet card installed in the computer, for use with the Raid-1 storage.

Another major issue is internet speed. The faster the better, to help with sending and downloading quicktimes and music and graphics and facebook.

-Matt Rittorno

Awesome Avid Quicktime Export Settings

The following settings generate approximately 6 MB’s for every 2 minutes of footage. So if you have a 24 minute television show, it should be less than 100 MB’s when it’s done exporting. It looks good and sounds good.

Avid Export Quicktime Movie Options…

If your show is 4×3, adjust the height / width to 486 x 365 and the aspect ratio to “Native Dimensions (486 x 365)”. This will create 8.5 MB’s for 2 minutes of footage.

Format Options…

Check “prepare for internet streaming” if you’re putting it onto a website.

Video Settings…

If you adjust the slider in the video settings to “Least”, it will generate 3 MB’s for 2 minutes of footage. So a 24 minute show would be less than 50 MB’s when done exporting.

Sound Settings…

I always export a small 2 minute chunk to test and make sure the settings are all right, and that everything looks and sounds good, before spending half an hour exporting a 30 minute chunk that might be messed up. It should export at approximately real-time (2 minutes of footage takes 2 minutes to export).

If I’m going to send the quicktimes to someone over the internet, I try to limit them to 100 megabytes. If you go higher than that, it might take awhile for the recipient to download the file. Gmail will send up to 15 Megabyte attachments. YouSendIt.com will send up to 50 Megabytes for free. Pay $10 and send up to 2 Gigabytes.

-Matt Rittorno

Digitizing in Avid

Digitizing. If you’re a producer, editor, post-supervisor, production assistant or wannabe, here’s how to do my job.

Go to “Tools” -> “Capture”. Make sure your deck or camera is hooked up properly, the tape is cued to the point you want it at, all deck & Avid settings are how you fancy, and hit the big fat red button with a white circle!

———————————

“…make sure your deck or camera is hooked up properly…”

Connect a 9-pin or firewire cable from the deck/camera to the computer, set the settings on the deck the way you fancy, and make sure Avid is getting audio, video & timecode (if you want all 3).

Here are some cables you might find useful:

———————————

“…make sure Avid is getting audio, video & timecode (if you want all 3)…”

If you want the tape’s timecode, make sure you have deck control and the timecode track enabled in the capture tool. If you don’t want the tape’s timecode, disable the timecode track or click the deck icon (toggle source icon) at the top of the tool so that it’s disabled (the picture above is disabled and will ignore the timecode on the source tape).

To get deck control, click the settings tab on the project window, double click deck preferences, and add a channel. If the deck is connected firewire then select “firewire” & “ohci”. If the deck is connected 9-pin then select “direct” & “keyserial1″. Click auto-configure and the channel should acknowledge the deck. If it doesn’t, make sure the deck is on, check the cables, try different channel options such as “direct” & “usa19h3d1p1.1″ or any other options until it finds the camera or deck. Click Apply!

If you want audio, select the audio input type in the capture tool, turn on the audio tracks you want to capture, and verify that you’re getting audio into the Avid (check levels in the audio tool, Cmd+1).

If you want video, select the video input type, turn on the video track, play the tape and make sure the video is coming thru.

———————————

“…make sure all deck & avid settings are how you fancy…”

Go to the Project Window -> Settings Tab -> Capture. Change the settings to your desire. I like to “capture across timecode breaks”, and I don’t “stop capture if a bad frame is detected”. My maximum capture time is “90 minutes”.

Download the deck or camera manual from the manufacturer’s website and go thru the menu and settings for that device. If using a deck, going composite out into a monitor (super I believe), will allow you to view a deck’s menu on a monitor. If using a 9-pin deck, you want to make sure the deck has reference from a blackburst generator, and that the “reference alarm” is turned on in the deck’s menu. If you want to upconvert or downconvert the video then this is the place to do it!

Are you digitizing 4×3? 16×9 full frame? With timecode burn? How many channels of audio are on the tape? I recommend going SDI out of a deck into the Avid if you have the right equipment, it allows for the video and audio to all travel through one cord at the highest quality possible. Firewire is fine if your tapes are HDV or Mini-DV.

Back in the Avid Capture Tool, fancy your settings there as well. What bin do you want to digitize to? What resolution? What drive? I prefer a 2 second pre-roll if I’ll be upresing with a 9-pin connection. I digitize at 10:1m for offline if I’ll be grouping, 15:1 if not, or 1:1 if space allows! What is your tape name? If you haven’t been asked already, specify the tape name now. For example, mine is “UB100913A01″

Capture some footage!

-Matt Rittorno

Ten Post-Production Tips

Here are ten ways to improve your life in post-production.

10. DVD Labels – Buy Memorex CD/DVD labels and download their free exPressit software, http://www.memorex.com/en-us/Support/exPressit-Label-Design-Studio/. The software makes it way too difficult to create straight text that is centered on the CD, and it’s only available for PC, but it’s free and should be respected. One workaround for centering text is to create a text box the exact length of the label and then center justify the text within the box.

9. Ripping Videos – Handbrake, http://handbrake.fr/, is an open source (free) software program for the Mac. It allows you to rip a DVD to the computer, something 99% of other DVD ripping programs make you pay for. KeepVid, http://www.Keepvid.com, is a free website for downloading YouTube videos to your computer.

8. Exporting – Exporting from Avid SAS takes 1/2 the time of the total duration. Exporting from Avid H.264 is real-time and is a more manageable file size.

7. Drama = Conflict & Movies = Magic – If there is no conflict, then there is no action and there is no movie. Supposedly the goal of a movie is to impress the audience, and should have a point.

6. UCLA Extensions – https://www.uclaextension.edu/r/default.aspx Open to the public, take classes at UCLA for about $500 / class. Art, Math, Science, Entertainment, Healthcare, Business, and more. Hundreds of classes available all year round.

5. QuicKeys Multigrouping – Skip assigning auxiliary timecode and subclipping timelines and have a program do it for you. It would take awhile to explain in great detail, but if anybody gets ambitious to try this, here are the necessary screenshots and steps. Leave a comment if you want help.

1. Setup QuicKeys with the macro as shown below. http://startly.com/products/quickeys/mac/4/ I can email you the macro if you’re cool and I’m nice.
2. Either change the macro or change your Avid keyboard so the keystrokes hit the right keys.
3. Create the timeline you want to group and add your add edits. Solo the track you want to subclip first.
4. Setup your bin as shown below with a mark in column and aux tc column.
5. Click on your timeline, go to the start of it, start the QuicKeys macro, enter the number of times you want the loop to run, and sit back.

I have to thank Travis for showing me this!

4. YouSendIt – https://www.yousendit.com/ Can email up to 100 MB files for free. Great for sending rough cuts to people.

3. Almonds and walnuts are great sources of vitamin E. The sun is almost the only source of Vitamin D. Kiwis are a ridiculous source of Vitamin C. Spinach and leafy vegetables are great for vitamin K. If we ate more foods with fiber and less calories, such as broccoli, bread, pears, apples, beans & peas, we would all be fuller and likely skinnier. Meat is the biggest source of cholesterol in our diets, and the best source of protein. Fish is great for omega-3 and should be eaten no more than once every 3 or 4 days. Most restaurants have a vegetarian option. All major fast food restaurants serve too much salt, sugar, meat and fat in almost every meal they serve. The recommended daily allowance of sugar is 0, but 40 mg is a reasonable limit. The RDA of sodium is approximately 1mg per calorie. The RDA of fat is approximately 65 grams per day. Strawberries, walnuts, and whole wheat bread are great for your skin. Blueberries are great for your brain. Mangoes are great for a lot of different vitamins.

2. Avid 5.0 can import XDCAM clips with low-res proxy video & high-res audio, therefore eliminating the need of digitizing ever again and eliminating a huge portion of my job description.

1. Nexus One – This is the Google phone released earlier this year. It is a HUGE resource of time and money saving thanks to it’s navigation feature and instant internet. I can weed out the bullshit in life and make smarter choices on what to buy, where to eat, and how to get it fast. I’m sure most smartphones are just as good, but who doesn’t like Google?

-Matt Rittorno

The 4-Hour Workweek

An American businessman took a vacation to a small coastal Mexican village on doctors orders. After a phone call from home that upset him, he walked out on the dock and saw a single fisherman in a small boat with several large yellow fin tuna. The American complimented the fisherman on the catch.

The American inquires, “How long did it take you to catch the fish?”

The Mexican answers, “Only a little while.”

“Why don’t you stay out longer and catch some more fish, returns the American.”

“I have enough for my family and to give to some friends,” the fisherman said as he unloaded the boat.

“Then what do you do with the rest of your time?”

He smiles and says, “I sleep late, fish a little, play with my children, take a siesta with my wife, and stroll the village in the evening while I sip wine and play guitar. I have a full and busy life, senor.”

The American laughs. “Sir, Im a Harvard M.B.A and can help you. You should spend more time fishing so that you can buy a bigger boat and hire a crew. Before you know it, you can buy more boats and increase the haul.”

He continues, “Instead of selling to a middleman, you could sell directly to customer, then you could expand the enterprise. The best part is that you could eventually sell the business stock as an IPO and become a millionaire.”

The fisherman asks, “How long will that take?”

The American replies, “15-20 years, 25 tops”

“Then What?”

“Then you would retire and move to a small coastal fishing village, where you would sleep late, fish a little, play with your kids, take a siesta with your wife, and stroll the village in the evenings while you sip wine and play guitar with your Amigos…”

———————————

“The 4-Hour Workweek” by Timothy Ferriss has given me a tremendous amount of tools and resources necessary for creating time and financial freedom in my life. I’m actively applying many of the book’s concepts to my life right now, and my family is planning for retirement this summer.

-Matt Rittorno

The Z7U, M35U & Avid

Popping an HDV into the HVR-1500, the deck violently spewed “Illegal Format” back at me and then proceeded to take a nap. A problem I’ve become familiar with the past couple months, if the Sony Z7U is set to shoot in a true progressive format (whatever that means) the HDV will not play in just any deck. Actually, the tape will only play in two devices; the camera itself, and the HVR-M35U deck.

Digitizing from the camera was not an option. Destroying the camera is a possible consequence, and the fact that we don’t have access to the camera doesn’t help either.

Me: “Can we tell the camera dudes to stop shooting in this weird format?”
Boss: “What format? Will it look different? They’re not going to do it if it looks different”
Me: “Otherwise, we have to rent the M35U to play the tapes”

So we have rented the expensive HVR-M35U, which comes with “all the bells and whistles”.

————

Sony Tech Support: “Hello this is Bruce”
Me: “I have some tapes shot on the Z7U, and to my understanding the M35U is the only deck that will play these tapes.”
STS: “Yes, it’s the latest HDV deck from Sony. It comes with all the bells and whistles.”
Me: “I use Avid for video editing, and when I hook up the deck with the firewire connection, it won’t capture across timecode breaks. I have to restart the deck every time there’s a timecode break.”
STS: “There’s a setting in Avid…”
Me: “It’s turned on…  I can capture across breaks whenever an HDV deck is hooked up with a 9-pin connection, such as the HVR-1500, but not with a firewire connection. Are there any other decks with a 9-pin connection that will play these HDV’s”
STS: “No…”

Avid Tech Support: “Hello this is Bill”
Me: “The deck….firewire….timecode breaks…yada yada yada”
ATS: “Try this, that, the other thing”
Me: “Doesn’t work”
ATS: “I’m looking at the notes for the deck to see if there are any problems with timecode breaks and I don’t see anything. The notes do say the deck has problems batch capturing, but nothing other than that”
Me: “How are we going to upres these tapes later?”
ATS: “Hmm”

I decide to capture the HDV’s anyway because they need to get in the system asap. Excruciating babysitting of the latest greatest HDV deck has me ecstatic. After week 1, we decide we should not be capturing 4×3 letterbox due to possible complications later when the shows go to online, so all the media needs to be recaptured. I proceed to smash my head in with a hammer.

————

January 29th, 2010

Night Assistant Editor: “I tried to batch capture one of the HDV’s from January 14th but the deck can’t seem to figure out how to batch capture.  Unfortunately I blew away the media for that tape before testing it.”

Editor: “WHY’S MY FUCKING MEDIA OFFLINE?”
Boss: “WHY’S HIS FUCKING MEDIA OFFLINE?”

Bells and whistles.

We are now dubbing all HDV’s shot with the Z7U to XDCAM.

-Matt Rittorno

Avid Matchframe Doesn’t Work

Hi Mikel,

There is 1 major concern that is going to upset a lot of the editors and story producers over the next several weeks and make their work more tedious; the matchframe issue. I’ve dissected the problem with several people, including Avid, O-bit, and an Avid Supergenius recommended by Lenny. We’ve concluded that this is a bug with the software.

Every scene for our show gets multigrouped. This is the general procedure for reality shows. A sync map is made of the footage, master clips are assigned auxiliary timecode, subclips are made from the sync map, and then the subclips are multigrouped by auxiliary timecode. Multigroups are the easiest way for editors to work because all the footage pertaining to a scene in the show can be put into one group.

When one of our multigroups goes offline, due to media being moved, drives not being mounted, or numerous other reasons, the footage needs to be relinked to come online again. Once the footage is relinked, the multigroup will come back online and everything will appear fine. However, any sequence that has been created with that group will become corrupt with a matchframe issue. When pressing matchframe on the sequence it will match back to the correct multigroup but the wrong frame.

I have nearly exhausted all possible solutions.
1. Have restructured the Unity to one allocation group, optimized, and media databases rebuilt on all the workspaces.
2. Have tried reverse corrupting the corrupt sequences with earlier bins that are uncorrupt.
3. Have tried editing the corrupt sequences into new sequences.
4. Have tried creating new user settings, projects, bins, and site settings.
5. Have tried relinking several different ways, including just the source clips, just the subclips, just the master clips, just the multigroup, and just the sequence.
6. Have turned on “enablecommitmulticam” in the console, then committed the sequence to multicam, which replaced the multigroups in the sequence with the individual groups that the multigroup is composed of. This was a test to understand the problem better, but not a solution.

Moving forward, there are 3 possible scenarios, as almost all our sequences have the potential, and will most likely, become corrupt with this issue.

1. Whenever an editor wants to matchframe to the multigroup, it will still load the correct multigroup, just the wrong frame. They can then type the timecode for the correct frame and continue working.
2. We are currently working in Avid 3.5.4. We can test this issue on Avid 4.0 and see if the bug has been cleared up. Lenny said he may be able to get us a 4.0 system to test it on. I’m not sure how much it would cost to upgrade all the systems to 4.0 if this is the solution.
3. And the final solution, which is the most outrageous and asking for more trouble than its worth, is to find a version of Avid and a different operating system that does not have this issue, Mac or Windows. I have never had this issue on a show before, which leads me to believe its a bug between Avid 3.5.4 and Macintosh 10.5.5.

-Matt

—————————————–
I have found a solution.

If a sequence or multigroup has offline media in it, it should not be relinked. Instead, the master clips that the sequence is comprised of should be relinked. The sequence will then matchframe correctly.

When I tried relinking before, I was trying to fix already corrupted sequences. Relinking the master clips will not fix a corrupt sequence, but it will prevent a good sequence from going corrupt.

If a good sequence or multigroup are offline, and someone tries to relink those items, instead of relinking the underlying master clips, the sequence will become corrupt. The only way to fix the matchframe problem is to then recut the multigroup into the sequence again, replicating all the edits that have been made (aka cleanprinting). Or, pulling a bin out of the project backups.

This can definitely be considered a bug with Avid. I have found no official documentation of this error on the internet, but Avid was designed to allow relinking of sequences and multigroups and it doesn’t work correctly. Perhaps a newer version of Avid would have fixed this, but it’s not necessary as I will inform everyone to never relink a sequence or multigroup.

-Matt Rittorno

XDCAM Reality Television Workflow

XDCAM blows. What initially seemed like the greatest invention in the history of video production should NOT be used for any high end video production.

10. Digitized XDCAM clips cannot be imported, and imported XDCAM clips cannot be digitized. Neither can be relinked to the other, modified, or manually imported. I’ve spent hours and hours to prove it can’t be done.

9. Avid’s Symphony Nitris can not playback the XDCAM’s native HD video format. It must be transcoded to DNxHD in order for it to play correctly.

8. Avid cannot export “Long GOP” media, such as a quicktime reference of an imported XDCAM clip. It must first be mixed down from MPEG-4 525 to a more familiar video format such as 10:1m.

7. Avid’s function “Import XDCAM Proxy…” will not import clips longer than 10 minutes to a Unity drive without giving you an STD.

“Exception: std::exception, what:[MainThread] Threads::SendThreadMessage: Thread message queue full XDCAMMPEG4Processor FifoSpaceAvail”

It must first be imported to the internal hard drive, and then copied over to the Unity.

6. XDCAM proxy media will only come into the system 16:9 anamorphic.

5. XDCAM proxy audio is not the same quality as 10:1 audio. It’s much worse. And when going to online, all the low quality audio must be recaptured (not a usual step in a television workflow). Producers and editors may also complain about the audio quality.

4. When the XDCAM setting “Batch Import Hi-Resolution Media” is turned off, it still batch imports Hi-Resolution media.

3. It got my hopes up, the spirits of those around me, and then smashed my head in.

2. There are probably even more Avid issues when dealing with XDCAM proxy media I have not yet discovered, whereas working 10:1 has been proven in the past.

1. According to the Line Producer, it was a waste of money to shoot in XDCAM.

We will be digitizing XDCAM tapes as opposed to importing to avoid all these issues, saving us no time at all and probably costing us more in equipment rentals vs. shooting in a different tape format.

-Matthew Rittorno

Breaking Into Post

Hey Matt,

I just joined the Post Production Networking Group for Picture Editorial on facebook and checked out your blog.

I have been an aspiring editor (scripted TV and features) for a few years now. I have some formal training for Avid and FCP from interning at Moviola. The only real experience I have in post so far was as an Avid AE on a Discovery Channel reality show. Unfortunately, I wasn’t quite ready for it and they couldn’t afford to have someone learning on the job. It didn’t last long and that was almost 2 years ago. I’ve been doing background/extra work for the last year and was a regular on Scrubs this season. However, I have no desire to pursue acting and would like to try getting into post again. I always apply for internships, post PA and runner positions with no success. Friends of friends who work in post haven’t been any help either.

My experience and contacts are minimal. I’m also not getting any younger at 32. What are your suggestions to pursue this elusive dream of becoming an editor? Any advice would be greatly appreciated.

Thanks,
John

————————————-

Thanks for the email John. With the turn in the economy, I have realized exactly how important networking is in finding a job. In the past year I applied to every job post on the internet, called companies cold, emailed companies, and even drove around dropping off resumes a couple of days. Out of over 100 applications, no one called me back.

I did find 4 jobs over the course of the year though, and every single one of them came about by emailing people I’ve worked with before. It would be hard for me to recommend you for a job where I work, because I can’t attest to your work personally and if you do a poor job then I look bad and we lose time and money. Plus I know very qualified people who are out of work that would do great.

If you want to get into post-production, I recommend going back to a school where you can meet people that can get you a job. Then impress them. Or, email people you’ve worked with in the past that can and want to help you get a job. Or, start interning for free at post-facilities and impress some people that way. That’s the best I can come up with.

-Matt Rittorno

P.S. If anyone is interested in contacting John, I’d be glad to point you his way.