In late December 2019, the New York University IT audiovisual services team received some foreboding news. On a call with colleagues at the school’s Shanghai campus, they were brought up to speed on the novel coronavirus, and how it was only a matter of time until it would upend life in America.
“I saw some stuff [about the virus] on the news around Christmas time, not really thinking that much about it, but then we started to get some pings from the global team,” said Chris Blackburn, NYU’s associate director of audiovisual services and engineering, and the university’s Zoom service owner. “It became real to me when our colleagues in China were audibly upset about what was happening there.” He and his team went immediately to work, toiling through double shifts to scale up its Zoom operations. “We were making a service out something that was previously just a product we had, a tool in the toolbox. But suddenly, it became the focus of the entire university’s operations and livelihood.”
Related: How USC's Joe Way Anticipated the HyFlex Campus Revolution
The Road to AV
Blackburn’s extensive background managing videoconferencing operations certainly made him the right man for the job—a job that, like many in the AV industry, he didn’t exactly plan on doing when he was younger. “Like many people in the industry, I’m passionate about music and sound,” he said.
A decade before starting work at NYU, he was a graduate student there, earning a master’s degree in music technology. This led to a job as a production engineer at Sony Music Studios, where he worked until it closed in 2007. That fall, his path took a turn when a friend at Weill Cornell Medical College contacted him to see if he was interested in helping out with some AV work. “I got into AV, actually because the AV techs got a lot more overtime and I had a lot of NYU debt,” Blackburn quipped. “And, I had a good understanding of signal flow from my audio background, and much of AV is signal flow.”
After working at Weill Cornell for several years, Blackburn took a job as the audiovisual production manager at Etsy in 2012, just as the company was taking off. And though he was only in the position for a year, it proved very formative for his career. “I really learned a lot of lessons there about how IT worked,” he said. “I got to see a company growing at a really rapid pace and learned from their mistakes as well as my own—and there were many on both sides.”
Along with this growth—which included the addition of several hundred employees—came the need to devise a system for communications across a global footprint. Further, the company employed a large number of remote workers, engineers spread out across different time zones. “One of the first things I did was engage with the community, talk with them face to face, and understand what their needs were,” Blackburn said. “That was a big learning experience: learning how to translate that end-user need into an actionable technical solution. We all learn how to be deeply technical in many ways, but we often fail to translate a user experience into a technical solution, or vice versa.” One specific example is “trimming the fat” from user needs: sometimes the most vocal stakeholders ask for more capability than is actually needed from a system.
A Social Worker
Eventually, a path back to NYU presented itself when the university hired Blackburn’s former supervisor from Weill Cornell and gave him the task of “figuring out” its AV situation. He promptly brought Blackburn onboard to help.
“He was upfront with what I would be looking at and the challenges I would be facing, and I walked into a deeply siloed organization,” Blackburn said. NYU is comprised of 25 schools, each with its own AV department. Blackburn and his team were tasked with serving as a kind of consultant to them, as he put it. “One of my challenges was making inroads and making these people advocates and allies, and making them feel comfortable with me being an advocate and ally,” he said. “I joke about being a ‘technical social worker.’ A lot of times it’s just listening to people’s frustrations and letting them vent, and just being a sounding board, and sometimes a punching bag.”
This role—audiovisual systems implementation manager—lasted a little over two years, until the school transitioned him into his current position in February 2019. “There was a larger plan around the shift to Zoom, and I was told that I was going to be the Zoom service owner, based on my experience at Etsy and Weill Cornell,” Blackburn said. At that time, they had just onboarded the videoconferencing service and hit what they expected would be its maximum usage, with about 8,000 people on the account. “It wasn’t getting a lot of use; it was being justifiably used, but it wasn’t heavily used,” he said. “And we would have been happy if we had gotten to the 12,000-user mark by the end of the year.” Little did they know how the world would change by then.
Challenge of a Lifetime
By late 2019, Blackburn and his team had established the Zoom service and were just working some remaining bugs out of the system. Then came the call with the Shanghai campus, setting them frenetically to the task of scaling up. “I can’t say that we were prepared for it,” he said. “It was working nonstop, communicating nonstop.”
Before they could even address the equipment in the classrooms, the team had to first ensure there were enough accounts set up. This entailed navigating a lot of legal requirements around role provisioning. “We couldn’t just open Zoom to everyone because of the restrictions that were placed around it, due to the risk that our office of general counsel assessed with having a platform like Zoom,” Blackburn said. “Not everyone could have recording, not everyone could have the same privileges. Students and faculty had different levels of access from admins.”
The first step was to make Zoom into a proper IT service so that they didn’t have to go overboard manually provisioning special access accounts and user groups. Blackburn’s previous boss had written a script for role provisioning, but the team had to still do a lot of manual work, as the sheer volume of requests was overwhelming the script. The team eventually completed the task by March, in time for the shift to online classes. “I still have probably 15,000 emails in my inbox that I have to answer from March, but it was really quite a learning experience and a unique event in my lifetime,” Blackburn said.
New Standards
Next, the team turned its attention to the fall semester. “NYU is very focused on place: We’re a large school that provides intimate classes in one of the biggest cities in the world,” Blackburn said. “It was really valuable for NYU to replicate that experience as much as possible.” For practicality, this would require standardizing on a uniform setup across the board—a challenge given the varied state of classrooms at the time. “Some of the Polycom rooms kind of worked, and there were a lot of discrepancies between the user experiences and interfaces between the rooms,” he said. “The faculty was confused as to how to proceed. A lot of rooms didn’t have equipment to support Zoom.”
His initial assessments regarding the number of rooms they’d need to equip were staggering: “At one point we were up to 1,150 rooms that needed to be Zoom rooms, which made me cry a little bit,” Blackburn joked. That number eventually came down to around 400, and the team was able to lock in the supply chain by the end of the spring semester.
The strategy around hardware was one of compromise. “We knew that we were going into this with a deeply standardized approach, and it was going to work well in some rooms and not work as well in others,” he said. “Given the timeline, we didn’t have the time to put ceiling mics in, or approach it in a way that was appropriate in every space.”
The team eventually landed on a modular solution, with carts from Legrand and Salamander Designs outfitted with Crestron Mercury conferencing hubs and 86-inch LG displays topped with Huddly cameras. They chose the former because its robust audio performance, and because Crestron’s manufacturing, which is based just across the Hudson River in New Jersey, wasn’t affected by the restrictions on imports from China. The display choice came down to a similar supply-chain decision, and the screen size was the product of some compromise. “We had to hit a sweet spot of a solution as best as possible,” Blackburn said. “We have a lot of old buildings that we’ve appropriated for educational purposes, so freight elevators become a challenge. You can’t get a 98-inch display in some of them.” Thus, 86 inches was the maximum size that was feasible for all of the rooms.
The rollout of this standardized setup was phase one of the school’s new hybrid-flexible strategy. Though the orders were in by the end of May, restrictions prevented work from happening until much later; installation didn’t begin until late July and lasted until the beginning of September, right before classes started back up. Integration was carried out by HB Communications and Verrex, with additional work provided by Videosonic and Monolith Modular.
For the fall semester, 40 percent of classes have been fully online, with the remainder being conducted either hybrid or in-person. “We have to allow for hybrid in most of the classrooms in case a student has to quarantine,” Blackburn said. “We have to provide equity.”
Navigating the Normal
Managing the systems in these times hasn’t been without its challenges. The foremost is the issue of being able to deliver in-person assistance. “Our operations staff is really limited in the support that they can provide,” Blackburn said. “My team has partnered deeply with the operations team to keep the systems running.” In fact, Blackburn does almost all of his work remotely, relying on this partnership with the engineering and operations teams to manage work on the campus. “That’s hard for me,” he said. “But it doesn’t mean that you can’t manage it effectively if you have the right people on the ground, and luckily, we do. Our operations staff is really good, and they deserve all the credit in the world for their tenacity and willingness to deal with significant challenges with new technology.” He specifically credits his AV and Zoom Engineering team of Ken Eng, Paul Loverro, Julio Carrasco, and Alejandro Abarzua for helping him navigate this trying year.
There have been some unexpected learning experiences along the way, mostly stemming from the unprecedented scale of the deployment. “We are finding these issues in relatively large quantities that aren’t always readily apparent when you deploy two or three, or four at a time,” Blackburn said. “We deployed several hundred at one time, and now we’re finding issues with the BIOS. So now we’re working through updating the firmware on the BIOS to stabilize the systems.” There are also device-specific factors: “Disabling Windows updates on a DTEN is the best practice, but disabling Windows updates on a Crestron Flex is not the best practice. So we’re having to find the nuances of the systems and the errors that we encounter at the same time.”
The Road Ahead
Blackburn and his team are now in the process of working on phase two of the tech rollout, which improved upon the classroom setups in time for the spring semester start in January. This involves enhanced audio DSP, speakers, and mics in about 50 rooms, and a second camera in the remainder of the spaces. “Knowing some of the upgrades that Zoom has added to the Zoom Rooms app, with the dual camera view, we want to be prepared for that,” he said. “That was a big ask from the faculty, being able to have a dual camera view from the Zoom Rooms natively,” enabling participants to see both the professor and the students in the room.
As for other technologies down the road, Blackburn likes to take an analytical approach before going all in. “I like to be an early adopter, but the pragmatic side of me needs to stand back and really look at how the ecosystems evolve before I can make a clear decision,” he said. “I really only want to be an early adopter when the bugs are worked out.” That’s been his approach for AV-over-IP systems, which he has been deploying over time in situations where it makes sense. “As I get more clarity on how it can be an ecosystem, I’ll start to really pull the trigger,” he said.
And when it comes time for a big move like that, he stresses the importance of knowing how to communicate with key stakeholders, executives, and staff, and understanding the rationale for differences in perspective. Sometimes, that can mean bringing in someone from the outside who can better evangelize a technology’s benefits, as he did when recommending Crestron’s NVX AV-over-IP system for a network operations center on campus a few years back. “Alex Peras from Crestron was a master at really communicating with the networking team and explaining the needs and the benefits of the NVX units, and how we should proceed forward with the deployment,” Blackburn said. “Sometimes my job isn’t always being the absolute subject matter expert, but it’s more of knowing who needs to talk to whom and facilitating that conversation.”
And, in general, he stresses that candidness is key, especially in a field where the users don’t always know what to expect from the technology—and where technology sometimes doesn’t work as expected, either. “I try to be upfront with people so they understand what the limitations are,” he said. “Just being kind but also truthful. Well delivered transparency is maybe your best friend.”
