Emerging from Sport Shutdown: Venue IT

Seizing Opportunity: Remote IT Management for Sports Stadia in Post-COVID World

As we look to emerge from the coronavirus-induced Great Sports Hiatus, teams and venues everywhere are focused on reducing two things: infection risks and costs.

Whether fans are in seats when play resumes or not, reducing the number of staff on site checks both those boxes.

“Right now, there are a few things all venues are worrying about as everyone gets ready to re-open,” according to Ted Mondale, head of business development and government relations at Atomic Data, a Minneapolis-based IT as a Service (ITaaS) firm. “First, they are uncertain on exactly how everything is going to happen. Secondly, they want to open as efficiently–read: as cheaply–as possible. And thirdly, venues are concerned with safety and ensuring people don’t get sick.”

Ironically, old school thinking continues to dominate management of the newest and most advanced equipment in venues: information technology (IT).

“Coming at it as partners, both sides are fully invested in making things work the best ways possible.”

-Jim Wolford

As one senior stadium executive speaking on background told TMR, “To provide the best game day fan experience, we have to have multiple IT folks on property, especially as we ramp back up. Think about box office walk-ups and exchanges…concessions POS systems where we now are adding untested remote ordering for distancing…then there’s new merch sales processes…venue A/V changes…gate and security ops…it goes on and on. Our IT guys already ran around non-stop from one issue to the next. You never want premium seating Wi-Fi down, seating bowl security cameras going offline, craft beer vendor POS crashing, concourse video screens freezing, etc., etc.”

Okay, we get it. It’s a trying time, for sure, and you’re going to need some physical presence.

What if, instead of needing an IT army, venues could handle the job with just one person on site and remotely handle everything else?

Firms delivering ITaaS in the corporate world have realized a huge opportunity for sports venues: implementing remote management systems.

WolfordAtomic Data

Jim Wolford founded Atomic Data in 2001 to tackle exactly these kinds of issues. He set out to combine his experience in corporate sales, risk management and data centers with the goal of creating simpler, more efficient IT offerings for corporations.

Today, the privately-held firm has more than 250 employees, over 600 clients and reports 2019 revenue of $40+ million (including The Foundation, an Apple/iOS-focused affiliate). Despite the pandemic slowdown, Wolford projects to end up between $40-$45 million for 2020, depending on several venues and other large projects that are planned, but currently on hold.

“I started Atomic Data coming from a corporate risk management background,” explains Wolford. “I helped companies map their risk, meet federal requirements and things like audit, health care, insurance and employee risk. Everything tied to helping the business be the most efficient, and that is what we’ve done for 20 years: work with small, mid and large businesses to help them manage IT most efficiently.”

The company runs IT networks in thousands of remote locations including 4,000+ Great Clips stores and numerous health care facilities across the U.S., as well as remote monitoring, data centers and design and build work.

Along the way, Wolford and team repeatedly found sports and entertainment venues being underserved for IT.

A roof collapse that opens doors

Atomic Data’s work in sports dates back nearly a decade thanks to the Hubert H. Humphrey Metrodome‘s Dec 2010 roof collapse.

Dec 11, 2010: Fox Sports cameras set up for a Dec 12 Vikings game captured the Metrodome’s roof collapseFox Sports Screengrab

Remember that? The Metrodome’s fifth roof failure brought added attention to the aging facility’s untenable infrastructure. The NFL Minnesota Vikings were still fighting an uphill battle to get what would become U.S. Bank Stadium financed and had nowhere else to go.

To keep the facility simply NFL-worthy, numerous upgrades were necessary across the entire facility. Meanwhile, every moment that went by standing put, was placing the Vikings’ 2011 season at further risk.

Enter Mondale–a former Minn. state senator and the son of former U.S. Vice President Walter Mondale–who was appointed the new chair of the Metropolitan Sports Facilities Commission (MSFC) in the days immediately following the collapse.

Ted Mondale recalls those chaotic next few months to get the venue playable with a chuckle: “The dome’s IT set up was so desperately in need of updates. For one thing, the POS system took three minutes per transaction! At another point, we found that our firewall had been off for two years!”

Having spent time working at tech start-ups, Mondale knew Wolford and asked him and the Atomic Data team to take a crack at stabilizing the Metrodome’s IT operations, and then running game day IT operations for 2011-12.

Saddle up for more sports

The sports business case prove point for Wolford did not come via stick-and-ball sports, however. He can thank the ponies.

In 2013, Canterbury Park Race Track & Card Room in Shakopee, Minn., needed to completely overhaul their IT infrastructure. Not just a Wi-Fi boost, but all IT across hundreds of acres from track operations and stables to front office and handle reporting to off-track streaming and results.

Scoff at comparing a horse track to an NFL stadium if you must, but the project not only proved out Atomic’s value proposition. Canterbury Park gave them valuable experience with not just any sports facility, but both a heavily regulated one and one that chewed up a massive amount of bandwidth.

Think about the infrastructure required over a space big enough to fit all 30 NBA arenas–hundreds of TVs, hundreds of work stations, track broadcasting needs, fool-proof wagering systems, spread out concessions, and, oh yeah, thousands of attendees placing mobile wagers and streaming races at other tracks, etc.

Back to the gridiron

In 2015, Mondale, now the Minnesota Sports Facilities Authority (MSFA)‘s Executive Director, was dealing with cost overruns left and right on construction of the finally approved U.S. Bank Stadium. (The MSFC was disbanded with the MSFA formed to replace it and then oversee the new stadium.)

For example, IT costs were budgeted at $80 million for the entire venue, but thanks in no small part to Mondale’s predecessor, actuals were projected to blow by the $80 million mark. Mondale found extraneous spending–he mentions $1,200 Cisco video conferencing phones already purchased for every suite and hundreds of $800 iPads solely to serve as suite A/V remote controls–that he had to put a stop to immediately.

“Ted knew that new tech and its integration into sports venues was coming,” Wolford says. “Wi-Fi in particular was becoming a huge issue. Fans were now expecting to stream highlights, follow other games and share their own content out while at a game. We also knew we had to help the venue with their needs–capture and compile info on season ticket holders, concessions, lines at bathrooms, etc.”

At the same time IT had to be top-notch, it also had to be budget-friendly.

Enter the glue that holds it all together: project manager Yagya Mahadevan.

The MFSA hired Mahadevan to fill the vacant owners technology representative role, and he and the MSFA/Vikings project team quickly brought IT costs back in line with budget. Actually, not only did IT spend end up within budget, they actually came in nearly $4 million under, and all work was completed ahead of schedule.

It was during this time Wolford, Mondale and Mahadevan first intersected. That connection was fortuitous.

“What we also looked at was figuring out ways for the stadium to monetize these services,” continues Wolford. “We work with clubs to identify opportunities to better identify their marketable assets with Wi-Fi implementation and to leverage data from Wi-Fi for better understanding of fan behavior.”

In other words: What about selling sponsorship of stadium WiFi access? How can teams measure not just sales, but which fans did what while at the game? And most importantly, What can we learn to sell more and be more efficient?

Minnesota United relationship pays off big time

Next up for Atomic Data was the buildout of MLS Minnesota United FC‘s (MNUFC) impressive new Allianz Field which broke ground in Dec 2016.

With U.S. Bank Stadium opened and Mahadevan’s work complete, Wolford snapped him up. He tasked Mahadevan with building a new network to run MNUFC’s merchandise and box office transactions as well as game day IT during their temporary stay at the University of Minnesota‘s TCF Bank Stadium. That was to be followed by implementing all Allianz Field technology.

In 2014, Atomic Data sponsored the back of MNUFC’s home and away jerseys

It helped that both Atomic Data and MNUFC already had well-established trust, as the two started working together in 2012 (when MNUFC was an NASL team). That lead to becoming MNUFC’s day-to-day IT support firm–and in 2014, the team’s first kit sponsor–while the club was playing in the USL Championship.

For the privately-financed Allianz Field build, Atomic Data first ran the back-end of the “Allianz Field Experience Center.” The 2,100sf immersive space designed by St. Paul-based cmnd+M with digital flyovers, 3D renderings and even holograms gave fans and sponsors an interactive preview of the then under construction facility.

Meanwhile, Chris Wright, MNUFC’s CEO, was looking for a true IT partner for the team and new venue, not a transactional relationship.

Coming from 26 years–including the final 13 years as president–with the NBA Minnesota Timberwolves and WNBA Minnesota Lynx, he was familiar with what most large IT companies tell large leagues and teams: ‘We’ll sponsor you for $X+Y and you pay us $X for our equipment.’

“Those trade out deals are done at a high-level, often without regard to actual needs, which leads to either too much–we typically see those facilities using just 5-10 percent of capacity–or worse yet, the wrong equipment,” notes Wolford. “The difference is we come with the approach of helping the business be most efficient.”

In the case of MNUFC and Allianz Field, that meant Atomic actually owns all the hardware (routers, servers, switches, access points and firewalls) across WiFi, LAN, VoIP.

It’s a zero-down, 6-year pay down agreement. It also saw Atomic Data named “Official Wi-Fi, IT and Cloud Partner of MNUFC” in Apr 2018.

“It’s a unique arrangement and the only sports stadium deal like this that we’re aware of,” says Wolford. “But coming at it as partners, both sides are fully invested in making things work the best ways possible.”

As Wright explains (below, in one of the most glowing case study videos you’ll ever see), “When I was looking at the IT/Wi-Fi/DAS space, we needed a partner to come along side me, and educate me about what we needed inside of this stadium to accomplish all of our goals. We also decided we needed a mobile-first strategy.”

 

In addition to building a mobile-friendly Wi-Fi experience that’s called “the most robust in MLS,” Mahadevan had a breakthrough getting a new dashboard live and working across all screens (mobile, desktop and tablet), or as those in IT call it, “Single Pane of Glass.”

An example of the Game Day Dashboard streamlining IT monitoringAtomic Data

The “Game Day Dashboard” not only incorporates all IT remote management essentials including IT Closet temperatures, humidity and water alarms, as well as networkwide connectivity and bandwidth in use, but also brings together many specific to sporting venues. What previously involved perhaps 10 different platforms, special software or endless browser tabs open, was now all pulled together in one place: concessions POS, merchandise vendors, box office reporting, back office operations, A/V management, HVAC, gate security, traffic, etc. Even team and league operations such as analytics and player tracking can be integrated.

In addition to simplifying all this IT management, Atomic Data adds a very simple process to the remote management: the reboot.

The reason every IT Help Desk person always asks “Did you reboot?” first? Because it works.

Now, if an access point is down, it automatically reboots that hardware–and that resolves the connectivity issue nine times out of 10.

As Mondale says, “We’re moving from reactive to preventative, before guests even experience any problem.”

Traditional IT Response
Remote IT Response
6:55pm Gary Guest arrives at arena, receives urgent email while heading to suite 6:55pm Gary Guest arrives at arena, receives urgent email while heading to suite
6:57pm WiFi router nearest destination suite goes offline 6:57pm WiFi router nearest destination suite goes offline
7:00pm Guest arrives in the suite to realize Wi-Fi is down, tries troubleshooting 6:57pm Remote Monitoring Service realizes access point intermittently connects, activates reboot
7:04pm Guest tells Sue Suiteholder, who also tries troubleshooting on own, both increasingly frustrated 7:00pm Guest arrives in the suite, pulls up key files and sends them off, crisis averted and game enjoyed
7:06pm Suiteholder heads out to track down attendant several suites away; Attendant tries to troubleshoot, they can’t resolve it either
7:09pm Attendant calls Service Desk, is placed on hold; Service Desk returns and requires Attendant to walk through venue IT protocol; still no resolution
7:14pm Service Desk dispatches an IT person following protocol failure
7:25pm IT Person needs 11 minutes to finish resolving first issue and depart, then has a 10-minute walk to suite
7:35pm IT arrives, runs diagnostics, can’t fix issue there: “Yep, need to go reboot that router!”
7:39pm IT heads to nearest IT Closet, a 5-minute walk, plus a bathroom stop
7:47pm IT Person reboots access point; access is restored; notifies Service Desk
7:50pm Guest, a huge fan of the home team, remains unable to respond and angrily leaves arena to make calls
7:59pm Service Desk reaches Attendant who then returns to suite to notify Suiteholder; Suiteholder and Guest’s significant other are beside themselves

 

We asked why the dashboard and remote monitoring concepts didn’t exist in sports? Wolford, Mondale and Mahadevan all shrug with sly smiles when asked. There’s no good answer, but they’re really glad to be the first.

Mahadevan: “Every time we sit down–every team or stadium–they tell us, ‘We’ve wanted to build this for at least the last five years!”

“I was shocked [the sports venue space] was as wide open as it is,” adds Wolford.

It’s in the details

With Allianz Field launching as a paperless ticketing facility from Day One, special attention during planning was paid to the entry gates to make sure fans could easily connect to Wi-Fi to access their digital tickets. Atomic Data also staffed a display with staff to assist with the free Wi-Fi and answer any questions.

(Left) Atomic Data staffed a display at Allianz Field to assist fans with WiFi; (Right) Atomic branded their equipment and were recognized as “Official IT Partner” of MNUFC Atomic Data

Over the course of the season, the most robust Wi-Fi network in MLS averaged 40 percent usage per game compared to the typical 20 percent.

The POS infrastructure handled more than 315,000 transactions and in-seat ordering was tested out successfully with more than 2,500 orders completed.

MNUFC and Atomic have earned numerous plaudits for their work in Allianz Field’s inaugural 2019 season. That includes MLS commissioner Don Garber calling the stadium “the standard” for all venues going forward.

What’s next?

In no small part part due to the success of Allianz Field, Atomic Data was retained last year by MLS FC Cincinnati ownership to provide IT scope and building owners rep services for their new West End Stadium, now under construction.

FC Cincinnati’s West Ed Stadium, shown Apr 7, 2020, is progressing well; Atomic Data is handling IT planningFC Cincinnati

When pressed on their impact for a client like FC Cincinnati, Atomic Data broke down proprietary numbers for TMR to verify saving millions of dollars compared to the original budget. How? By right-sizing infrastructure without compromising end user experience. That includes optimizing IDF/access control, data ports, security networks and A/V, as well as switching to a neutral host provider.

Another example (which again included allowing TMR a look at proprietary pricing details) includes a recent pitch to an NFL facility. By implementing the Atomic Monitoring System and remote management, the stadium’s existing IT support costs dropped from $318,000 to $173,000, a 46% reduction.

Wolford, a sports fan originally leery to get Atomic into sports ITaaS, is now fully on-board.

In a bit of fortuitous timing, Wolford formally launched Atomic’s sports stadia business unit, Game Day Technologies Group, this Jan before the pandemic took hold.

Focused on “bringing a holistic approach” to meet the various IT needs of both existing and new stadia, Wolford added staff with MLB team and sports marketing agency business development experience. The group immediately began aggressively reaching out to teams, venues, venue management companies and stadium design and construction firms.

Wolford says two facilities were dotting i’s and crossing t’s on agreements with Atomic Data when the pandemic paused everything in sports.

“We’re still working with them and we’ve let them know you don’t have to pay until when, not if, you host a game,” says Wolford. “In all, we’re working with about nine others right now.”

Still skeptical, we prodded, ‘Say a building jumps today to retrofit, it takes what–six to nine months–to implement a dashboard and remote monitoring?’

“We’re ready to deploy immediately,” promises Mahadevan. “From client approval, we can deliver in two to four weeks.”

Sounds like it makes sense to give Atomic Data’s Game Day Technologies team a call. MNUFC’s Wright agrees.

“You should be talking to Atomic.” Wright says he tells his peers. “They should try to replicate the relationship we have with them.”

Chris brings deep sports business experience to his role as publisher of TMR. He first put his sales, experiential marketing, PR, sales and valuation skills to work in sporting goods retail in 1986. He has since worked for brands and agencies across all major league sports, plus golf, college athletics, marathons and motorsports. Chris is also the proud founder of Painless Networking.