Background
The Village of Schaumburg is a well-known northwestern suburb of Chicago and recognized by many in our industry as the former home of Motorola. With a population of just under 75,000 residents, Schaumburg was recently ranked as the best place to live in Illinois (2018) and the 9th best place to live in the United States (2017) by Money Magazine. In talking to both residents and village employees, the reason behind these high scores becomes apparent; there is an unrelenting focus on cost effectively delivering value to residents in terms of safety, services, and community resources.
In 2016 the village began to implement a plan for adding video cameras to various properties as a proactive measure, with a goal of creating an enterprise platform for video security. “This was not driven by a specific incident or group of events,” explains Peter Schaak, Director of Information Technology for the village, who led the project. “We felt there was a need to capture certain activity at critical areas, without being intrusive to residents and employees.” The system was to look outward, not inward, and was divided into five (5) logical steps, or phases, with a multi-year implementation envisioned.
The initial approach was to design the system internally, as with other IT projects, but the benefits of working with an outside consultant became apparent as the team got further into the project. “We wanted someone to look over our shoulder and catch common oversights based on their experience,” Schaak explained. A higher level of detail based on similar experience would minimize ambiguities in the technical specification, reducing costs as there wouldn’t need to be as great a buffer for unknown contingencies. A more comprehensive and well documented design would also make the project more enticing to a higher tier of integrator and gave the village a sense of security, in that there would be someone on the team who had done this before. “And it was nice not to have to do this all on our own.”
Designing the System
The village took the work they had done to date on a system design and incorporated it into a Request for Qualifications (RFQ) for consulting services. 13 responses were received, and our firm, R. Grossman and Associates, Inc. (RG&A), was ultimately selected to design, bid, and project manage this multi-year project. Having clearly defined requirements from the client was tremendously helpful and using their RFQ as a starting point for the system design, an extensive information gathering effort was undertaken to create a design that, when complete, will connect and centrally monitor over 200 cameras in 30 locations throughout the village. These include the Village Hall, performing arts center, public safety building, nursing division, senior center, municipal airport, commuter rail station, engineering and public works building, five (5) fire stations, and 12 water infrastructure facilities (towers and tanks).
The system was designed with a distributed architecture to minimize the bandwidth requirements while maintaining image quality (framerate, resolution, and compression). For the most part, cameras would be recorded on a server at each location, with the exception of a few locations that were already connected via fiber and had dark fiber strands available. Servers installed in conditioned space (or open interior space where temperature was not an issue) utilized 1RU Razberi servers with four (4) data drives that can be put into a RAID5 configuration, and a solid-state drive (SSD) for the operating system and VMS software. Locations that did not have any sort of temperature control (such as water towers) utilized hardened servers rated for an operating environment ranging from -13OF to 140OF (-25OC to 60OC) with 10% to 90% non-condensing relative humidity, to be on the safe side.
All servers were integrated boxes, incorporating integrated PoE+ network switches for the associated cameras, and were sized for the implementation of all phases. For example, if there were two adjacent buildings that were covered in different phases, the server was sized to handle the total camera count, reducing costs in later phases. Reliability was important as well; All patchable cabling utilized pass-through keystone jacks rather than 110-style punch down blocks, to simplify troubleshooting later. SSDs were used exclusively on the hardened servers for both the operating system and VMS software, and a separate SSD was used for data. Surge protection and UPS systems were implemented throughout.
Integrator Selection
The RFP was released in October of 2018 and was designed to be scalable. There was a base bid that included certain elements common to all phases of the project (i.e. operator workstations), and four (4) additional phases that could, for the most part, be implemented in any order – all required the base bid, but only Phase 5 had other dependencies. The 11 responses were generally excellent and included a good cross section of national, regional, and local integrators. Pricing was varied – the high bidder was almost three times as much as the low bidder — but fell into roughly three pricing bands, once bidders that lacked credibility were disqualified. Choice of a video management system impacted pricing, so some time was spent on that to narrow the field (see sidebar “Modular System Design”).
After careful consideration, Addison, IL based Pentegra Systems was selected as the integrator that we felt would best serve the village and this project. Founded in 2000, Pentegra has won several awards, and was extremely responsive throughout the evaluation process. Their team, including CEO Ed Karl, President Greg Augspurger, Security Engineering Manager Gene Brierton, and Sales Engineer Jim Lichter, impressed us with their knowledge, experience, and collaborative approach, and their references echoed those sentiments. Their solution included Antaria Technologies hardened network switches, Axis cameras, EnGenius EnStation AC wireless links and Razberi servers (both hardened and rack mounted). Once the final submittals were received and approved and contracts were in place, we began work on Phase 1 of the installation which included 6 different locations and roughly 70 cameras
Installation
The on-the-ground task of managing the installation fell to Chris Westgor, Technical Services Manager for Schaumburg, and Gene Brierton at Pentegra, with remote support and site visits at key milestones from myself and our team at RG&A. Almost immediately, I spotted a commonality between Chris and Gene in that they were always looking ahead. Understanding that this was a multi-phase project, they seemed to focus on lessons learned, proactively. While this led to some fine tuning on Phase 1, the idea was to have consistency across all phases, improving quality while being mindful of efficiencies that could be achieved.
Phase 1 included cameras in the Village Hall, Schaumburg’s renowned Prairie Center for the Performing Arts (including an outdoor stage), a senior center, the Engineering and Public Works building (including a fuel station), the municipal airport (including fueling station, terminal building, parking areas, and hangers), the commuter rail station, and one of the water towers. With the number of different venues involved, Pentegra liked the variety. “Each location was different and required adjustment and flexibility to install, and each presented unique challenges to provide the camera views that the village was looking to achieve,” noted Gene Brierton of Pentegra. This presented a major coordination effort, however. “Proactive communication and scheduling with the village departments and personnel required extra attention. Each have their own daily tasks and calendars that work in conjunction with each other,” Brierton added. This requires multi-collaboration, and Pentegra earned high praise for this. “For the most part, unless there were specific issues to address, we didn’t even know Pentegra was there” remarked Westgor.
Asked to describe a specific area that presented challenges, Brierton pointed to the municipal airport. “The Schaumburg Airport site was one of the most challenging to coordinate since it involved tenants, flight crews, aircraft, village staff, and the public on a daily basis. There were special power considerations that required specialized cameras and lenses to achieve specific views as well as point-to-point wireless systems for the camera network.” In particular, we had specified the use of light poles for camera mounting positions that were later ruled out for various considerations, forcing us to seek out alternate locations and mounting methods to achieve the desired views. Pentegra was instrumental in making this happen, offering field modifications and product substitutions that improved upon the initial design.
Pentegra went the extra mile in this and other locations. For example, a specification requirement was to paint conduit and housings to match, to the greatest extent possible. At the airport hangers, this meant using three (3) separate colors to hide the conduit; at the performing arts center where the cameras and wireless antennas were visible as part of the stage, even the antennas were painted to blend in with the surroundings.
Water towers required careful consideration as well. “You can’t just drill a hole in a water tower for cabling,” observed Westgor. “Engineering review and approval are essential to minimize the number of penetrations, and cabling sharing the same penetrations needed to be compatible to minimize signal interference.” As all but one of the water towers are in later phases, this will be an area of focus and coordination going forward.
Asked how working with a municipal contract like Schaumburg is different from working with commercial/industrial clients, Gene Brierton remarked “There are many departments within a municipal entity that are involved rather than one individual. Collaborating with all departments in a team environment was key to providing a successful project and satisfied customer. Working with the Village of Schaumburg, you begin to understand they are all working together to support their public customers. Working within that environment is much different from a commercial/industrial company who is mainly responsible for their employees at a given facility.”
Lessons Learned
The advantage of a multi-year project is that you aren’t sitting back when it’s all over, talking about what you would do differently in the future. You can implement the improvements you would like to make, even going back to earlier phases to make improvements. In some areas, pole mounted cameras were moved to buildings because of power issues at the poles. Server data drives, initially in a JBOD configuration, are slated to be configured as RAID 5 arrays, both in future phases and retroactively for the Phase 1 servers. Some of the locations with multiple fixed cameras facing different directions on a single pole will be getting multi-sensor cameras to improve the outward appearance and add camera views. A radio antenna tower that was to be used for a camera mounting platform was replaced, requiring plan modifications to achieve the desired coverage.
The first phase also included several change orders due to unforeseen construction issues. Working closely with Jim Lichter and Gene Brierton at Pentegra, we did a pricing refresh for Phase 2 which incorporated these lessons learned and should eliminate change orders except for the most extenuating of circumstances. “This refresh gave us an opportunity to make the unpredictable less so” commented Lichter. “The project team (VOS personnel, Consultant) were very receptive to new ideas and applications of the video system when presented, and we’d be foolish not to take advantage of that” added Brierton.
As we head into Phase 2, the Phase 1 cameras that had previously been installed were already paying dividends. Without naming specific incidents for confidentiality reasons, Peter Schaak remarked on the comparison of design intent to results. “We may not have gotten everything on our wish list right away – there are always other considerations, and we are just starting down this road. But we got exactly what we were told we would get during the design process.” Referring to the team of client, consultant, and integrator, Schaak concluded “The team has turned our RFP into reality, and we are happy to see that happen”.
Sidebar: Proper Network Design Prevents Cameras from Phoning Home
There has been a lot of attention paid to cameras from certain manufacturers being able to “phone home” and potentially report sensitive information back to their manufacturer and potentially a government entity that may be affiliated with said manufacturer. While we understand the theoretical danger, with a properly designed system this is simply not an issue.
One way to do this is to build two (2) separate LANs, either with discreet network switches or by setting up a VPN. We like to do this with separate switches as it makes servicing the system easier, but either way only the Client LAN is exposed to the outside world; cameras are on their own network with no Internet access. To view a camera, you need to go through the system firewall and the VMS software, as the servers bridge the two LANs. Cameras cannot phone home, regardless of manufacturer or country of origin, and a hacker cannot disable a camera without first going through the server and proprietary client software that they are likely unfamiliar with.
Sidebar: Modular System Design
When we design IP-based CCTV systems, we liken it to a “Lego” approach, dividing the components into three distinct categories. First and foremost are edge devices, or cameras in the case of CCTV systems. These should conform to industry standards, providing a wide range of compatibility. There can be special features, but they shouldn’t require a proprietary client to take advantage of them.
Second are servers and hardware, which includes cabling and infrastructure. Again, adherence to standards is critical, and we try to stay away from proprietary boxes with embedded operating systems. If a client likes Dell servers, for example, they should be able to use them here, and leverage their service and support expertise.
If these first two areas are done properly, there is some flexibility in selecting a video management system (VMS) vendor, and it doesn’t necessarily have to be done up front. In the case of Schaumburg we picked a VMS vendor for the specification that used industry-standard servers and integrated with a wide variety of edge devices, and allowed substitutions. We did not accept VMS submissions that were essentially the offerings of camera companies as these rarely integrate with their competitors beyond the feature limited ONVIF standard.
While it was not the specified solution, after an on-site “shoot-out” we selected the Genetec Security Center VMS. It has an excellent feature set, the village liked the user interface, and they felt Genetec made an excellent foundation for future electronic security efforts. Surrounding communities and other agencies have already committed to this VMS, and it is in use in the local school district as well.
Robert Grossman is president of R. Grossman and Associates (www.tech-answers.com), a consulting group specializing in electronic security projects, including casinos, government facilities, commercial, and retail applications. He can be reached at (609) 383-3456 or rdgrossman@tech-answers.com.
