What keeps us up nights when we’re designing, installing or maintaining a system is this question: What will I do if it doesn’t work (or stops working)? We spend a lot of time on any electronics system, outlining specific goals and defining what will happen when Mr. Electron flows down your neatly bundled cable. Yet we’ve all hit those times when equipment just doesn’t function the way it should.
Fortunately, we are not helpless when problems arise. Let’s take a look at some techniques that people around me have used effectively, which I have gratuitously stolen for my own bag of tricks. After reading this, please send me ideas you know of – I’ll be happy to steal them, too. As Foghorn Leghorn, the legendary Warner Brothers cartoon character, was fond of saying when everything went wrong, my feathers for just such an occasion.”
Let’s start by setting the scene: You’ve been called because a system you designed, installed or sold does not work. It worked fine yesterday (maybe you had just fired it up for the first time), but today it’s a pile of junk with wires attached. The owner is hopping mad and is threatening to stop payment on the check, to endorse your competition or to start sticking pins into that little doll that looks like you.
If this predicament sounds familiar, the solution may be closer than you think. Although I can’t claim credit for the following ideas, they have all worked for me in such situations, and most have made me appear more knowledgeable than I really am.
Eliminate self-doubt
Step back and be objective for a second. You might not be the world’s greatest technician (or propeller-head), but you are good at your job. That’s why you enjoy it. seek to improve and read magazines like this one. We’re all held back because inside our minds we are forever the person we were at five years old. It is sometimes difficult to have confidence in yourself and your abilities in stressful situations. A mental pep talk can help. Remember, you don’t need to know everything; you just need to know who to call when you run out of ideas. More on that later.
The second doubt that we all hit is “Why are they calling me?” Although you may think the person on the other end of the line has a much better grasp of that particular system, never underestimate the forest-forthe-trees syndrome. They may not necessarily need you — just someone to whom they may relate the litany of events in hopes that the solution they had overlooked will become apparent.
Here’s another premise to put you in the proper frame of mind: If it worked once, it will work again. There are no problems that cannot be fixed. In the case of a new installation, if it has been done before, it can be done again. I have worked with a lot of prototype or first field-test equipment, and I still maintain that philosophy. Granted, the only place that some devices have worked is in the controlled environment of an engineering lab, but they did work there as expected. If I can find out what is different, I am well on the way to setting things right.
Too often we assume that all electrical power systems are created equal, and that a large uninterruptible power supply (UPS) system adds further reliability. In my experience, neither is the case.
I have had great success with power-line conditioners and surge suppressors, so they are now specified on all jobs. In one instance, on a project we installed in the Philippines, our CCTV system was the only electronics system in the facility that included these items. To date (and I am knocking on wood), that system is the only one not to have crashed because of power-line fluctuations. These units add so much reliability that one manufacturer (Tripp-Lite, Chicago) provides a lifetime insurance policy of up to $25,000 for repair or replacement of any equipment damaged by surges when connected to its devices. Pointing this out to the customer invariably closes the sale.
When equipment is connected to an on-line UPS system, the need for this protection does not go away. Your equipment is theoretically protected by the nature of the system as it regenerates line current. That is true, but the equipment is not protected from malfunctions in the UPS system itself Often, these systems are too poorly maintained – or not maintained at all – for you to have any confidence in them once the warranty runs out Battery banks that had bad batteries jumpered and removed (decreasing line voltage), surges being passed when line current is removed from the UPS and reapplied, and a host of other horror stories make me wary. Although a UPS is essential for uninterrupted power service, it doesn’t eliminate the need for additional protection for delicate electronics equipment.
Among the things that can be done to avoid power-related problems, two in particular are frequently missed. Usually neither one costs much to implement, and both can save a great deal of head scratching. They also gain you a great deal of credibility with the electrical contractor, who would otherwise be called in to correct the problems if you don’t take these steps.
First, specify that all equipment – racks, consoles and remote devices – be powered from the same electrical phase. This allows CCTV cameras to be synchronized successfully to power-line frequency in most cases, and avoids problems with ground and voltage differential. Careful planning here can avoid troubleshooting nightmares associated with buzzes, hums and video hum bars that often accompany the commissioning of a sound or video system. If you ignore this factor, even an installation that works perfectly on opening day is a sword of Damocles hanging over your head. As electrical contacts, connectors, cabling, switches and circuit breakers slowly deteriorate, these problems can appear suddenly. Because their cause is out of your control, it can be a nightmare to identify and troubleshoot at that time; it’s far better to avoid it all in the first place.
The second step is much more obvious but missed just as often; over-specify your electrical requirements, both with current and quantity of power drops. I can think of no drawback to having more electrical power available than is needed and more power outlets than plugs. The days of shutting off half the system to plug in your soldering iron because the demand was too great or there weren’t enough outlets should be forever behind you.
Many times people get so involved in the situation that they can’t step back and look at it objectively. I work with several people who all network with each other, calling with problems that always seem impossible. Often the solution is found by the person relating the story; as he talks about it, the flaw becomes apparent. This is why many of us believe our spouses are so technically astute. A good listener can solve more problems than the most complex computerized expert system ever invented.
Check the obvious
A friend of mine who works for the technical-support department of a large closed-circuit television (CCTV) equipment manufacturer is amazed at how many people are offended when he follows his standard troubleshooting procedure. “The reason we have this procedure is because it works,” he says, “not because we want to insult our callers.”
First he asks the caller, “Is the unit plugged in?” Then he asks, “Did you check to make sure the outlet is hot?” His customers think these are stupid questions and want him to get on with the business of solving their problems.
“When they start telling you how long they’ve been in the business, you know you’re not going to be able to tell them anything,” he said.
In a similar vein, check all connectors for cable stress and pressure. Contrary to popular belief, they do go bad just from sitting there if the cable is applying enough pressure on the connector. “You would be amazed,” my friend told me, “how much equipment comes back to us for repair with nothing wrong. If people would just stop and look at the installation before packing it up in a box and sending it out for repair, they would make us feel like the Maytag repair man.” The irony is that these obvious things — so often overlooked — are by far the easiest to check and fastest to solve.
Another obvious booby trap that I and others have fallen into again and again is “color dyslexia.” One of the first troubleshooting steps anyone will take is checking the color codes on cables. This is usually done by looking at one end of the wire, writing down the color codes (or bringing along the document with them) and verifying the order of colors at the other end of the line. In my experience, if the colors are all transposed at the second end (red-green-whiteblue to blue-white-green-red, for example), the technician will not notice nine times out of 10. I have seen this so often that I have used it as a test when interviewing technicians. The solution is simple; undo and reconnect both ends of questionable cables, or be aware of human nature and compensate.
New installations
A different set of rules applies when you troubleshoot a new installation. Once everything has been installed, if it doesn’t work, works partially, or dies after working perfectly for a little while, you can drop the problem into one of three categories: design, equipment or installation. I use a method outlined in the simplified pass-the-buck flowchart to select the category. (See page 20.) Generally, once you can figure out where to turn for help, you’re well on the way to solving the problem. When you run out of ideas, try not to run out of telephone numbers.
When installing equipment, always assume that you’ll be the one called to fix things at odd weekend hours. This assumption makes you do things that will make life easier. When you label harnesses and cables, assume that someone else will be reading those designations and make them sell-explanatory, perhaps providing a key or chart of explanation on the back of the rack door. What was crystal clear when you were in the thick of things can become awfully hazy a few days later.
It also helps to build test or service points into your installations. A full patchbay and a diagnostics system are a wonderful luxury, but in this day and age they are not always in the budget or among the customer’s priorities. Why should the customer care about repair? The customer won’t be repairing anything.
Sometimes you can add an extra set of in-line connectors here and there and save yourself time and headaches. If a cable will be inaccessible or is not connectorized at either end, consider running a service panel with pass-through connectors for key lines. In applications with long cable runs, I sometimes install service panels strategically in the field. They let a technician hook up a monitor without running up three flights of stairs to the control room every time a change is made.
As someone who has had the misfortune of working on many installations that used newly introduced equipment, I find myself relying heavily on the applications and technical-support departments of many vendors. I asked about some of the obstacles that these people face in their efforts to help the installation and repair process. A few universal suggestions prevailed:
- Don’t play favorites. When calling for help, we all like to talk to the last person who helped us, but in many tech-support departments, it
just makes the busiest people even busier.
Get to know everyone, and let them sort out who would be best to answer your questions. - Do explain your problem very briefly to the first person that answers and to each person you talk to until you are sure you’ve reached the
person who can help. This will keep you from getting testy by the time you actually reach the right person and will ensure that the one who
is helping you gets all of the details. - Don’t ignore any suggestions, no matter how inane they may seem. Listen to and consider their advice seriously, just as you wish them
to take you seriously. They want to solve your problem as much as you do, and they understand that you will keep calling back until they
get it right, so work together. if they suggest anything — even jumping up and down and doing an antler dance — it is simply because it has
worked for others in the past. - Do call back and let them know how things worked out. If they solve your problem, or if it turned out to be something altogether different,
share the knowledge. You might help them with the next person calling, and that person could be me.
What has changed
When something stops working or doesn’t work as expected, my first point of locus is to see what has changed. This is another area where the proper documentation can save your skin. When servicing a system, put some sort of service record near the equipment as well as in your shop. Note changes carefully. A change could be as simple as moving a plug or reattaching a ground wire that was hanging in the rack. This could be a big mistake if someone else had disconnected the wire or unplugged the connector to solve a ground-loop problem and neglected to document the work. The technician tidying up and putting things back where they belonged might well think, “I didn’t do anything,” and the casual observer would never spot the change.
In this wonderful microprocessor-based age, we find software or firmware on much of the equipment we use. With control systems, upgrades are frequent, and different versions of the software can often be found in otherwise identical components. If you are performing an upgrade, note the version on the outside of the case and save the old software.
A manufacturer might provide a software upgrade that will solve a problem you are not having or that will create a feature your customer doesn’t want. Often, these upgrades and bug-fixes create new problems to replace the ones they have solved. Thus, the ability to back up to the previous software version can be a lifesaver. In the quest to provide your customer with the latest and greatest, it is often easy to forget the adage, “If it ain’t broke, don’t fix it!”
Environmental issues
Temperature and environmental changes can play a tremendous role in equipment reliability, but once you get past the obvious — clogged filters, dusty equipment and the ever-present cigarette haze — problems can be tricky to pinpoint. Many times the operating conditions are different from those experienced during maintenance and repairs, and these differences are not always apparent to the technician.
A little detective work can help here. Questions like, “Is it always this cold in here?” can provide unexpected results. One large CCTV installation I worked on had terrible intermittent system crashes that we couldn’t trace. When we asked that question, we found out that there was a 15° temperature swing between the day and night shifts. (Some like it hot, it appears.) Because the change was over a brief shift-change period and occurred on a daily basis, we looked at how well the circuit boards were seated in their card cages. Sure enough, we found our answer: The constant temperature fluctuations were causing the boards to flex as they expanded and contracted. The problem was not the temperature but rather the constant changes.
Finally, there is no substitute for seeing the problem in action. Years ago I installed a sound system for live music in a bar. It looked and sounded great when installed and was a textbook example of installation neatness and efficiency. The customer hated it, however, complaining of strange rumbling noises whenever the bar got busy. Try as we might, we could never duplicate the problem on a service call. Finally, one busy Saturday night, I went to listen for the phantom noises, fully expecting an apology from the bar owner when I proved that it wasn’t the sound system making the noises.
It turned out that I was the one making the apology. At that time, spring reverb units were the rage, and this one was rack-mounted. The equipment rack was bolted to a wall common to a dishwashing machine that was only fired up when things got really busy. Needless to say, when the dishwasher started to vibrate it sounded like thunder in the distance.
The funny thing was, once the customer found out what the problem was, it didn’t matter to him any more. He even joked about it to his patrons and friends. He was more frustrated by my inability to see the problem and by the feeling that I didn’t believe him when he said there was a problem.
Sometimes when we can’t see or solve a problem right away, we forget that, as corny as it sounds, “The customer is always right.” Better to go in with that attitude, confident that any problem can be fixed, than to have our valuable clients find someone else who believes in them.
