A couple more thoughts about detection diagnostics

This is a continuation of the earlier post on detection diagnostics, at:

http://ntcip-unleashed.blogspot.com/2012/02/controller-detection-diagnostics.html

The detector settings need to be reviewed to make sure that the global settings are not creating undue consequences.

In one recent example, a left turn pocket had moderately high traffic volumes, and the stopbar detector was a 60-ft long quadrapole.  The length of the quadrapole created a situation where there was never a gap in the traffic, so the standard stopbar (standing queue expected) timing of 5 minutes max time / 10 seconds detector fail time, created a situation where the timer for the left turn signal gapped out cycle after cycle.

This was not a new signal, but one where the operation had been changed from one brand of signal to another brand of signal as a part of a systemwide upgrade.  The settings were consistent with the previous brand's operation, but some changes were made in the conversion.

In looking at the new operation, it appeared that the signal was gapping out consistently, but there was a static queue forming.  The signal was gapping out because the detector fail mode was only placing a call for 10 seconds, and if the traffic was over the advanced detector, then the siganl would extend for time beyond the 10 seconds.

After observing the operation for a short while, it was apparent that the signal was not getting enough time for the movement.  Determing the reason for this operation was complicated by the fact that the 60-ft quadrapole also had NTCIP Queue mode turned on.

The detector diagnostics operate on the raw feed of the detector into the controller, not on the processed controller operations of the detector.

After observing the left turn pocket's operation via video feed, some modifications were made to the detector diagnostics from 5 minutes max time / 10 seconds detector fail time, to 15 minutes max time / 20 seconds fail time.

Once the timing changes were made, the left turn pocket cleared out in a few cycles, and began operating as expected.

The moral of the story is that the signal operation needs to be monitored after changes are made.

Why Communicate To Your Signals.

This may seem obvious.  But what needs to be communicated with?  What is appropriate to spend the extra money to purchase the optional Ethernet port on?  When two products are available that meet the specification, and one has data logging capabilities (and costs more), is it worth getting the device with the data logging?

Controller Communications

This is the most obvious one to connect.  Connecting the controller allows for remote uploading and downloading of the controller database, plus monitoring of the controller’s current operation.  If the signal system includes a central system, then the signal can be configured to provide data on scheduled intervals back to the central system.

The data fed to the central system can include alarms, traffic counts, current status of the controller (what phases are green, yellow, red etc.) and other information.  In some cases, the central system can be set up to log this data.  The central system that I use logs a great deal of information.  

One key piece of information that is logged, is the time served for each split division.  This is done whether the signal is in coordination, or free.  This is logged as historical information, so it is easy to run a report to look at what was happening recently for the amount of time served for any or all movements at the signal.  This can be helpful to determine what a reasonable cycle time may be for a signal.  It can also be helpful in determining if there is a problem with a signal.  Citizens will call and make statements about long waits for a green, or that a green was never served for a side street.  I have had phone calls where a citizen stated that they waited 25 minutes for a green.  There may be some truth to that statement, but having the ability to look back on the split division logs allows me to confirm that there was, or was not a problem at the specific time that the citizen stated that they had the undue wait.

The logging of data also helps identify problems with the signal operation.  If a controller develops internal problems, that information may be part of the alarm stream that is reported back to the signal.  We recently had a controller develop a problem with the TS2 communications and the controller started showing thousands of SDLC detector failures an hour.  Since the detector failures are not a critical SDLC problem for the controller, the signal kept chugging along.  A quick review of the alarm logs showed that there was a problem with the signal.  The problem was manifesting itself by causing the signal to extend, and serve movements that had no cars.  Based on the field review of the equipment, it was determined that there was a problem with the controller, and the controller was swapped out.  The BIU’s were swapped out, and the problem continued.  The problem could have been the CPU or the SDLC communications on the 2070-2N connector.  Bench testing will help determine that.  But having the controller connected to the central system gave us the first piece of information that there was a problem.

Another key piece of information is when a signal goes into all-red flash.  When a signal goes into red flash, the central system knows this within a second or two, and within another few seconds the central system sends out an email to key personnel stating that the signal is in red flash.  In some cases, I have seen signals in red flash for several days before a citizen calls in the problem.

Monitor Communications

There is a new generation of monitors generally called smart monitors.  These include advanced features for data logging, communications ports, and special programming to accommodate things like flashing yellow arrow configurations.  Many agencies are still holding on to their old monitors.  The old monitors work, but may not provide the same level of information that the more modern monitors provide.

I have had citizen calls where the citizen states that they are waiting at an intersection, and they are never getting a green.  By checking on the central system, I can see what the system states the controller is telling the system.  By checking remotely on the monitor software, I can confirm that the field indications are consistent with the central system.  

The monitor software allows for remote viewing of the signal inputs and outputs, along with the line voltages.  The picture below shows the Reno AE software talking to a Reno 1600Ge MMU.  This is a static picture, so you can’t get a real feel for the information.  The line voltages are dynamically being updated.  The current greens, yellows and reds are showing up on the screen.

Reno A&E PC software view of monitor operation

Another feature of the monitor software is that when a signal goes into flash, the monitor software can provide very good information about why the signal is in flash.  Is the problem a controller issue, a cabinet issue – that requires somebody to replace a component in the signal cabinet, or a problem with an indication that requires a bucket truck?

Problem Controllers

In the example below, the prior faults log is shown for a Reno 1600Ge MMU.  In this example, the controller CPU was dying. 

Reno MMU reporting a Port 1 failure

A little explanation of the standards is probably in order here.  The NEMA TS2 standards require that if a controller exhibits a Port 1 failure 3 or more times in a 24-hour period, that the monitor latch the signal in all-red flash.  The Port 1 is the main port out of the controller that provides the SDLC communications.  The controller sends out command frames, and gets response frames from the peripheral devices.  This is a very important communications node.  A Port 1 fail would typically be a failure of specific command frames from the controller.  These frames are generally routed around the controller, Terminals and Facilities (T&F, also known as the load bay) BIU’s and the MMU.  When a Port 1 failure occurs, the monitor puts the signal in flash.  If the Port 1 failure clears, the monitor will allow the controller to automatically come out of flash.  If 3 port 1 failures occur in a 24-hour period (also called in some controllers as 3 critical SDLC failures in 24 hours), then the monitor will transition the signal into all-red flash and keep it there until a technician comes by and services the cabinet.

Because the TS2 standard requires that 3 3 port 1 failures in 24 hours occur before the monitor locks the system in all-red flash, you can have a situation where the signal will transition in, and out of flash on its own.  This is not like a CVM fault, or 24V fault, where you can put in jumpers on the MMU board to latch a CVM or 24V fault.  This is a programming issue in the monitor.  To deal with this, many controller software packages allow the user to modify this by having the controller internally latch the signal in red flash when 1, 2 or 3 (user specified) Port 1 failures occur in a 24 hour period.  This will allow the controller to keep track of its own problems, and upon seeing a Port 1 failure, keep the signal in red flash.  This should allow the controller to not bounce in and out of red flash multiple times before going into locked red flash.

In the example above, the controller’s CPU was not working properly.  The controller would lock up, and create a Port 1 fail which the monitor reacted to, then the controller CPU would restart itself, then lock up, creating another Port 1 fail, restart and create another Port 1 fail, at which time the MMU would latch the red flash.

In the attached example, you can see that the controller went into red flash 3 times 8:38 AM and 8:45 AM, then a technician cleared the controller failures, restarted the controller, and the signal worked normally until it went into 2 Port 1 failures (in and out of red flash 2 times) between 3:51 PM and 4:56 PM.  This looks odd, because it looks like it went into red flash twice before it locked this time.  The reality is that I had a replacement CPU in my hand, and had just opened up the cabinet door at 4:56 PM, and the signal went through its restart procedure.  I then placed the signal into cabinet red flash, replaced the CPU, and restarted the signal with the new CPU.

Determining Conflicts

Another example of how the monitor software can help determine what is going on is when a conflict occurs.  In the example below, phases 2 and 6 were green, and the monitor showed that phases 3 and 4 showed green and reds on simultaneously.  This may not be the best example to show, but this was a case where one of the field technicians was working inside a cabinet, and momentarily contacted the greens for phase 3 and 4 to ground, which caused the signal to go into flash.

Reno MMU reporting a conflict

 While this may not be the best example, it does show information.  I got the email from the central system that the signal was in flash, and then almost immediately out of flash.  I pulled up the monitor PC software, and looked at what the monitor said it was doing.  By the time I got the monitor PC software running, the signal was out of flash.  I called the signal tech and asked what was going on.  He was a little sheepish, but told me what had happened.  Nobody got hurt, and the tech got a good reminder that you need to be careful when working around the contacts in a cabinet.

Oddities

One thing that you may see occasionally is the signal going into all-red flash because of a short yellow.  This is rare.  Most traffic signal controllers do not want to have a yellow change interval less than 3.0 seconds.  Some controllers will allow yellow change interval settings at less than 3.0 seconds, but only if the controller is programmed in one place to allow less than 3.0 seconds, and in another place to specifically program a phase to have less than 3.0 seconds.  I have been a traffic engineer for a long time, and have found exactly zero signals where I would time the signal at less than 3.0 seconds of yellow change interval.

The MMU also monitors the yellow change interval.  In the event that the yellow change interval is less than 2.7 seconds, the monitor will place the signal into all-red flash.  This can also be overwritten in the monitor settings, but like the short yellow in the controller, I have found exactly zero signals where I would override this setting in the field.

Below is an example from a traffic signal controller in a cabinet that is in a test environment.  This was forced to a short yellow as a part of testing out some features in the controller and monitor. 

If for some reason the signal actually experienced a yellow of 2.7 seconds or shorter, the signal would automatically go into all-red flash.

Reno MMU reporting a short yellow fault

Having the communications in the field allows the technician to know what has happened to the signal before leaving the desk.  This helps the technician to determine what level of equipment and expertise will be able to fix the problem.

The last example for monitors shows a known problem with the TS2 specification.  This is a minor irritant from a maintenance standpoint, but it may be a bigger deal if you are a pedestrian.

Reno MMU reporting dual indication fault

The NEMA TS2 specification requires that the monitor look at the time that the Flashing Don’t Walk (FDW) is on, and off.  The specification essentially requires that the FDW be turned on for half a second, then turned off for half a second.  The monitors generally allow for some slop in the SDLC communications frames, by allowing extension either of the on or off to 6/10^th of a second.  This is because the controller is driving the outputs, and in the event that a frame is missed, the status of the indication won’t change.

Occasionally, the monitor will see the FDW on or off for 7/10^th of a second.  This is seen as a problem by the monitor, and the monitor transitions the signal into all-red flash immediately.  This extra 1/10^th of a second happens very rarely, but it happens.  In my experience, when a TS2-1 traffic signal is left in ped recall, this will occur about once a month, on one of the pedestrian movements that is in recall.

Video Detection Communications

Cameras fail.  Nature sometimes helps.  What appears to be a good viewing angle when you set it up may be very different at night, or in the rain.  

Cameras fail

The following three pictures show older generation cameras that have problems with the video feed images to the video processor.  When the cameras are not working properly, and feed bad information to the video processor, this defines the term “garbage in, garbage out”.

Problem camera.  Note that the picture is divided where the bottom of the screen shows the far advance, and the left side of the screen shows what should be on the right.  The video detection zones are correct, if the camera were working properly.

Problem camera.  Note how the picture is divided at the bottom of the screen.  The bottom of the screen shows what should be the far horizon of the camera.

Problem camera.  Note how the picture is divided about in the middle of the screen.  The bottom half of the picture should be above the top half.  The zones are correct, assuming that the camera is working properly

These cameras are being replaced.  The key here is, this signal was not connected to any system.  It was a lone signal, with no communications.  The only reason we knew about the problem is because a citizen got frustrated and called in that the signal wasn’t operating properly.  If this signal were connected to the system, we would have been able to look at the operation and see what was working properly and what wasn’t.

Once a week, or more frequently, I spend a chunk of time and pull up multiple signals and look at what they are doing.

In a past job, I found another unique type of problem.  The video detection was not working properly.  It didn’t take long to figure out what was causing the problem.  It probably would have taken a little less time, if I had already had my first cup of coffee before I started looking at the cameras.  In that situation, the video detection cameras were mounted on Astrobrac connections to the mastarms.  During the night, the Astrobrac straps slipped on the mastarm, and the camera swiveled from being on top of the mastarm to being on the bottom of the mastarm, hanging down instead of up.  This caused the camera to go from the normal looking at traffic approaching eastbound to the intersection, to upside down, looking at traffic departing the intersection westbound.  The stopbar detection that the camera was driving didn’t work so well.

Sometimes Nature Helps

Below is a video I took where we had a spider who created a web over the video detection camera lens.  You can see how the spider is causing false calls to the controller.

Below is a video of a poorly placed camera, but showing how the glare of headlights is causing false calls on the detectors.

The key here is to watch the detection zones for the cars traveling north (almost straight up) in the picture.  The glare from the southbound headlights does not cause false calls in the two northbound detection zones.

The two northbound detection zones are count stations.  I experimented with how to deal with the oblique angle, and headlight glare.  The remote video feed allowed me to modify the detectors and try things to see what happened in multiple weather and lighting scenarios.  

There are actually two detectors in each northbound lane of travel.  The large, visible, box is a directional detection zone.  This box is looking only for traffic heading northbound.  Any glare from a southbound vehicle’s headlights don’t cause a true condition in the Boolean logic in the detection scheme.  When a northbound vehicle causes a True condition in the box, and then the vehicle crosses an invisible line at the far north end of the box, then the detection system places a call.

Having remote communications to the video detection system allows me to test a variety of ideas in vehicle detection strategies, and observe them in various weather and lighting conditions.  Based on the glare on the wet pavement from the overhead luminaires I moved the detection zones around into the glare, out of the glare, and observed how the system operated.  Based on this, I was able to hone in on a detection strategy for this particular brand and model of video detection system that works for multiple situations.

Conclusions

These are just a few examples of how communicating to the field equipment can provide significant benefits.

The main benefit is being able to see what is going on from your desk.  In my case, I can VPN into the system and see what is going on.  Last year, I was visiting family in Idaho, and received a call that there was a problem with a signal that was under construction.  I pulled out my laptop, connected it to my MiFi card, and was on the system within 5 minutes – from Lewiston Idaho.  It took about another 3 minutes to figure out that the contractor had shifted traffic from the normal lanes to another area of the pavement that had no detection.  I modified the radar stopbar detection and in about 15 minutes total, I was back on the phone to the construction manager telling him what I had done.  He reported to me that the signal appeared to magically begin working properly.

Magically.  I like that.  Magic is how things are explained that happen mysteriously that we can not explain with other knowledge or understanding.

Loops vs. Radar and Video Detection

Most of the sales reps in my area have figured out that I like technology.  I am willing to put technology in place in intersections.  One of the new buzzwords in traffic signals is radar detection.  Even better, the neat new technology is combining radar and video detection in one device.  There have been multiple generations of radar and video detection.  Some have worked really well.  Some, not so much.

This post is not attempting to sell a product, rather a general discussion of what why some technologies work well beyond the technical specifications. 

I currently deploy video detection and radar detection for count stations, stopbar detection and advanced detection.  It is true that nonintrusive detection (radar, video detection etc) cost more to implement than traditional loops.  There are several advantages to nonintrusive detection.  These include, but are not limited to increased flexibility, immediate gratification, long term cost savings, ease of repair and other factors.

Flexibility

The easiest way to envision flexibility is that someone always has the idea that they should restripe the road.  Move the lanes around to create a bike lane, squeeze in a turn pocket, repurpose an approach to create dual lefts, or some other operational modification.  The non-intrusive detection allows for a decreased cost for reallocating the detection to the new lane configurations.  If the vehicle detection is all induction loops embedded in the pavement, then someone has to pay for the traffic control and labor to cut in the new induction loops in the new locations. 

One particularly troublesome intersection that vexes me has no detection at all for a left turn pocket with a protected left turn (it was cut out a long while ago and never replaced),  and another approach where the lane lines were restriped that ended up with the old left turn lane loop detection still being used, but it was on the lane line separating the left and thru lane.  If this had been video or radar detection, these zones could easily be reprogrammed in the detection units.

For some reason, engineers have designed lots of bike lanes, and in many cases, have no detection in the bike lanes.  Worse yet, in too many cases, the bike lane detectors are configured badly, where you have a small (I have seen 2-ft wide by 10-ft long quadrapoles as bike detection) loops in the bike lane at the stopbar.

Several states now require that traffic signals detect all legal vehicles.  A well designed bike loop can detect a steel bike (through the ferrous metal affecting the inductance field) or an aluminum bike (where the aluminum causes an eddy current in the loop that causes a change in the inductance field), well designed loops have problems at best sensing a carbon fiber bike.  By extending the radar stopbar or video detection zones into the bike lane, the bike is sensed by the signal.

Immediate Gratification

There appear to be a dearth of loop cutting contractors.  Regularly, traffic signal turn ons occur where the signal is ready to be turned on today, but the loop cutters have not been scheduled yet.  It is not unusual for a signal to be turned on, and operated in pretimed mode for an extended period of time before the loops get cut in and terminated.  In many cases for developments, the requirement is for the signal to be fully functional, but the political decision is made that getting the signal turned on, going green, yellow and red is good enough.  The opening of the development can occur even though the signal is not fully functional.  Once the developer has the occupancy permit, the traffic engineer has no hammer to get the loops cut in.  The developer is usually capable of making the straight faced argument to the elected officials that the loop cutter is just a short time away, so, please let the business open.  I understand the argument by the developer.  They have a significant capital investment in the property.  The business has a significant capital investment in getting to the point where they are ready to open.  The loops can’t be cut in until the asphalt is placed…  The asphalt is placed two days before opening, then comes striping… It was wet over the weekend, so the asphalt is late, the striping is late…  loops get the short shrift.

I have had traffic signals that have operated in pretimed mode for 3 months or more, between when the signal was turned on, and when the loops were finally installed and terminated.

By going to video or radar detection, all of the vehicle detection installation can take place when the poles are erected, weeks to months before the opening day.  We work with the contractor to aim the cameras and radar detection panels early.  In many cases, we go out with a generator and set up the video or radar detection systems after the cabinet is installed and wiring is pulled and terminated, but before the electrical service is connected to the transformer.  That means on the day of turn on, all we are doing is refining the detection zones.

Another form of immediate gratification is where utility work cuts through loops, or loop lead ins.  We had an intersection where a water line blew out immediately below the southbound stopbar loops for a major intersection.  In order to repair the water line, the stopbar loops for all 4 lanes of the southbound approach had to be cut.  Between the immediate utility work, patching, then removal of patching, and replacement of the pavement, the loops were reinstalled about 2 months after the water line blew out.  During that entire 2-month period the signal was in recall for the left and thru lanes, 24 hours a day.  It is pretty straightforward to take a coordinated traffic signal, and put the main street coord phase into recall.  During the hours of the day that coordination is in effect, the drivers will most likely not see any difference in the signal operation for that particular movement.  During the non-coordinated hours of operation, this will make the signal very sluggish.

The left turn lane is a different thing.  Putting a left turn lane into recall for 2 months is painful, especially when the opposing thru is very heavy.

We have Ethernet communications to all of our signals that have video and radar detection.  Where we have had utility work at signals with radar and video detection, we are able to quickly modify the detection zones, and observe that the changes are working well. 

Long Term Cost Savings

Video and radar detection systems likely cost more than initial installations of induction loops.  However, over the long term, the nonintrusive vehicle detection systems will pay for themselves.

Grind and Overlay

There are several methods to maintain the pavement surface.  When the decision to grind and overlay the existing asphalt this can be accomplished several ways.  One way is to grind out the entire pavement surface, the other is to retain as much of the pavement surface as possible, but to grind out a wedge of asphalt approximately 2-inches deep at the curbline, and taper to no grinding at 10 to 12-ft from the curb.  This second method is quite common where the pavement is in relatively good shape, but the desire is to keep the top of the curb at 6-inches above the gutter elevation.  It is not uncommon for a paving project to include significant patching, where small to large sections of the pavement are completely cut out to below the subgrade, and replaced full depth.

The specifics of the pavement replacement method is not really important to the loops.  What is important to understand is that even if the standard plan shows that the loop wires be at 3-inches below existing surface of the pavement, when the top 2 inches of asphalt are ground out, the grinding will most likely pull up the sealant, which is bonded to the loop wires running from the loop to the junction box.  When the sealant, which is bonded to the loop wires is pulled up, the pavement grinder will shred the loop wires, typically between 4 and 1 feet from the gutter line – where the grinding includes a wedge of asphalt removal.

As an aside, it is important to understand that if loops are replaced above the existing loops, the old loops will create false calls if the old loops are not dealt with properly.  If the old loop is not fully cut in multiple locations, then the shredded loop wires near the gutter line will allow the old loop to couple and decouple below the new loops, which will cause false calls on the new loop.  This will occur most frequently when the pavement gets wet.

On one grind and overlay project I worked on, the grinding took out the main street loops on a 5-lane Principal Arterial at 6 traffic signals.  Replacing the loops at these 6 traffic signals cost over $100,000.  All of the existing loops worked properly on that corridor, but when the asphalt wedge was ground out, the grinding shredded every loop wire at or near the gutter line.  The loops themselves were not affected, just the wires between the loops and the splices in the junction boxes.

Four years after the loops were installed due to the grind and overlay, the same road is going to have a sanitary sewer trunk line installed, which will require replacement of most of the loops… again.  If the vehicle detection systems had been video, or radar, instead of loops, the initial grind and overlay, and then the upcoming sewer work would have not required the replacement of all those loops, and the cost associated with replacing the loops. 

Underground Utility Work

Recently, I was approached by the local sewer utility, where they had four locations where minor sewer work needed to be accomplished.  Each of the four locations had 200-ft or less of sanitary sewer to be installed.  The problem was, that in each of the four locations, the sanitary sewer work was going right through a bunch of stopbar loops.  Each location was going to require between $4,000 and $6,000 worth of loop replacement.  In each case, the cost of the loop replacement will need to be passed on to the sewer rate payers.

Once again, if the vehicle detection systems were radar or video, instead of loops, the cost of replacing the vehicle detection would not be borne by the citizens.

Ease of Repair

When a loop shows problems, it could be any number of issues.  The problem could be any combination of one or more of the following:  the loop itself has gone bad, the splice between the loop and loop lead in wires is bad, the loop lead in wire is bad, the field wires are not firmly connected to the terminal screws in the cabinet, the loop amplifier has gone bad…

There is a similar laundry list of things that can go bad for video or radar detection.

The ease of repair comes from replacing an induction loop that has gone bad in the street, compared to replacing a camera or radar panel.  To replace a camera or radar panel, the detector needs to be replaced in the same location, usually with a bucket truck.  This likely requires that one lane be coned off for a relatively short period of time to replace the bad detector with a new detector.

When a loop, or series of loops goes bad, there needs to be multiple vehicles, equipment, and personnel out in the street for extended periods of time.  This causes one or more lanes of traffic to be routed around.  The wire from the loop to the junction box needs to be cut and laid across multiple lanes of traffic, which requires multiple sets and resets of traffic control.  Work needs to be done at the curb, for where the loop’s sawcut goes into the conduit that leads to the junction box.  There is a great deal of risk associated with all of this activity in the street.

Going from an in-pavement detection system to an out of pavement detection system reduces the risk associated with traffic maintenance operations.

Conclusion:

Induction loops can be more accurate than video or radar detection.  There are short term and long term benefits to seeking other methods of vehicle detection than induction loops.

In a later post I will discuss the benefits of different types of radar detection that far exceed the ability of induction loops.