Revival and LTE saves the Industrial Cellular World

For a long time, I have been out of the industrial cellular scene - laid off by one employer who felt my industrial focus was not useful in the modern world, and hired by a new employer who disagreed.

GPRS began to vanish from usability 4 years ago, and despite Verizon's "promise" that 1XRTT will survive for eternity, all sane folk understand that eventually, the cell towers need to focus on the MOST band for the BUCK.

LTE is a unique opportunity, as for once the entire cellular world is focused on a single HW and radio spec. While LTE tends to highlight uber-speed cellular (like 400Mbps/sec), there is of course efforts to support the low-end, the sub-1mbps tank gauging solutions.

Some of you may have heard of the LTE 'CAT' ratings, such as CAT 6 or CAT 1, CAT 1 is targeted at devices needing less than 10mbps, and a future CAT 0 less than 1mbps. I believe a key concept to understand that people who call GPRS or 1XRTT 'cheap' fail to understand that ultimately, it is how many paying customers can 'pay' for data in one instant that matters. So eventually, cellular data technology which allows MORE customers to send tiny amounts of data in less time benefits the cellular companies greatly.

Adding GPS to a GOBI 3000 cellular module

GOBI 3000 Cellular Module

Digi products such as the ConnectPort X4 or WAN 3G which support the GOBI 3000 cellular module include the ability for direct GPS support. However, without a special antenna, you will find the signal weak and the GPS only semi-reliable. This may be good enough for a static device which never moves, because as long as the device can get a signal once within the first few days, then you'll know where it is.

However, most of Digi's GOBI-enabled products include two SMA connectors for two cellular antenna. CDMA can directly use two antenna, but GSM only uses one, ignoring the second. You can remove one of the two antenna, installing a PASSIVE GPS antenna with a male SMA connector. You'll still need a clear view of the sky.

Manual Position

Don't have a clear view of the sky? Don't want to use a special antenna? Most Digi gateway allow you to manually enter a static position. For example on the Digi ConnectPort products, you can enter the latitude and longitude manually on the Configuration > Position web page.

You can obtain the information from a cell phone - sadly, most mapping apps show you a Google Map, but not the coordinates. On my Andorid devices, I use Yigiter's "Sensor Data Monitor" app (github.com/yigiter) as it gives me the raw GPS information in real-time.

USB / Serial Modules

Alternatively, many of the gateway (for sure any of the ConnectPort with a host USB port) can support some USB-based GPS units such as the USGlobalStat BU-353 (available at amazon.com).

Some of these gateways also support RS-232 models, which is listed under the Serial Profiles as GPS.

Be Careful with 'Assisted GPS'

Many cellular modules have GPS capability, and support up to 3 modes:

1. Standalone (no network assistance): the hardware tries its best with GPS satellite signals captured through the cellular antenna path. Depending on your antenna, the signal could be difficult to obtain and hold, but it won't add anything to your cellular data costs.  A fix might take as long as 12 minutes after power up or gaining clear sky view.
2. Mobile-based (network assisted) or MSB: again, the local hardware is calculating the position, but obtains various data files by billable network traffic. In MSB, the hardware must detect viable satellite packets and the network help is primary to speed up the initial fix, plus help when fewer satellites are visible to the device.
3. Mobile-assisted (network calculated) or MSA: is like MSB, but will likely have more billable network traffic. In this case, the mobile device literally forwards corrupt GPS satellite packets and other information to a network-based server, which (over-simplified) can use pattern-matching with good packets supplied by other GSP-MSA devices in your neighborhood. The network server literally offers error-correction with the very worst of satellite conditions.

The problem is that the billable network traffic amount is pretty weakly defined. Some of the network traffic is free, while other traffic is billable.

Now an example: I had a Digi ConnectPort X4 IA with a Gobi3000 cell module. An early firmware incorrectly allowed the cell module to default to GPS active in MSB mode, which wasn't a huge problem with I had the Gobi module configured for GSM/AT&T, as I had a good cell signal in my odd inner-corner office. I have been told the Gobi 3000 'ephemeris' data file & related data is about 40K, and is loaded after a cell module restart, and then refreshed once a day (I was told by word of mouth only!) 

However, after 6 months of happy GSM operation, I reconfigured that unit for CDMA/Verizon to test some CDMA issues, and at my desk the CDMA signal is a much worse than GSM. A few days later I started getting 'usage alerts' from my carrier warning that my daily limit of 100K was being breached. Some investigation lead me to discover that my 10MB per MONTH cell data plan was now seeing nearly 8MB per DAY!

Fortunately, I asked a fellow engineer about this, and he'd seen the problem at a large customer site.  Apparently, the AGPS subsystem in my unit was failing to fetch the ephemeris data file from Qualcomm's web site, and retrying quite aggressively.  This retry was creating the 8MB of unwanted data every day for enough days that I began seeing the alerts!

Solution? Well, technically I didn't even need GPS, plus even if I did, since the unit runs 24/7, the faster initial fix wasn't of much value. My solution was to set GPS to standalone mode. In the end, the overage cost me about $34. Without the alert, I would not have detected the overage until my bill arrived, and then likely would have seen (at my $2/MB overage charge) about $500. Image if I had 100 units doing this for the same month!

Moral of the story: Be very deliberate about GPS use within a cell module; make sure the mode selected is correct, or even turn it off if you have no need for it.  Of course, also make sure your usage alerts are enabled at your carrier!

The future of GPRS (update)

In May 2012 I created a blog entry on the future of GPRS and Industrial Automation, so here is an update for February 2013.

First, AT&T has officially announced its shut down of 2G / GPRS will be complete by the end of 2016.

• Why? To free up radio bandwidth for 4G technologies, which makes much better use of that scare resource, so more customer-bang for less-bucks
• What does this mean? Between now and then, customers using GPRS will increasingly suffer more and more problems. AT&T officially says GPRS customers with problems can contact AT&T and they will try to sustain suitable GPRS support, however I've already heard from customers who say the answer from such contact is "Sorry, in your situation, you'll need to switch to 3G".

Second, even if AT&T is trying to maintain existing GPRS service, they are unlikely (or certainly NOT?) adding new GPRS hardware to towers. For example we had one customer in northern Minnesota who checked with AT&T, confirmed AT&T had home-network service in an area, but upon equipment installation, it turned out there was no GPRS support - the AT&T towers in that area where NEW, so did not have any GPRS hardware. This customer could not use their six existing GPRS-only products, instead they had to buy six new sets of 3G cellular hardware.

Controlling Overages on Your Cellular PLC Account

Over the years, I cannot count how many irate customers I've heard of who scream "We expected a $50 bill last month, but it was $3000!"

Sadly, it is their own fault, not mine, not the maker of the cellular device. 

They are at fault for NOT making use of the tools their cellular carrier provides - or they have the WRONG carrier if the carrier lacks such tools!  Any good cellular carrier must allow you to set up daily limits, which cause an email or text message to be sent if breached.

For example, this week one of my $14-per-month devices was on it's way to cost me $182 in overages.  What saved me?  The right tools doing the right job! 

My cellular devices are handled by Kore Telematics  and I make proper use of the tools which they give me.  The device in question has a 10MB account, yet I normally see from 50-80KB of traffic per day, which is 2-3MB per month.  Therefore I turn on 2 alerts within my Kore Telematics web portal - I am warned if the device uses more than 100KB in a single day, or if it is on track to use over 3MB per month.  It recently did so.  When I received the email alert, here is the DAILY summary, where I have circled the start of the offending days. 

Oct 21st had 1168KB of traffic, with over 1000% too much.  I had accidentally enabled a function within the device which was creating traffic I neither wanted nor valued.  The 1.168MB listed is actually only for 2/3rds of the day, so a full day of traffic would be closer to 2MB, and clearly my 10MB cell plan can't sustain 2MB per day every day of the month! 

I turned off the offending feature, and my usage will drop back to the expected range.

Here is the Kore monthly summary, which is estimating the monthly total.

The What-If:
Now, consider what would have happened if I had NOT seen this alert.  There were 10 days left in October, so I'd have an extra 20MB of traffic on my 10MB/month cell plan - so probably 13MB of overage since I have a 7MB 'buffer' to begin with (I expect 3MB of 10MB to be used).  On my 10MB plan, I pay $7 per MB for overages.  So 13MB of overage in October would cost me $91.

However, I would not have seen the cellular bill until the 10th or 11th of November, so another 10 days and 20MB would have already been logged in the month of November. Had I noticed the bill when it came and fixed the problem on the 11th of November, then I still would have had roughly a $91 overage on the November bill (20MB + 3MB - 10MB = 13MB).

Moral of the Story:

Make sure your cellular carrier gives you access to usage alerts - and use them!

One Device for both GSM or CDMA

Can you have one cellular device which works with AT&T, T-Mobile, Verizon, Sprint and others?  The answer is: Yes.

I have begun cellular tests of new units of Digi ConnectPort X4 and Digi TransPort WR21, both of which include a GOBI 3000 cellular module.  This means if I select AT&T and reboot, it 'converts' the radio to use AT&T compatible frequencies and firmware, while if I select Verizon or Sprint, it 'converts' to work with those networks.

Okay, what is the catch?

1. Modest cost increase.  I'm not sure what, but the GOBI module requires a fairly large FLASH to hold multiple firmware images, plus special radio hardware.
2. Slower reboots - so say some customers.  Since the cell module has more software-controlled hardware, it does take longer to start up.  I have not done any testing here, but eventually I will produce some numbers.

The carrier change goes surprising easy - one just selects AT&T, Verizon, Sprint (etc) on the mobile configuration web page and reboot.   That is it. 

Digi Support Documentation: Gobi 3G Cellular Carrier Conversion Guide

Digi is also working on a Python script which can be used with a 'test unit' activated on multiple networks (for example, the 3 big ones).  The script can be run for a few days and records various statistics - radio strength, connection delays, upload throughput.  This will be very useful for a customer with perhaps 100 sites to install, and they have reason to believe that no single carrier will be available at all 100 sites.

Holding open TCP sockets on cellular

A reader asked me to offer suggestions for keeping sockets open long-term - for example, you might open a connection to a status/event ASCII printer port and then on the host collect events.

First, let me say up front that it will be impossible to ever see such a cellular-based TCP socket remain open 'forever', which in this context is say a year or more.  Cellular just does not work that way.

Although not a scientific response, after my 6+ years of work with cellular, I will guest-imate that the longest connection I have ever seen is probably about 20 days, and that the most common pattern is to be connected a few days (2 to 5), then a day with several disconnects and reconnects.  If your application cannot function in this scenario, then do NOT use cellular.

Things which hinder long-term socket opens:

• Environmental dynamics: for example a rain storm will interfere and drop signal quality.
• Load Sharing: the bottom line for cellular carriers is always "Make the most users as happy as possible", with a bias towards humans with smart-phones.  This causes:
• All users share bandwidth, so the more users are active, the less throughput and 'service' any of them receive.
• Since most human interaction is fast bursts of one-shot, always give priority to new sockets.  The longer your 'session' exists (aka: your TCP socket is open), the more likely it is to be dropped during high tower loads.
• Time-Of-Day issues: many cell towers literally shift (or point) in different directions during the day.  For example, when highway traffic is light they may point out over a residential or industrial area, but during heavy traffic they shift towards the roads.  Seems weird, but the goal is to maximize the bandwidth usage, so the tower always tries to point to the most active users.
• The 5M-KOD (5-Minute Kiss- Of-Death):  All idle TCP sockets should be assumed to vanish after being idle for 5 minutes.  You should always use TCP keepalive set to about 4 minutes, 50 seconds.

Cost considerations:

• Most carriers now round up per socket, per hour.  Why do they do this?  Because they can!  And because they bill in somthing called 'units', which are 1K blocks.  So if you look at your bill you'll see that a certain session started at for example 8:12AM, ended at 1PM and includes 47 units of data.
• This means if you hold 1 socket open for 1 hour, you will suffer at most a 1K data round-up per hour (or 24K per day).
• This means if you instead open a new socket every 5 minutes, then you open 12 'sessions' per hour and you will suffer at most a 12K data round-up per hour (or 288K per day).
• In both cases above I use the 'at most' phrase and do not attempt to average anything.  if your data is random, then fine - worst case is likely 1/2 of what I state, but unfortunately most industrial systems are moving predictable data each time.  So if you just happen to move 1 byte 'too little' each hour, then you'd get a wonderful 1 byte round-up - but if your standard size makes you 1 byte too large, then your roundup is always 999 bytes per hour.
• Opening/Closing a socket includes from 500 to 1000 bytes of overhead.
• A TCP keepalive includes about 100 bytes over overhead, however MOST TCP stacks only send the keepalive if the line is idle.
• Never poll at a once-per-5 minute rate!  If you really want once per 5 minutes, you really need to poll at for example once per 4 minutes and 45 seconds.  You must stay away from the 5M-KOD like the plague, or you risk all kinds of weird cell network issues.  Again, I will be unscientific, but state that since many vendors are involved in your link, you have a huge race condition at 5 minutes where part of the link is already down and part is still up for the moment.  And since the cellular system is NOT IP based, this increases you probability that the cellular link appears up to your cell module, yet is really broken & unusable.

 Suggestions to create a better TCP socket experience:

• Use UDP/IP when ever you can!  It has been shown in my own tests to cot up to 85-90% of your data costs!
• Accept that you cannot keep the TCP socket open forever.  If you wish to (for example) have a Dynamic IP and manually program the host to use an IP which changes each time the cell link drops - this is just not feasible!
• Ask, do you really need a socket open all the time?  My own tests have show that if you expect data less than once per 30 minutes you are probably better off to open/close the socket between uses.  The trade-off is the data cost to close/reopen the socket (plus the 'session round-up') verse the TCP keepalive required faster than once per 5 minutes to keep the socket healthy.

The future of GPRS and Industrial Automation?

I have been working more on local RF mesh than cellular, so my blog has been quiet.

However, users should be aware that GPRS in on the way out.  AT&T (and others) will swear up and down that they still fully support it, however we have customers reporting that their mobile (truck) based systems using GPRS are finding more and more dead-spots.

I agree that GPRS is not dead.  However, before investing in a GPRS-only cellular devices, consider the facts and your needs:

1. AT&T stopped certifying new non-3G devices years ago.
2. AT&T has been rumored to be reallocating air/RF-space away from GPRS to 3G technologies because the 3G makes much better use of this scarce resource - so more happy customers with 3G than GPRS. 
3. AT&T claims to have fully adjusted their pricing so that 3G is comparable to GPRS - they were forced to do this because A) they don't want more/new GPRS users, but B) huge customers were still going GPRS because it was cheaper.  So I have been told this is less true now.
4. AT&T claims to not be killing GPRS ... yet.

So if you roll the above facts together, I speculate that what is happening is that more GPRS devices are being squeezed into sharing less bandwidth, so whereas before a dozen GPRS devices might share 5 or 6 units of 'resource', now they might need to share 1.

If true, this means that the devices have lost bandwidth in recent years, so are now running on a fraction of what they had a few years ago.  The net result is that devices which upload say 200 bytes a day likely won't notice a difference, however anyone trying to open a fancy web page might discover the task nearly impossible.

GPRS runs at from 9600 to 80kbaud, but devices are allotted 'slots', so think time-division-multiplexing & other forms of fair sharing.  So if a tower offers 28.8kbps service to one GPRS device, and 3 more GPRS devices come online, then in theory they all may see closer to 7kbps throughput.  So the device sending a single small UDP or SMS packet won't notice, while the device which requires successfully sending 350 large packets end-to-end will notice a huge drop in performance, and if the client/server have 'task timers', they may discover that they can no longer complete the task before the connect drops or task-timers expire.

So bottom line - AT&T has not yet killed GPRS, but they clearly have it lined up against the firing-wall!  People who have tiny data needs can still safely use GPRS if their ROI window is fairly short-term (a few years).  However, people wishing to EVEN OCCASIONALLY move a 1/2 Meg file for firmware update or diagnostics may find GPRS less and less usable.  Those who plan to routinely move even dozens of K-bytes of data might begin to see reliability problems, so should move to 3G devices.

Increasing cellular robustness for Modbus

A customer of ours is creating a custom Python program in one of Digi's cellular gateways. The gateway uses Modbus/RTU to poll a flow-computer, then based upon a time cycle or abnormal events, the gateway uses HTTPS and XML to push data up to a central server.

They have noticed that at times the gateway has trouble connecting to the server for hours at a time, so wondered what the best solutions are to increase success. Their code buffers the data, so no data is lost.

The customer had asked about using a Windows PC to PING the remotes, but I don't think that will help.

Mobile-originated (meaning the gateway calling out to cellular) will be more robust than mobile-terminated (meaning Windows calling remote). Therefore the best solution will be for the cellular gateway to work a bit harder to connect out.

First, a few technical details:

• Most IP-based cellular devices maintain a full-time IP connection. However, the cell tower 'parks' the device when it is not talking, which means the device loses any radio channels and can only talk after requesting and WAITING for reallocation of bandwidth. And no, there is no way to compel your cellular provider to preallocate full-time a scarce radio resource to a device that is not paying $4000 per year for full-time data movement.
• This means there is always a delay when a device starts to send data after being idle. I have witnessed this delay being up to 18 seconds long, but it could be longer. Not all TCP/IP clients expect to see such long delays in the connect/socket-open phase.

So assuming the customer's problem is unexpected long delays causing the gateway to give up too soon, the best suggestion is to prime the pump early. Lets say your gateway sends data once per hour, so when your Python task wakes up to send data, then do NOT try to open a TCP socket right away. Instead, first send a garbage UDP/IP packet to your host, then sleep another 15 seconds. It does not matter if the UDP packet fails. You do not need to enable UDP on your host, nor watch for a good or error response. The only goal is to force the cell tower to start allocating radio bandwidth 15 seconds before your task actually tries to create your TCP/IP socket.

This pre-allocation may solve your problem when the problem is caused by slower allocation of radio bandwidth. Since most cellular providers now round up data traffic to the 1K charger per hour, sending an extra 30 bytes per transaction should not increase costs.

However, the problem may NOT be radio bandwidth, but authorization through a chain of roaming service providers. This is especially true for low-cost GPRS/GSM providers. Your data is passing through 2, 3 or more companies, and although they may all have agreed to allow the IP session (the PPP session) to open, one of them might drop the ball and block your data access in a way which the PPP session doesn't detect. The only way to recover is to completely flush and rebuild the PPP session - which in plan English means renegotiate with the base carrier to be assigned your IP address. It simulates a reboot of the product.

Fortunately, Digi includes a low-level feature called SureLink, which can use a PING packet to confirm end-to-end connectivity. You can (as example) configure the Digi to PING an internet IP once per hour (or 'X' hours), then physically reboot the cellular module if access fails. This nicely forces a complete renegotiation of the PPP session, which costs you nothing.

Just be aware that use of DNS can be costly and error-prone. A single DNS lookup can cost over 1K of data plan. For SureLink, you literally want to use a static IP as the target to minimize the overhead costs and false-negatives where things other than PPP health block success.

Sorry about the move

Google/Blogger decided to change the way they interact with externally hosted blogs - probably a reaction to some search-engine abuse since blogger blog 'index' very well under Google.

(This post is a test, plus to move down the redirect message)

This blog has moved

This blog is now located at http://iatip.blogspot.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://iatip.blogspot.com/feeds/posts/default.

How Long to Obtain Cellular IP?

Well, I finally have a SIM, so I can actually do cellular tests again. It is a GSM 'Capped Plan' with 20/MB of data per month for $30. The 'capped' means that even if I go over the 20MB by dozens of GB, the maximum charge will be $130. It is a truly 'unlimited plan' for that $130. What's the catch? If I hit the 'Cap' two months in a row, then the carrier either cancels the account or forces me to start paying full-rate for a normal 1GB or 5GB/mo plan.

The 20MB is fine for me since I am testing methods to keep data small.

My current tests have a sleep devices waking, then obtaining a cell-link. For example, I am working on a Digi gateway called an X3 (not on Digi's web site yet). The X3 can power itself down completely, using a real-time clock chip to wake it at a future time.

I ran a test for 5 days, waking the X3 device every 10 minutes and clocked how long it too for the GSM carrier to assign an IP to the X3:

• The fastest time was 23.7 seconds (half a minute)
  
• The slowest time was 97.0 seconds (1 and a half minutes)
• The average time was 28.6 seconds (so most near 30 seconds, with a few long outliers)
  
• Note that there will be several seconds of CPU start-up time in there - I can only know the RTC wake-up time (which I loaded into the RTC last cycle) compared to the RTC time when the IP address is assigned.
  

Why is this important? Because the cellular modem consumes a LOT of power for the duration of this variable time. It is one of the variable which makes estimating battery life difficult.

My next test will include actually doing work - probably sending a Modbus/TCP transaction to a DSL-based server.

Cellular DNS Lookup Cost

I have been working with a customer using two cellular devices with a DDNS provider, so the 'Client/Master' device used a DNS name to connect to the remote 'Server/Slave' device.

Cost #1 - added delay
The first cost they were seeing was added time lag to open a connection.

• To open a TCP socket between two idle (but connected) cellular devices requires each device to be "unparked" by their cell tower. So the client device would need to request a resource and be assigned it by the local cell tower - this can require 2 to 5 seconds.
• Next, a DNS query would be sent and a response would be required. Some cellular carrier DNS servers don't seem very peppy, so this could add a few more seconds.
• Then the client device would receive notice from the cell-tower control channel to request a resource and be "unparked"
• Finally, after "X" seconds of delay and lag the client device would receive confirmation that the socket is open (or it might have given up to fast already & aborted).

So using a DNS name instead of an IP Address might add a few seconds delay - which might be just enough to cause problems. The customer I am working with has a client tool with a hard-coded MAX timeout of 25 seconds, and it seems to often hit this during original socket open. So they have no trouble communicating once the TCP socket is open, but it can be difficult to get the first data packets moving.

Cost #2 - added traffic
Of course the DNS query also adds raw traffic. DNS uses UDP, and a query might be 44 bytes and a response 60 bytes. So 104 bytes extra to open a TCP socket (which can be hundreds of bytes by itself).

Is this traffic important? Perhaps. Perhaps Not. Most cell plans today round up traffic hourly, so an extra 104 bytes for DNS and 150 bytes for a Modbus poll/response is still "1K of traffic".

For TCP/IP, the DNS lookup would only occur for a socket open. So regardless of the DNS name Time-To-Live, you might only see this added 104 bytes once per day or month - or anytime you open/reopen a socket.

However for UDP/IP you MIGHT see this DNS lookup for every packet. For example, a DDNS record from dyndns.org might expire in 60 seconds, so sending one 100 byte UDP packet every 5 minutes, could become 204 bytes per 5 minutes, so 2448 bytes per hour rounded up to 3K per hour or 72K per day and NOT the 49K expected.

So while UDP/IP can save a lot of traffic cost over TCP/IP, if short-lived DNS names are required then some of UDP's savings will be lost.

One solution is a custom application to do indirect DNS lookup. In other words, to directly lookup the DNS name within the application, then to ignore the DNS "Time-to-Live" and use only the cached IP address UNTIL COMMUNICATIONS FAILS - then do a new lookup. This is not how normal operating system DNS caches work - they literally honor the DNS server's time to live and would cause a new look up based on the 60 second time-to-live. Yet this is general safe for cellular end-devices since normally the dynamic IP only changes when the cell link is lost, so the remote IP is unlikely to change more than a few times per day.

Does Cellular 3G help your Rockwell PLC Access?

In short - No.

Cellular data systems are continuously being sub-optimized for the unidirectional fire-hose paradigm of web page viewing, media streaming, image viewing, email download and so on. In other words, you only see the 3G "broadband" performance when you have tens of thousands of bytes to shove in one direction.

Actual field tests of RSLogix 5000 downloading to ControlLogix processors show that using 3G is only a few percent faster than standard technology - plus still probably slower than analog dialup. The problem is caused by RSLogix and Ethernet/IP moving thousands of small transactions in a half-duplex manner. So RSLogix downloads the PLC object by object, confirming success after each object is written. With cellular latency, each of these transactions can take up to a second to complete, so multiple 1 second by 3000 to 8000 transactions and you have your PLC download time.

3G apologists will claim "Oh, but latency in 3G is much lower than older, coarser technologies."

That may be, but when the rubber meets the road (aka RSLogix5000 downloads), 3G doesn't perform any better. 3G might have other value, but for most people moving data "faster" just means your data usage and the final bill racks up faster as well.

Cellular to Redundant Rockwell ControlLogix

Can RSLinx access a redundant pair via cellular?

Background
A Pair of Rockwell Automation ControlLogix racks with SRM Module and dual ENBT's will share a pair of IP addresses. One IP address is "the primary", and the other IP is "the backup" (if/when it is online). When the ENBTs switch role, they will issue the requisite Gratuitous ARPs to cause other local Ethernet devices (like a Digi cellular gateway) to update their ARP cache, thus comprehending that the IP-to-MAC address mapping has changed. Thus a NAT/Router forwarding to the primary IP should handle the fail-over with only modest bumps.

Config Details
A user desiring RSLinx (and RSLogix or RSView etc) to access a remote Rockwell ControlLogix (any RA/AB PLC) will be doing what the industry calls "Mobile-Terminated" access. The user needs to arrange a cell plan which offers either a fixed IP address to target, or at least a Dynamic DNS name to target (like tk101.iatips.com or panel2.digi.com). This is NOT what you obtain with an iPhone or personal air-card data plan. Those will have private IPs which only permit outgoing connections - called "Mobile-Originated". That was a buzzword lesson - expect to be asked about those two terms when you ask about cellular data plans!

So once you can arrange your targetable IP or DNS name, you need a cellular router such as one of the Digi Connect WAN family products. My favorite model today is the ConnectPort X4, but that has large memory for Python programming, wireless mesh and other goodies you won't need to link up your Rockwell PLC (Hey, I said it was MY favorite - doesn't mean it has to be yours!)

Note that contrary to folklore or urban legend, all cellular devices need certification to work on a system - even GSM devices. Many small suppliers get around this by including fine-print that say the device buyer is responsible to arrange such legalities, and since you (the buyer) don't read such fine print the salesperson will just say "Heck, it's GSM - so it is allowed everywhere world wide!" Deal with this issue as you see fit, but Digi has more formal certs in more countries worldwide than any of the other industrial players.

But back on subject, when the gateway comes up, it is assigned your known IP or DNS name. This is exactly how your home or business DSL/T-line line works. Yet when RSLinx tries to talk to the gateway on Ethernet/IP's well-known TCP port of 44818, the gateway will reject the connection as a weird attempt at hacking. You need to instruct the gateway:

1. to not reject the Ethernet/IP traffic on TCP port 44818

1. to instead forward it to a local IP on the Ethernet - which would be the IP of your ControlLogix ENBT (or the primary IP of the redundant pair)

The details of how this man-in-the-middle fake-out works is fascinating (to me), but quite a pile of text. If you are interested, this older blog entry goes through the NAT/Router details blow by blow. But bottom-line, your ENBT receives the RSLinx packet and needs to have its own Gateway IP set to the Digi gateway's local Ethernet IP address. Free hint: 9 out of 10 guys who call saying "Why can't RSLinx see my PLC through your Digi gateway?" have failed to set the correct Gateway IP in the PLC/ENBT.

So assuming your gateway and PLC are setup correctly, then targeting the RSLinx "Ethernet Devices" driver (with timeouts slowed down to 30 seconds) will cause your PLC's little icon to show up. With RSLinx running, you will create up to 200MB of billable cell traffic per month doing absolutely nothing - so don't leave it active. Note that the "Ethernet/IP Driver" won't work as it requires UDP broadcast, which can't be routed over the Internet.

At this point you'll say "Cool, now can I see my backup ControlLogix or a second PLC?", and the simple answer is "No." One of the realities of RSLinx and AB PLC is that the Ethernet/IP protocol is hard fixed to only the TCP/IP port 44818, and the NAT/Router can only forward TCP port 44818 to a single local PLC. The easy fix would be for Rockwell to change RSLinx to enable adding both an IP/DNS name and TCP port number - then the NAT/Router could forward TCP port 44818 as 44818 to the primary ENBT, TCP 44819 fixed-up to 44818 to the secondary ENBT, TCP 17256 (a random number :-]) fixed up to 44818 to an RSView panel and so on. Because the NAT/Router can restore all traffic to 44818 on the local Ethernet, RSLinx is the only tool needing to change.

Will Rockwell ever do this? It would take a programmer half a day to do - then a few weeks to test - then a few months to document and forestall support headaches. So who knows. They might. They might not.

But bottomline is a simple cellular NAT/Router can be used to talk to a pair of ControlLogix running in a redundant configuration - you will just be limited to seeing only the primary ENBT and the primary IP address.

Cellular Antenna and your PLC

One of my customers is having fun with one of their own customers. My customer uses Digi gateways running a Python application to collect hourly tank levels, which are fetched by cellular once per day. The tanks hold 250-gallons of a chemical additive which is injected at a variable rate into crude oil pipelines. Small battery-powered ultrasonic level sensors (www.massa.com) push the last 8 hourly samples every few hours using wireless IEEE 802.15.4 (aka Zigbee). The end customer's goal is to forecast when the tanks need to be refilled ... and I don't mean just receive an alarm when it is running low, they literally want to plan truck routes in advance.

Fun stuff.

The problem is that one of the pilot sites has bad cell coverage, meaning many days the central system cannot upload the log. Of course the data eventually it all uploads since it is held for over a month. Now, never mind that this system sits down in a dry wash (a small valley), the end-user says "Hah, that's because carrier A*** stinks; we all love carrier S****." My customer does not want to mix carriers - especially since negotiated pricing between the two in question is so different.

So remember that "coverage" for fixed RTU must be viewed differently from "coverage" for mobile uses.

For a mobile device (or user) "coverage" is defined by the probability that a valid carrier connection will be available as the device moves around. Thus the number of towers (etc) is important, and if there is no signal in one spot, then hopefully will be one a mile or two away.

Unfortunately, our little RTU bolted to a power pole in that gully won't be moving a mile or two ever. So in the end, "coverage" for a fixed device is defined only by the tower(s) which can be seen. So the carrier with the best cell-phone coverage might not be the best carrier to support a particular fixed RTU.

The first step will be to move the RTU up out of the gully, which is easy since the data signals are all wireless and power can be tapped from any of the privately owned power poles. Probably carrier A*** (stinky or not) will work fine once the RTU moves. As plan B and make the end-customer feel listened to, a S****-based RTU will be placed on the same pole as the relocated A*** one.

Could they run out a long external antenna? Sure, they could. But why bother? Put the RTU where it has a nice signal. Even in 2 or 3 store buildings we suggest customers put the cellular router up in the roof-area and run the Ethernet UTP it's 100 meters instead of trying to deal with the signal loss in long antenna cables. We even have customers using 900Mhz Ethernet bridges to link a cellular router placed where it must be placed back to the Ethernet devices which want access.

Worst case, a directional yagi antenna could be used, however you need to understand that cell towers are routinely turned off without warning. The carriers are truly geared towards mobile users who expect bad signals in some places. Towers (or the active elements) also move. Most of the cell towers you see along the highway work like strip-malls; a company owning the tower and supplying power leases tower space to a mix of carriers. This allows everyone to be flexible.

So prematurely locking down a cellular device to a single tower with a directional antenna can cause future problems since it will not see other weaker towers should the targeted tower be turned off or even moved.

Ethernet/IP PCCC Service Codes

Summary: the Rockwell PCCC Service Codes (or how to get a 1761-NET-ENI or Digi One IAP to send specific DF1 DST/SRC bytes)

A customer had a product which acted as an Ethernet/IP originator (client) and needed to talk through a Digi One IAP to feed DF1 into a DH+ gateway. The catch is the DH+ gateway used the DF1 destination byte as the DH+ node address - and so far they could only make the Digi One IAP spit out a destination = 1. This made for pretty boring and unprofitable DH+ design!

(Sorry: I cannot explain "How to write an Ethernet/IP stack" in a single blog entry, so this entry assumes you already have a product speaking Ethernet/IP and just want to tweak how you talk to remote DF1 devices.)

The PCCC object (code: hex 0x67) is not part of the ODVA spec - it is a vendor specific object used by Rockwell/Allen-Bradley to talk to SLC5, PLC5E and MicroLogix PLC. I won't fully explain it here ... not much to explain; do a wireshark trace of a ControlLogix talking to a SLC5/05 or MicroLogix 1100 and you'll see all you need to see about the simple object.

The default service code of 0x4B has this form:

Exec PCCC Service	4B
IOI to PCCC Object	02 20 67 24 01
Originator Info	07 03 85 50 4d 41 41
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

The only mystery here should be the "Originator Info". The 07 means a total of 7 bytes (including the 07). 0385 is a unique sequence number which you should change between messages. The 504d4141 is a "CIP Serial Number" which must be unique to your vendor id - most vendors just use the last 4 octets of an Ethernet MAC address.

However, there is NO remote node or slave address info, so the Digi One IAP or 1761-NET-ENI will create a DF1 message for destination=1 and source=0. So back to the customer's question ... how to force a DF1 message with say destination=5 and source=2?

The easiest solution is to switch to service code 0x4C, which has this form:

DH+Like Service	4C
IOI to PCCC Object	02 20 67 24 01
DH+ Like Header	00 00 02 00 00 00 05 00
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

So the "Originator Info" has been swapped with an 8 byte structure of the form AA AA BB XX CC CC DD XX. The "XX" are control bytes of some sort - just leave 0x00. "AA AA" is the Destination Link; "BB" is the Destination node; "CC CC" is the Source Link; "DD" is the Source node.

Voila - both the Digi One IAP and the Rockwell 1761-NET-ENI will now create a DF1 message with destination=5 and source=2. Not too painful, was it.

To my knowledge all Rockwell Ethernet/IP PLC accept either code, and if you'd like to see the service 0x4C in action just set your RSLogix5000 MSG block to use "CIP with Source ID" and the ControlLogix switches from using service 0x4B to using 0x4C.

For completeness, there is a final service code of 0x4D with this form:

Local PCCC Service	4D
IOI to PCCC Object	02 20 67 24 01
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

It is documented in DeviceNet gateway manuals to reduce total byte count - but it also lacks any destination or source info so is of limited use in Ethernet-based systems.

Creeping Firewalls and Change

Summary: Network infrastructure changes can create unexpected havoc (surprised?)

We had a customer come in with a complaint recently about some old products - a 4-port terminal server (a TS4) running Modbus bridging firmware. Things had gone on smoothly for years, but suddenly they started having problems which power-cycling the units solved ... obviously it's a Digi problem, right?

Well, I'm not going to run through all of the issues, but I'll highlight one which will affect more and more people over time. Remember the old days - first you had dumb dialup modems, then you had to pay a lot for various error-detection standards, and now all analog modems have a big chip which does a hundred different standard things and you CANNOT buy any low-cost analog dial-up modem which is any dumber than state-of-the-art.

Such creep occurs in Ethernet switches also - once hubs were cheap and switches cost a king's ransom. Now even the "cheap" $19 home switches purchased in big-box stores are auto-sense, auto-cross, auto-everything. The same things happens - once the features are rolled into software or firmware, future revs of the product inherit such features for free ... said another way, it is often cheaper to sell lots of one powerful model than handle the marketing and support costs of a dozen models of various power and feature-set.

Well, this customer had been upgrading their wide-area-network infrastructure - a tangle of routed IP-based leased lines, old analog lines, radio/microwave links and so one, and as part of the improvement the various router/PPP end-points had added stateful awareness of packets ... part of an increase in US government security demands for utilities. After all, when you have hundreds of raw Ethernet ports scattered across rural American protected with little more than a padlock and chain-link fence, one has to consider the possibility of someone trying to 'jack' back into the 100% private network from a remote location.

The issue with such an addition is that routers with firewalls commonly default to a 5-minute TCP life rule. Every stateful firewall creates a table entry for every TCP or UDP activity going "out" from a safer network and into a wilder, less safe network - your home DSL/Cable router does the same thing. NAT (etc) is optional, but bottom line is the firewall needs to understand and agree to the context of every packet trying to pass through.

For a TCP socket this usually means at least 1 packet (data or ACK or Keepalive) must move every 5 minutes to refresh the context and inform the firewall that the TCP socket is still authorized. If no packets are seen, the fire wall discards the context and the socket is now ignored and 100% blocked without comment. Of course both TCP peers might think the socket is alive and well, but they will never be able to use it again.

So back to our TS4 off in some remote field - a host computer has 4 TCP sockets open to it through various routed subnets and for whatever reason the host doesn't send any new packets for 6 minutes. The TS4 has no reason to talk since it is connected to Modbus slaves. So 6 minutes later the host issues 4 Modbus requests on 4 sockets, which hit the stateful firewall. The firewall looks up the context (called forwarding or trigger rules in some systems) and discovers there is no existing context for these TCP packets. Well, once upon a time the firewall would just assume since these were safe since they came from the safe side, and thus auto-create a new context, yet modern stateful firewalls might NOT do this since it was a common hacker exploit to fool firmwalls to move malformed packets. Thus modern security demands that the first packet of any new TCP socket be a [SYN] packet. All four packets are silently discarded as unknown (note: this is a configurable behavior in most firewalls - to auto-recreate a context for an 'old' TCP socket continuing or to only allow this for new TCP sockets.)

The TS4 never sees the four new request; the host sees neither an [ACK] nor a [RST] of the TCP socket, so it retries after 3 seconds, then 6 and so on. Eventually it comprehends the TCP socket has died and opens four new TCP sockets. These are allowed through the firewall since they contain the correct TCP state as being new sockets. The TS4 sees the four new socket requests and now has eight (8) sockets connected.

Now you might say "What a minute - the four old sockets were closed!". Well, yes - the HOST knows they are dead and closed them, but the TS4 does not know this. Six minutes later, the same is repeated and the TS4 now has 12, then 16, then 20 sockets open. Eventually the TS4 runs out of resources, and the customer's open recovery option is a hard reboot of the TS4. This is of course worse if the host only occasionally takes longer than 5 minutes to poll, since the increase in TS4 sockets could take an unpredictable amount of time to build up.

How to solve this problem?

1. The first line of defense in the modern IP age is to always, ALWAYS enable TCP keepalives with an aggressive 4 minute 30 second idle time in any product you install. Even if you don't have stateful packet inspection within your IP network today, that doesn't mean the next time someone replaces a router or serial PPP end-point you won't gain one. Unfortunately the Digi products usually default to a 2 hour TCP Keepalive - which is also disabled (for historical reasons). Yet having a 4.5 minute keepalive would have saved the TS4 because the first time the host delayed longer than 4.5 minutes to poll, the TS4 would have sent a TCP keepalive packet back through the stateful firewall, which the host returns and the firewall's context for this socket is refreshed. Thus when the host does finally send the next Modbus poll after 6 minutes (1.5 minutes after the TCP keepalive exchange) the firewall is satisfied with the packets and forwards them out to the TS4. Everyone is happy. Plus even if the TCP socket has been aborted by the firewall, the TS4's TCP keepalive will be silently discarded and the TS4 will retry and eventually comprehend that the TCP socket is no longer valid. This is the purpose of TCP Keepalive - to allow a TCP peer with no data to move to test (and refresh) the health of an idle connection.
   
2. The second possibility is unique to the Digi IA/Modbus application. You can enable a setting referred to as "idletimeout" on incoming client or outgoing server connections. Unlike the TCP Keepalive which create traffic and keeps an idle socket open, the idletimeout literally just aborts an idle socket without giving it any chance to prove it is healthy. So setting a 5 minute idle timeout in the TS4 would cause it to just assume any incoming Modbus client (master) connection which has NOT sent a new request is bad. This setting would also have saved the TS4, forcing the four old sockets to be aborted before the TS4 had a chance to build up a herd of dead sockets.

So understanding TCP Keepalive is a critical skill for all modern industrial Ethernet users, after all more and more facilities are breaking their floors and functional area into distinct subnets with stateful firewall protection to keep the Accountants from changing the color of the widgets being produced - and to keep the production engineers from printing on the accounting departments laser printer which uses red ink :-).

More complete discussion of TCP Keepalive is here (one of dozens of web site you can find in a web search).

Tunneling Serial Data over Cellular

Tunneling Serial Data over Cellular

Question: you have a Windows-based application which expects to talk over normal serial ports - how can you connect to a remote serial device over a cellular-IP link?

Products: works with Digi Connect WAN, WANIA or WANVPN, Digi ConnectPort VPN, X4 or X8

Answer: Our Digi RealPort driver for Windows 2000/XP/Vista dated Nov 2008 now supports a nice low-overhead "UDP: Serial Data Only" mode. This will cost a fraction of what normal TCP-mode Realport or competitors TCP redirectors will cost.

Here is a web page explaining how to install RealPort for UDP mode (DDNS names can be used!)
http://digitips.wikispaces.com/Digi+Realport+with+UDP

Here is a web page explaining how to set the WANIA to UDP Sockets.
http://digitips.wikispaces.com/Digi+UDP+Sockets+WAN

I set this up yesterday and have been polling a Modbus/ASCII serial slave on my CPX4 using RealPort and UDP mode. Responses take from 2 to 3 seconds to return … in part because I am polling so slowly that the modem PARKS in between every poll and I wait 30 seconds for a response timeout. I see perhaps 0.25% error / no response timeout but then my SIM doesn’t have the best signal where my product sits.

However, I still believe people need to be realistic – even in the situations where the slave response times out I’d suggest NOT retrying immediately since the retries have a higher than normal probability of failing as well. Missing a few polls a day is better than needlessly paying more for data plans.

Evolution of Data Plan Billing

Summary: the big three have moved away from unlimited data, towards limited data.

It is interesting - I once (as in last year) had a talk with a potential partner who'd been at some European conference and was convinced the world was on the verge of low-cost (sub-$20/month) unlimited cellular data plans. We were discussing the creation of report-by-exception tools to reduce SCADA costs, and this partner's strong faith in this belief caused them to eventually bail out of the talks, saying "In a year or two, no SCADA company will care about how much cellular data they use."

Yet as of the summer of 2008 the world of cellular data is moving in the opposite direction. Last year the big three (AT&T/Sprint/Verizon) offered "Unlimited Data" for personal users with the Service Terms listing a VERY narrow list of permitted activities - mainly email and web browsing, with many common things like file download/upload, media-streaming prohibited. So when ever one of the big three would cut off a user for moving too much data on an "unlimited plan", the service provider would fall back on the "You are doing prohibitted things, thus impacting our network, thus take your business elsewhere". What a way to cause bad feelings, eh? Note that this change is CONSUMER plans - machine-to-machine have always been limited, priced by the MB/month without rollover, plus with charges for data overages.

Now all three have dropped the price from the $80/month range down to $60/month range ... but added a hard limit of 5GB per month. Isn't free & vigorous market competition wonderful?

Sounds reasonable - 5,000 megabytes of data is a lot, yet this doesn't mean 5GB of data transfer. It means 5GB of metered activity, with many activities I've studied including up to 95% overhead. Thus someone only moving 20-30MB of real data in small packets per month might hit pretty close to their 5GB limit! My experience with normal wide-area-network traffic hints that a real PC user doing simple email and web-browsing once a day would probably move 1-2GB of data before hitting the 5GB total activity limit.

To paraphrase the wireless data service terms for all three:

• Data transport is always measured in full kilobytes
• Actual transport is always rounded up to next full-kilobyte at "end of session"
• Network overhead and resend requests caused by network errors can increase measured kilobytes.
• 2 of 3 mention always rounding up to nearest kilobyte every hour period.
• All warn that you will NOT receive an itemized detail of how your charges are calculated; you will NOT see which services were used or during which time periods the charges were inccurred under.
  

So if I send a single 50 byte UDP/IP packet, is that a full session and billed as 1024 bytes? Could be under this language since UDP is 'sessionless'.

Hmm, the term session is pretty ambiguous. Perhaps it means per "time you enable your PC-based cellular data card." That seems likely - plus if you left your device on twenty-four hours a day then the once per hour round-up would catch you.

I'm afraid I haven't offered any new answer here, other than to suggest you understand that low-cost unlimited data plans ARE NOT just around the corner ... at best we left them behind last year and I don't foresee them ever returning. I suppose all three now understand that huge new profits are to be made with these 5GB limits, which will cause many "super-salesman" using their cellular data plan daily to spend an extra $50 to $500 in monthly overage charges.