Revival and LTE saves the Industrial Cellular World

For a long time, I have been out of the industrial cellular scene - laid off by one employer who felt my industrial focus was not useful in the modern world, and hired by a new employer who disagreed.

GPRS began to vanish from usability 4 years ago, and despite Verizon's "promise" that 1XRTT will survive for eternity, all sane folk understand that eventually, the cell towers need to focus on the MOST band for the BUCK.

LTE is a unique opportunity, as for once the entire cellular world is focused on a single HW and radio spec. While LTE tends to highlight uber-speed cellular (like 400Mbps/sec), there is of course efforts to support the low-end, the sub-1mbps tank gauging solutions.

Some of you may have heard of the LTE 'CAT' ratings, such as CAT 6 or CAT 1, CAT 1 is targeted at devices needing less than 10mbps, and a future CAT 0 less than 1mbps. I believe a key concept to understand that people who call GPRS or 1XRTT 'cheap' fail to understand that ultimately, it is how many paying customers can 'pay' for data in one instant that matters. So eventually, cellular data technology which allows MORE customers to send tiny amounts of data in less time benefits the cellular companies greatly.

Adding GPS to a GOBI 3000 cellular module

GOBI 3000 Cellular Module

Digi products such as the ConnectPort X4 or WAN 3G which support the GOBI 3000 cellular module include the ability for direct GPS support. However, without a special antenna, you will find the signal weak and the GPS only semi-reliable. This may be good enough for a static device which never moves, because as long as the device can get a signal once within the first few days, then you'll know where it is.

However, most of Digi's GOBI-enabled products include two SMA connectors for two cellular antenna. CDMA can directly use two antenna, but GSM only uses one, ignoring the second. You can remove one of the two antenna, installing a PASSIVE GPS antenna with a male SMA connector. You'll still need a clear view of the sky.

Manual Position

Don't have a clear view of the sky? Don't want to use a special antenna? Most Digi gateway allow you to manually enter a static position. For example on the Digi ConnectPort products, you can enter the latitude and longitude manually on the Configuration > Position web page.

You can obtain the information from a cell phone - sadly, most mapping apps show you a Google Map, but not the coordinates. On my Andorid devices, I use Yigiter's "Sensor Data Monitor" app (github.com/yigiter) as it gives me the raw GPS information in real-time.

USB / Serial Modules

Alternatively, many of the gateway (for sure any of the ConnectPort with a host USB port) can support some USB-based GPS units such as the USGlobalStat BU-353 (available at amazon.com).

Some of these gateways also support RS-232 models, which is listed under the Serial Profiles as GPS.

Be Careful with 'Assisted GPS'

Many cellular modules have GPS capability, and support up to 3 modes:

1. Standalone (no network assistance): the hardware tries its best with GPS satellite signals captured through the cellular antenna path. Depending on your antenna, the signal could be difficult to obtain and hold, but it won't add anything to your cellular data costs.  A fix might take as long as 12 minutes after power up or gaining clear sky view.
2. Mobile-based (network assisted) or MSB: again, the local hardware is calculating the position, but obtains various data files by billable network traffic. In MSB, the hardware must detect viable satellite packets and the network help is primary to speed up the initial fix, plus help when fewer satellites are visible to the device.
3. Mobile-assisted (network calculated) or MSA: is like MSB, but will likely have more billable network traffic. In this case, the mobile device literally forwards corrupt GPS satellite packets and other information to a network-based server, which (over-simplified) can use pattern-matching with good packets supplied by other GSP-MSA devices in your neighborhood. The network server literally offers error-correction with the very worst of satellite conditions.

The problem is that the billable network traffic amount is pretty weakly defined. Some of the network traffic is free, while other traffic is billable.

Now an example: I had a Digi ConnectPort X4 IA with a Gobi3000 cell module. An early firmware incorrectly allowed the cell module to default to GPS active in MSB mode, which wasn't a huge problem with I had the Gobi module configured for GSM/AT&T, as I had a good cell signal in my odd inner-corner office. I have been told the Gobi 3000 'ephemeris' data file & related data is about 40K, and is loaded after a cell module restart, and then refreshed once a day (I was told by word of mouth only!) 

However, after 6 months of happy GSM operation, I reconfigured that unit for CDMA/Verizon to test some CDMA issues, and at my desk the CDMA signal is a much worse than GSM. A few days later I started getting 'usage alerts' from my carrier warning that my daily limit of 100K was being breached. Some investigation lead me to discover that my 10MB per MONTH cell data plan was now seeing nearly 8MB per DAY!

Fortunately, I asked a fellow engineer about this, and he'd seen the problem at a large customer site.  Apparently, the AGPS subsystem in my unit was failing to fetch the ephemeris data file from Qualcomm's web site, and retrying quite aggressively.  This retry was creating the 8MB of unwanted data every day for enough days that I began seeing the alerts!

Solution? Well, technically I didn't even need GPS, plus even if I did, since the unit runs 24/7, the faster initial fix wasn't of much value. My solution was to set GPS to standalone mode. In the end, the overage cost me about $34. Without the alert, I would not have detected the overage until my bill arrived, and then likely would have seen (at my $2/MB overage charge) about $500. Image if I had 100 units doing this for the same month!

Moral of the story: Be very deliberate about GPS use within a cell module; make sure the mode selected is correct, or even turn it off if you have no need for it.  Of course, also make sure your usage alerts are enabled at your carrier!

The future of GPRS (update)

In May 2012 I created a blog entry on the future of GPRS and Industrial Automation, so here is an update for February 2013.

First, AT&T has officially announced its shut down of 2G / GPRS will be complete by the end of 2016.

• Why? To free up radio bandwidth for 4G technologies, which makes much better use of that scare resource, so more customer-bang for less-bucks
• What does this mean? Between now and then, customers using GPRS will increasingly suffer more and more problems. AT&T officially says GPRS customers with problems can contact AT&T and they will try to sustain suitable GPRS support, however I've already heard from customers who say the answer from such contact is "Sorry, in your situation, you'll need to switch to 3G".

Second, even if AT&T is trying to maintain existing GPRS service, they are unlikely (or certainly NOT?) adding new GPRS hardware to towers. For example we had one customer in northern Minnesota who checked with AT&T, confirmed AT&T had home-network service in an area, but upon equipment installation, it turned out there was no GPRS support - the AT&T towers in that area where NEW, so did not have any GPRS hardware. This customer could not use their six existing GPRS-only products, instead they had to buy six new sets of 3G cellular hardware.

Controlling Overages on Your Cellular PLC Account

Over the years, I cannot count how many irate customers I've heard of who scream "We expected a $50 bill last month, but it was $3000!"

Sadly, it is their own fault, not mine, not the maker of the cellular device. 

They are at fault for NOT making use of the tools their cellular carrier provides - or they have the WRONG carrier if the carrier lacks such tools!  Any good cellular carrier must allow you to set up daily limits, which cause an email or text message to be sent if breached.

For example, this week one of my $14-per-month devices was on it's way to cost me $182 in overages.  What saved me?  The right tools doing the right job! 

My cellular devices are handled by Kore Telematics  and I make proper use of the tools which they give me.  The device in question has a 10MB account, yet I normally see from 50-80KB of traffic per day, which is 2-3MB per month.  Therefore I turn on 2 alerts within my Kore Telematics web portal - I am warned if the device uses more than 100KB in a single day, or if it is on track to use over 3MB per month.  It recently did so.  When I received the email alert, here is the DAILY summary, where I have circled the start of the offending days. 

Oct 21st had 1168KB of traffic, with over 1000% too much.  I had accidentally enabled a function within the device which was creating traffic I neither wanted nor valued.  The 1.168MB listed is actually only for 2/3rds of the day, so a full day of traffic would be closer to 2MB, and clearly my 10MB cell plan can't sustain 2MB per day every day of the month! 

I turned off the offending feature, and my usage will drop back to the expected range.

Here is the Kore monthly summary, which is estimating the monthly total.

The What-If:
Now, consider what would have happened if I had NOT seen this alert.  There were 10 days left in October, so I'd have an extra 20MB of traffic on my 10MB/month cell plan - so probably 13MB of overage since I have a 7MB 'buffer' to begin with (I expect 3MB of 10MB to be used).  On my 10MB plan, I pay $7 per MB for overages.  So 13MB of overage in October would cost me $91.

However, I would not have seen the cellular bill until the 10th or 11th of November, so another 10 days and 20MB would have already been logged in the month of November. Had I noticed the bill when it came and fixed the problem on the 11th of November, then I still would have had roughly a $91 overage on the November bill (20MB + 3MB - 10MB = 13MB).

Moral of the Story:

Make sure your cellular carrier gives you access to usage alerts - and use them!

One Device for both GSM or CDMA

Can you have one cellular device which works with AT&T, T-Mobile, Verizon, Sprint and others?  The answer is: Yes.

I have begun cellular tests of new units of Digi ConnectPort X4 and Digi TransPort WR21, both of which include a GOBI 3000 cellular module.  This means if I select AT&T and reboot, it 'converts' the radio to use AT&T compatible frequencies and firmware, while if I select Verizon or Sprint, it 'converts' to work with those networks.

Okay, what is the catch?

1. Modest cost increase.  I'm not sure what, but the GOBI module requires a fairly large FLASH to hold multiple firmware images, plus special radio hardware.
2. Slower reboots - so say some customers.  Since the cell module has more software-controlled hardware, it does take longer to start up.  I have not done any testing here, but eventually I will produce some numbers.

The carrier change goes surprising easy - one just selects AT&T, Verizon, Sprint (etc) on the mobile configuration web page and reboot.   That is it. 

Digi Support Documentation: Gobi 3G Cellular Carrier Conversion Guide

Digi is also working on a Python script which can be used with a 'test unit' activated on multiple networks (for example, the 3 big ones).  The script can be run for a few days and records various statistics - radio strength, connection delays, upload throughput.  This will be very useful for a customer with perhaps 100 sites to install, and they have reason to believe that no single carrier will be available at all 100 sites.

Holding open TCP sockets on cellular

A reader asked me to offer suggestions for keeping sockets open long-term - for example, you might open a connection to a status/event ASCII printer port and then on the host collect events.

First, let me say up front that it will be impossible to ever see such a cellular-based TCP socket remain open 'forever', which in this context is say a year or more.  Cellular just does not work that way.

Although not a scientific response, after my 6+ years of work with cellular, I will guest-imate that the longest connection I have ever seen is probably about 20 days, and that the most common pattern is to be connected a few days (2 to 5), then a day with several disconnects and reconnects.  If your application cannot function in this scenario, then do NOT use cellular.

Things which hinder long-term socket opens:

• Environmental dynamics: for example a rain storm will interfere and drop signal quality.
• Load Sharing: the bottom line for cellular carriers is always "Make the most users as happy as possible", with a bias towards humans with smart-phones.  This causes:
• All users share bandwidth, so the more users are active, the less throughput and 'service' any of them receive.
• Since most human interaction is fast bursts of one-shot, always give priority to new sockets.  The longer your 'session' exists (aka: your TCP socket is open), the more likely it is to be dropped during high tower loads.
• Time-Of-Day issues: many cell towers literally shift (or point) in different directions during the day.  For example, when highway traffic is light they may point out over a residential or industrial area, but during heavy traffic they shift towards the roads.  Seems weird, but the goal is to maximize the bandwidth usage, so the tower always tries to point to the most active users.
• The 5M-KOD (5-Minute Kiss- Of-Death):  All idle TCP sockets should be assumed to vanish after being idle for 5 minutes.  You should always use TCP keepalive set to about 4 minutes, 50 seconds.

Cost considerations:

• Most carriers now round up per socket, per hour.  Why do they do this?  Because they can!  And because they bill in somthing called 'units', which are 1K blocks.  So if you look at your bill you'll see that a certain session started at for example 8:12AM, ended at 1PM and includes 47 units of data.
• This means if you hold 1 socket open for 1 hour, you will suffer at most a 1K data round-up per hour (or 24K per day).
• This means if you instead open a new socket every 5 minutes, then you open 12 'sessions' per hour and you will suffer at most a 12K data round-up per hour (or 288K per day).
• In both cases above I use the 'at most' phrase and do not attempt to average anything.  if your data is random, then fine - worst case is likely 1/2 of what I state, but unfortunately most industrial systems are moving predictable data each time.  So if you just happen to move 1 byte 'too little' each hour, then you'd get a wonderful 1 byte round-up - but if your standard size makes you 1 byte too large, then your roundup is always 999 bytes per hour.
• Opening/Closing a socket includes from 500 to 1000 bytes of overhead.
• A TCP keepalive includes about 100 bytes over overhead, however MOST TCP stacks only send the keepalive if the line is idle.
• Never poll at a once-per-5 minute rate!  If you really want once per 5 minutes, you really need to poll at for example once per 4 minutes and 45 seconds.  You must stay away from the 5M-KOD like the plague, or you risk all kinds of weird cell network issues.  Again, I will be unscientific, but state that since many vendors are involved in your link, you have a huge race condition at 5 minutes where part of the link is already down and part is still up for the moment.  And since the cellular system is NOT IP based, this increases you probability that the cellular link appears up to your cell module, yet is really broken & unusable.

 Suggestions to create a better TCP socket experience:

• Use UDP/IP when ever you can!  It has been shown in my own tests to cot up to 85-90% of your data costs!
• Accept that you cannot keep the TCP socket open forever.  If you wish to (for example) have a Dynamic IP and manually program the host to use an IP which changes each time the cell link drops - this is just not feasible!
• Ask, do you really need a socket open all the time?  My own tests have show that if you expect data less than once per 30 minutes you are probably better off to open/close the socket between uses.  The trade-off is the data cost to close/reopen the socket (plus the 'session round-up') verse the TCP keepalive required faster than once per 5 minutes to keep the socket healthy.

The future of GPRS and Industrial Automation?

I have been working more on local RF mesh than cellular, so my blog has been quiet.

However, users should be aware that GPRS in on the way out.  AT&T (and others) will swear up and down that they still fully support it, however we have customers reporting that their mobile (truck) based systems using GPRS are finding more and more dead-spots.

I agree that GPRS is not dead.  However, before investing in a GPRS-only cellular devices, consider the facts and your needs:

1. AT&T stopped certifying new non-3G devices years ago.
2. AT&T has been rumored to be reallocating air/RF-space away from GPRS to 3G technologies because the 3G makes much better use of this scarce resource - so more happy customers with 3G than GPRS. 
3. AT&T claims to have fully adjusted their pricing so that 3G is comparable to GPRS - they were forced to do this because A) they don't want more/new GPRS users, but B) huge customers were still going GPRS because it was cheaper.  So I have been told this is less true now.
4. AT&T claims to not be killing GPRS ... yet.

So if you roll the above facts together, I speculate that what is happening is that more GPRS devices are being squeezed into sharing less bandwidth, so whereas before a dozen GPRS devices might share 5 or 6 units of 'resource', now they might need to share 1.

If true, this means that the devices have lost bandwidth in recent years, so are now running on a fraction of what they had a few years ago.  The net result is that devices which upload say 200 bytes a day likely won't notice a difference, however anyone trying to open a fancy web page might discover the task nearly impossible.

GPRS runs at from 9600 to 80kbaud, but devices are allotted 'slots', so think time-division-multiplexing & other forms of fair sharing.  So if a tower offers 28.8kbps service to one GPRS device, and 3 more GPRS devices come online, then in theory they all may see closer to 7kbps throughput.  So the device sending a single small UDP or SMS packet won't notice, while the device which requires successfully sending 350 large packets end-to-end will notice a huge drop in performance, and if the client/server have 'task timers', they may discover that they can no longer complete the task before the connect drops or task-timers expire.

So bottom line - AT&T has not yet killed GPRS, but they clearly have it lined up against the firing-wall!  People who have tiny data needs can still safely use GPRS if their ROI window is fairly short-term (a few years).  However, people wishing to EVEN OCCASIONALLY move a 1/2 Meg file for firmware update or diagnostics may find GPRS less and less usable.  Those who plan to routinely move even dozens of K-bytes of data might begin to see reliability problems, so should move to 3G devices.

Increasing cellular robustness for Modbus

A customer of ours is creating a custom Python program in one of Digi's cellular gateways. The gateway uses Modbus/RTU to poll a flow-computer, then based upon a time cycle or abnormal events, the gateway uses HTTPS and XML to push data up to a central server.

They have noticed that at times the gateway has trouble connecting to the server for hours at a time, so wondered what the best solutions are to increase success. Their code buffers the data, so no data is lost.

The customer had asked about using a Windows PC to PING the remotes, but I don't think that will help.

Mobile-originated (meaning the gateway calling out to cellular) will be more robust than mobile-terminated (meaning Windows calling remote). Therefore the best solution will be for the cellular gateway to work a bit harder to connect out.

First, a few technical details:

• Most IP-based cellular devices maintain a full-time IP connection. However, the cell tower 'parks' the device when it is not talking, which means the device loses any radio channels and can only talk after requesting and WAITING for reallocation of bandwidth. And no, there is no way to compel your cellular provider to preallocate full-time a scarce radio resource to a device that is not paying $4000 per year for full-time data movement.
• This means there is always a delay when a device starts to send data after being idle. I have witnessed this delay being up to 18 seconds long, but it could be longer. Not all TCP/IP clients expect to see such long delays in the connect/socket-open phase.

So assuming the customer's problem is unexpected long delays causing the gateway to give up too soon, the best suggestion is to prime the pump early. Lets say your gateway sends data once per hour, so when your Python task wakes up to send data, then do NOT try to open a TCP socket right away. Instead, first send a garbage UDP/IP packet to your host, then sleep another 15 seconds. It does not matter if the UDP packet fails. You do not need to enable UDP on your host, nor watch for a good or error response. The only goal is to force the cell tower to start allocating radio bandwidth 15 seconds before your task actually tries to create your TCP/IP socket.

This pre-allocation may solve your problem when the problem is caused by slower allocation of radio bandwidth. Since most cellular providers now round up data traffic to the 1K charger per hour, sending an extra 30 bytes per transaction should not increase costs.

However, the problem may NOT be radio bandwidth, but authorization through a chain of roaming service providers. This is especially true for low-cost GPRS/GSM providers. Your data is passing through 2, 3 or more companies, and although they may all have agreed to allow the IP session (the PPP session) to open, one of them might drop the ball and block your data access in a way which the PPP session doesn't detect. The only way to recover is to completely flush and rebuild the PPP session - which in plan English means renegotiate with the base carrier to be assigned your IP address. It simulates a reboot of the product.

Fortunately, Digi includes a low-level feature called SureLink, which can use a PING packet to confirm end-to-end connectivity. You can (as example) configure the Digi to PING an internet IP once per hour (or 'X' hours), then physically reboot the cellular module if access fails. This nicely forces a complete renegotiation of the PPP session, which costs you nothing.

Just be aware that use of DNS can be costly and error-prone. A single DNS lookup can cost over 1K of data plan. For SureLink, you literally want to use a static IP as the target to minimize the overhead costs and false-negatives where things other than PPP health block success.

Sorry about the move

Google/Blogger decided to change the way they interact with externally hosted blogs - probably a reaction to some search-engine abuse since blogger blog 'index' very well under Google.

(This post is a test, plus to move down the redirect message)