Lynn's Industrial Protocols over IP

Increasing cellular robustness for Modbus

2010-06-02T10:03:00.000-07:00

A customer of ours is creating a custom Python program in one of Digi's cellular gateways. The gateway uses Modbus/RTU to poll a flow-computer, then based upon a time cycle or abnormal events, the gateway uses HTTPS and XML to push data up to a central server.

They have noticed that at times the gateway has trouble connecting to the server for hours at a time, so wondered what the best solutions are to increase success. Their code buffers the data, so no data is lost.

The customer had asked about using a Windows PC to PING the remotes, but I don't think that will help.

Mobile-originated (meaning the gateway calling out to cellular) will be more robust than mobile-terminated (meaning Windows calling remote). Therefore the best solution will be for the cellular gateway to work a bit harder to connect out.

First, a few technical details:

Most IP-based cellular devices maintain a full-time IP connection. However, the cell tower 'parks' the device when it is not talking, which means the device loses any radio channels and can only talk after requesting and WAITING for reallocation of bandwidth. And no, there is no way to compel your cellular provider to preallocate full-time a scarce radio resource to a device that is not paying $4000 per year for full-time data movement.
This means there is always a delay when a device starts to send data after being idle. I have witnessed this delay being up to 18 seconds long, but it could be longer. Not all TCP/IP clients expect to see such long delays in the connect/socket-open phase.

So assuming the customer's problem is unexpected long delays causing the gateway to give up too soon, the best suggestion is to prime the pump early. Lets say your gateway sends data once per hour, so when your Python task wakes up to send data, then do NOT try to open a TCP socket right away. Instead, first send a garbage UDP/IP packet to your host, then sleep another 15 seconds. It does not matter if the UDP packet fails. You do not need to enable UDP on your host, nor watch for a good or error response. The only goal is to force the cell tower to start allocating radio bandwidth 15 seconds before your task actually tries to create your TCP/IP socket.

This pre-allocation may solve your problem when the problem is caused by slower allocation of radio bandwidth. Since most cellular providers now round up data traffic to the 1K charger per hour, sending an extra 30 bytes per transaction should not increase costs.

However, the problem may NOT be radio bandwidth, but authorization through a chain of roaming service providers. This is especially true for low-cost GPRS/GSM providers. Your data is passing through 2, 3 or more companies, and although they may all have agreed to allow the IP session (the PPP session) to open, one of them might drop the ball and block your data access in a way which the PPP session doesn't detect. The only way to recover is to completely flush and rebuild the PPP session - which in plan English means renegotiate with the base carrier to be assigned your IP address. It simulates a reboot of the product.

Fortunately, Digi includes a low-level feature called SureLink, which can use a PING packet to confirm end-to-end connectivity. You can (as example) configure the Digi to PING an internet IP once per hour (or 'X' hours), then physically reboot the cellular module if access fails. This nicely forces a complete renegotiation of the PPP session, which costs you nothing.

Just be aware that use of DNS can be costly and error-prone. A single DNS lookup can cost over 1K of data plan. For SureLink, you literally want to use a static IP as the target to minimize the overhead costs and false-negatives where things other than PPP health block success.

Sorry about the move

2010-04-06T08:22:00.000-07:00

Google/Blogger decided to change the way they interact with externally hosted blogs - probably a reaction to some search-engine abuse since blogger blog 'index' very well under Google.

(This post is a test, plus to move down the redirect message)

This blog has moved

2010-04-06T07:16:00.001-07:00

This blog is now located at http://iatip.blogspot.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://iatip.blogspot.com/feeds/posts/default.

How Long to Obtain Cellular IP?

2010-04-06T06:39:00.000-07:00

Well, I finally have a SIM, so I can actually do cellular tests again. It is a GSM 'Capped Plan' with 20/MB of data per month for $30. The 'capped' means that even if I go over the 20MB by dozens of GB, the maximum charge will be $130. It is a truly 'unlimited plan' for that $130. What's the catch? If I hit the 'Cap' two months in a row, then the carrier either cancels the account or forces me to start paying full-rate for a normal 1GB or 5GB/mo plan.

The 20MB is fine for me since I am testing methods to keep data small.

My current tests have a sleep devices waking, then obtaining a cell-link. For example, I am working on a Digi gateway called an X3 (not on Digi's web site yet). The X3 can power itself down completely, using a real-time clock chip to wake it at a future time.

I ran a test for 5 days, waking the X3 device every 10 minutes and clocked how long it too for the GSM carrier to assign an IP to the X3:

The fastest time was 23.7 seconds (half a minute)
The slowest time was 97.0 seconds (1 and a half minutes)
The average time was 28.6 seconds (so most near 30 seconds, with a few long outliers)
Note that there will be several seconds of CPU start-up time in there - I can only know the RTC wake-up time (which I loaded into the RTC last cycle) compared to the RTC time when the IP address is assigned.

Why is this important? Because the cellular modem consumes a LOT of power for the duration of this variable time. It is one of the variable which makes estimating battery life difficult.

My next test will include actually doing work - probably sending a Modbus/TCP transaction to a DSL-based server.

Cellular DNS Lookup Cost

2009-10-13T14:09:00.000-07:00

I have been working with a customer using two cellular devices with a DDNS provider, so the 'Client/Master' device used a DNS name to connect to the remote 'Server/Slave' device.

Cost #1 - added delay
The first cost they were seeing was added time lag to open a connection.

To open a TCP socket between two idle (but connected) cellular devices requires each device to be "unparked" by their cell tower. So the client device would need to request a resource and be assigned it by the local cell tower - this can require 2 to 5 seconds.
Next, a DNS query would be sent and a response would be required. Some cellular carrier DNS servers don't seem very peppy, so this could add a few more seconds.
Then the client device would receive notice from the cell-tower control channel to request a resource and be "unparked"
Finally, after "X" seconds of delay and lag the client device would receive confirmation that the socket is open (or it might have given up to fast already & aborted).

So using a DNS name instead of an IP Address might add a few seconds delay - which might be just enough to cause problems. The customer I am working with has a client tool with a hard-coded MAX timeout of 25 seconds, and it seems to often hit this during original socket open. So they have no trouble communicating once the TCP socket is open, but it can be difficult to get the first data packets moving.

Cost #2 - added traffic
Of course the DNS query also adds raw traffic. DNS uses UDP, and a query might be 44 bytes and a response 60 bytes. So 104 bytes extra to open a TCP socket (which can be hundreds of bytes by itself).

Is this traffic important? Perhaps. Perhaps Not. Most cell plans today round up traffic hourly, so an extra 104 bytes for DNS and 150 bytes for a Modbus poll/response is still "1K of traffic".

For TCP/IP, the DNS lookup would only occur for a socket open. So regardless of the DNS name Time-To-Live, you might only see this added 104 bytes once per day or month - or anytime you open/reopen a socket.

However for UDP/IP you MIGHT see this DNS lookup for every packet. For example, a DDNS record from dyndns.org might expire in 60 seconds, so sending one 100 byte UDP packet every 5 minutes, could become 204 bytes per 5 minutes, so 2448 bytes per hour rounded up to 3K per hour or 72K per day and NOT the 49K expected.

So while UDP/IP can save a lot of traffic cost over TCP/IP, if short-lived DNS names are required then some of UDP's savings will be lost.

One solution is a custom application to do indirect DNS lookup. In other words, to directly lookup the DNS name within the application, then to ignore the DNS "Time-to-Live" and use only the cached IP address UNTIL COMMUNICATIONS FAILS - then do a new lookup. This is not how normal operating system DNS caches work - they literally honor the DNS server's time to live and would cause a new look up based on the 60 second time-to-live. Yet this is general safe for cellular end-devices since normally the dynamic IP only changes when the cell link is lost, so the remote IP is unlikely to change more than a few times per day.

Does Cellular 3G help your Rockwell PLC Access?

2009-06-10T06:33:00.000-07:00

In short - No.

Cellular data systems are continuously being sub-optimized for the unidirectional fire-hose paradigm of web page viewing, media streaming, image viewing, email download and so on. In other words, you only see the 3G "broadband" performance when you have tens of thousands of bytes to shove in one direction.

Actual field tests of RSLogix 5000 downloading to ControlLogix processors show that using 3G is only a few percent faster than standard technology - plus still probably slower than analog dialup. The problem is caused by RSLogix and Ethernet/IP moving thousands of small transactions in a half-duplex manner. So RSLogix downloads the PLC object by object, confirming success after each object is written. With cellular latency, each of these transactions can take up to a second to complete, so multiple 1 second by 3000 to 8000 transactions and you have your PLC download time.

3G apologists will claim "Oh, but latency in 3G is much lower than older, coarser technologies."

That may be, but when the rubber meets the road (aka RSLogix5000 downloads), 3G doesn't perform any better. 3G might have other value, but for most people moving data "faster" just means your data usage and the final bill racks up faster as well.

Cellular to Redundant Rockwell ControlLogix

2009-06-06T06:32:00.000-07:00

Can RSLinx access a redundant pair via cellular?

Background
A Pair of Rockwell Automation ControlLogix racks with SRM Module and dual ENBT's will share a pair of IP addresses. One IP address is "the primary", and the other IP is "the backup" (if/when it is online). When the ENBTs switch role, they will issue the requisite Gratuitous ARPs to cause other local Ethernet devices (like a Digi cellular gateway) to update their ARP cache, thus comprehending that the IP-to-MAC address mapping has changed. Thus a NAT/Router forwarding to the primary IP should handle the fail-over with only modest bumps.

Config Details
A user desiring RSLinx (and RSLogix or RSView etc) to access a remote Rockwell ControlLogix (any RA/AB PLC) will be doing what the industry calls "Mobile-Terminated" access. The user needs to arrange a cell plan which offers either a fixed IP address to target, or at least a Dynamic DNS name to target (like tk101.iatips.com or panel2.digi.com). This is NOT what you obtain with an iPhone or personal air-card data plan. Those will have private IPs which only permit outgoing connections - called "Mobile-Originated". That was a buzzword lesson - expect to be asked about those two terms when you ask about cellular data plans!

So once you can arrange your targetable IP or DNS name, you need a cellular router such as one of the Digi Connect WAN family products. My favorite model today is the ConnectPort X4, but that has large memory for Python programming, wireless mesh and other goodies you won't need to link up your Rockwell PLC (Hey, I said it was MY favorite - doesn't mean it has to be yours!)

Note that contrary to folklore or urban legend, all cellular devices need certification to work on a system - even GSM devices. Many small suppliers get around this by including fine-print that say the device buyer is responsible to arrange such legalities, and since you (the buyer) don't read such fine print the salesperson will just say "Heck, it's GSM - so it is allowed everywhere world wide!" Deal with this issue as you see fit, but Digi has more formal certs in more countries worldwide than any of the other industrial players.

But back on subject, when the gateway comes up, it is assigned your known IP or DNS name. This is exactly how your home or business DSL/T-line line works. Yet when RSLinx tries to talk to the gateway on Ethernet/IP's well-known TCP port of 44818, the gateway will reject the connection as a weird attempt at hacking. You need to instruct the gateway:

to not reject the Ethernet/IP traffic on TCP port 44818

to instead forward it to a local IP on the Ethernet - which would be the IP of your ControlLogix ENBT (or the primary IP of the redundant pair)

The details of how this man-in-the-middle fake-out works is fascinating (to me), but quite a pile of text. If you are interested, this older blog entry goes through the NAT/Router details blow by blow. But bottom-line, your ENBT receives the RSLinx packet and needs to have its own Gateway IP set to the Digi gateway's local Ethernet IP address. Free hint: 9 out of 10 guys who call saying "Why can't RSLinx see my PLC through your Digi gateway?" have failed to set the correct Gateway IP in the PLC/ENBT.

So assuming your gateway and PLC are setup correctly, then targeting the RSLinx "Ethernet Devices" driver (with timeouts slowed down to 30 seconds) will cause your PLC's little icon to show up. With RSLinx running, you will create up to 200MB of billable cell traffic per month doing absolutely nothing - so don't leave it active. Note that the "Ethernet/IP Driver" won't work as it requires UDP broadcast, which can't be routed over the Internet.

At this point you'll say "Cool, now can I see my backup ControlLogix or a second PLC?", and the simple answer is "No." One of the realities of RSLinx and AB PLC is that the Ethernet/IP protocol is hard fixed to only the TCP/IP port 44818, and the NAT/Router can only forward TCP port 44818 to a single local PLC. The easy fix would be for Rockwell to change RSLinx to enable adding both an IP/DNS name and TCP port number - then the NAT/Router could forward TCP port 44818 as 44818 to the primary ENBT, TCP 44819 fixed-up to 44818 to the secondary ENBT, TCP 17256 (a random number :-]) fixed up to 44818 to an RSView panel and so on. Because the NAT/Router can restore all traffic to 44818 on the local Ethernet, RSLinx is the only tool needing to change.

Will Rockwell ever do this? It would take a programmer half a day to do - then a few weeks to test - then a few months to document and forestall support headaches. So who knows. They might. They might not.

But bottomline is a simple cellular NAT/Router can be used to talk to a pair of ControlLogix running in a redundant configuration - you will just be limited to seeing only the primary ENBT and the primary IP address.

Cellular Antenna and your PLC

2009-04-07T05:56:00.000-07:00

One of my customers is having fun with one of their own customers. My customer uses Digi gateways running a Python application to collect hourly tank levels, which are fetched by cellular once per day. The tanks hold 250-gallons of a chemical additive which is injected at a variable rate into crude oil pipelines. Small battery-powered ultrasonic level sensors (www.massa.com) push the last 8 hourly samples every few hours using wireless IEEE 802.15.4 (aka Zigbee). The end customer's goal is to forecast when the tanks need to be refilled ... and I don't mean just receive an alarm when it is running low, they literally want to plan truck routes in advance.

Fun stuff.

The problem is that one of the pilot sites has bad cell coverage, meaning many days the central system cannot upload the log. Of course the data eventually it all uploads since it is held for over a month. Now, never mind that this system sits down in a dry wash (a small valley), the end-user says "Hah, that's because carrier A*** stinks; we all love carrier S****." My customer does not want to mix carriers - especially since negotiated pricing between the two in question is so different.

So remember that "coverage" for fixed RTU must be viewed differently from "coverage" for mobile uses.

For a mobile device (or user) "coverage" is defined by the probability that a valid carrier connection will be available as the device moves around. Thus the number of towers (etc) is important, and if there is no signal in one spot, then hopefully will be one a mile or two away.

Unfortunately, our little RTU bolted to a power pole in that gully won't be moving a mile or two ever. So in the end, "coverage" for a fixed device is defined only by the tower(s) which can be seen. So the carrier with the best cell-phone coverage might not be the best carrier to support a particular fixed RTU.

The first step will be to move the RTU up out of the gully, which is easy since the data signals are all wireless and power can be tapped from any of the privately owned power poles. Probably carrier A*** (stinky or not) will work fine once the RTU moves. As plan B and make the end-customer feel listened to, a S****-based RTU will be placed on the same pole as the relocated A*** one.

Could they run out a long external antenna? Sure, they could. But why bother? Put the RTU where it has a nice signal. Even in 2 or 3 store buildings we suggest customers put the cellular router up in the roof-area and run the Ethernet UTP it's 100 meters instead of trying to deal with the signal loss in long antenna cables. We even have customers using 900Mhz Ethernet bridges to link a cellular router placed where it must be placed back to the Ethernet devices which want access.

Worst case, a directional yagi antenna could be used, however you need to understand that cell towers are routinely turned off without warning. The carriers are truly geared towards mobile users who expect bad signals in some places. Towers (or the active elements) also move. Most of the cell towers you see along the highway work like strip-malls; a company owning the tower and supplying power leases tower space to a mix of carriers. This allows everyone to be flexible.

So prematurely locking down a cellular device to a single tower with a directional antenna can cause future problems since it will not see other weaker towers should the targeted tower be turned off or even moved.

Ethernet/IP PCCC Service Codes

2008-11-25T06:29:00.000-08:00

Summary: the Rockwell PCCC Service Codes (or how to get a 1761-NET-ENI or Digi One IAP to send specific DF1 DST/SRC bytes)

A customer had a product which acted as an Ethernet/IP originator (client) and needed to talk through a Digi One IAP to feed DF1 into a DH+ gateway. The catch is the DH+ gateway used the DF1 destination byte as the DH+ node address - and so far they could only make the Digi One IAP spit out a destination = 1. This made for pretty boring and unprofitable DH+ design!

(Sorry: I cannot explain "How to write an Ethernet/IP stack" in a single blog entry, so this entry assumes you already have a product speaking Ethernet/IP and just want to tweak how you talk to remote DF1 devices.)

The PCCC object (code: hex 0x67) is not part of the ODVA spec - it is a vendor specific object used by Rockwell/Allen-Bradley to talk to SLC5, PLC5E and MicroLogix PLC. I won't fully explain it here ... not much to explain; do a wireshark trace of a ControlLogix talking to a SLC5/05 or MicroLogix 1100 and you'll see all you need to see about the simple object.

The default service code of 0x4B has this form:

Exec PCCC Service	4B
IOI to PCCC Object	02 20 67 24 01
Originator Info	07 03 85 50 4d 41 41
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

The only mystery here should be the "Originator Info". The 07 means a total of 7 bytes (including the 07). 0385 is a unique sequence number which you should change between messages. The 504d4141 is a "CIP Serial Number" which must be unique to your vendor id - most vendors just use the last 4 octets of an Ethernet MAC address.

However, there is NO remote node or slave address info, so the Digi One IAP or 1761-NET-ENI will create a DF1 message for destination=1 and source=0. So back to the customer's question ... how to force a DF1 message with say destination=5 and source=2?

The easiest solution is to switch to service code 0x4C, which has this form:

DH+Like Service	4C
IOI to PCCC Object	02 20 67 24 01
DH+ Like Header	00 00 02 00 00 00 05 00
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

So the "Originator Info" has been swapped with an 8 byte structure of the form AA AA BB XX CC CC DD XX. The "XX" are control bytes of some sort - just leave 0x00. "AA AA" is the Destination Link; "BB" is the Destination node; "CC CC" is the Source Link; "DD" is the Source node.

Voila - both the Digi One IAP and the Rockwell 1761-NET-ENI will now create a DF1 message with destination=5 and source=2. Not too painful, was it.

To my knowledge all Rockwell Ethernet/IP PLC accept either code, and if you'd like to see the service 0x4C in action just set your RSLogix5000 MSG block to use "CIP with Source ID" and the ControlLogix switches from using service 0x4B to using 0x4C.

For completeness, there is a final service code of 0x4D with this form:

Local PCCC Service	4D
IOI to PCCC Object	02 20 67 24 01
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

It is documented in DeviceNet gateway manuals to reduce total byte count - but it also lacks any destination or source info so is of limited use in Ethernet-based systems.

Creeping Firewalls and Change

2008-08-07T07:05:00.000-07:00

Summary: Network infrastructure changes can create unexpected havoc (surprised?)

We had a customer come in with a complaint recently about some old products - a 4-port terminal server (a TS4) running Modbus bridging firmware. Things had gone on smoothly for years, but suddenly they started having problems which power-cycling the units solved ... obviously it's a Digi problem, right?

Well, I'm not going to run through all of the issues, but I'll highlight one which will affect more and more people over time. Remember the old days - first you had dumb dialup modems, then you had to pay a lot for various error-detection standards, and now all analog modems have a big chip which does a hundred different standard things and you CANNOT buy any low-cost analog dial-up modem which is any dumber than state-of-the-art.

Such creep occurs in Ethernet switches also - once hubs were cheap and switches cost a king's ransom. Now even the "cheap" $19 home switches purchased in big-box stores are auto-sense, auto-cross, auto-everything. The same things happens - once the features are rolled into software or firmware, future revs of the product inherit such features for free ... said another way, it is often cheaper to sell lots of one powerful model than handle the marketing and support costs of a dozen models of various power and feature-set.

Well, this customer had been upgrading their wide-area-network infrastructure - a tangle of routed IP-based leased lines, old analog lines, radio/microwave links and so one, and as part of the improvement the various router/PPP end-points had added stateful awareness of packets ... part of an increase in US government security demands for utilities. After all, when you have hundreds of raw Ethernet ports scattered across rural American protected with little more than a padlock and chain-link fence, one has to consider the possibility of someone trying to 'jack' back into the 100% private network from a remote location.

The issue with such an addition is that routers with firewalls commonly default to a 5-minute TCP life rule. Every stateful firewall creates a table entry for every TCP or UDP activity going "out" from a safer network and into a wilder, less safe network - your home DSL/Cable router does the same thing. NAT (etc) is optional, but bottom line is the firewall needs to understand and agree to the context of every packet trying to pass through.

For a TCP socket this usually means at least 1 packet (data or ACK or Keepalive) must move every 5 minutes to refresh the context and inform the firewall that the TCP socket is still authorized. If no packets are seen, the fire wall discards the context and the socket is now ignored and 100% blocked without comment. Of course both TCP peers might think the socket is alive and well, but they will never be able to use it again.

So back to our TS4 off in some remote field - a host computer has 4 TCP sockets open to it through various routed subnets and for whatever reason the host doesn't send any new packets for 6 minutes. The TS4 has no reason to talk since it is connected to Modbus slaves. So 6 minutes later the host issues 4 Modbus requests on 4 sockets, which hit the stateful firewall. The firewall looks up the context (called forwarding or trigger rules in some systems) and discovers there is no existing context for these TCP packets. Well, once upon a time the firewall would just assume since these were safe since they came from the safe side, and thus auto-create a new context, yet modern stateful firewalls might NOT do this since it was a common hacker exploit to fool firmwalls to move malformed packets. Thus modern security demands that the first packet of any new TCP socket be a [SYN] packet. All four packets are silently discarded as unknown (note: this is a configurable behavior in most firewalls - to auto-recreate a context for an 'old' TCP socket continuing or to only allow this for new TCP sockets.)

The TS4 never sees the four new request; the host sees neither an [ACK] nor a [RST] of the TCP socket, so it retries after 3 seconds, then 6 and so on. Eventually it comprehends the TCP socket has died and opens four new TCP sockets. These are allowed through the firewall since they contain the correct TCP state as being new sockets. The TS4 sees the four new socket requests and now has eight (8) sockets connected.

Now you might say "What a minute - the four old sockets were closed!". Well, yes - the HOST knows they are dead and closed them, but the TS4 does not know this. Six minutes later, the same is repeated and the TS4 now has 12, then 16, then 20 sockets open. Eventually the TS4 runs out of resources, and the customer's open recovery option is a hard reboot of the TS4. This is of course worse if the host only occasionally takes longer than 5 minutes to poll, since the increase in TS4 sockets could take an unpredictable amount of time to build up.

How to solve this problem?

The first line of defense in the modern IP age is to always, ALWAYS enable TCP keepalives with an aggressive 4 minute 30 second idle time in any product you install. Even if you don't have stateful packet inspection within your IP network today, that doesn't mean the next time someone replaces a router or serial PPP end-point you won't gain one. Unfortunately the Digi products usually default to a 2 hour TCP Keepalive - which is also disabled (for historical reasons). Yet having a 4.5 minute keepalive would have saved the TS4 because the first time the host delayed longer than 4.5 minutes to poll, the TS4 would have sent a TCP keepalive packet back through the stateful firewall, which the host returns and the firewall's context for this socket is refreshed. Thus when the host does finally send the next Modbus poll after 6 minutes (1.5 minutes after the TCP keepalive exchange) the firewall is satisfied with the packets and forwards them out to the TS4. Everyone is happy. Plus even if the TCP socket has been aborted by the firewall, the TS4's TCP keepalive will be silently discarded and the TS4 will retry and eventually comprehend that the TCP socket is no longer valid. This is the purpose of TCP Keepalive - to allow a TCP peer with no data to move to test (and refresh) the health of an idle connection.
The second possibility is unique to the Digi IA/Modbus application. You can enable a setting referred to as "idletimeout" on incoming client or outgoing server connections. Unlike the TCP Keepalive which create traffic and keeps an idle socket open, the idletimeout literally just aborts an idle socket without giving it any chance to prove it is healthy. So setting a 5 minute idle timeout in the TS4 would cause it to just assume any incoming Modbus client (master) connection which has NOT sent a new request is bad. This setting would also have saved the TS4, forcing the four old sockets to be aborted before the TS4 had a chance to build up a herd of dead sockets.

So understanding TCP Keepalive is a critical skill for all modern industrial Ethernet users, after all more and more facilities are breaking their floors and functional area into distinct subnets with stateful firewall protection to keep the Accountants from changing the color of the widgets being produced - and to keep the production engineers from printing on the accounting departments laser printer which uses red ink :-).

More complete discussion of TCP Keepalive is here (one of dozens of web site you can find in a web search).

Tunneling Serial Data over Cellular

2008-08-01T09:51:00.000-07:00

Tunneling Serial Data over Cellular

Question: you have a Windows-based application which expects to talk over normal serial ports - how can you connect to a remote serial device over a cellular-IP link?

Products: works with Digi Connect WAN, WANIA or WANVPN, Digi ConnectPort VPN, X4 or X8

Answer: Our Digi RealPort driver for Windows 2000/XP/Vista dated Nov 2008 now supports a nice low-overhead "UDP: Serial Data Only" mode. This will cost a fraction of what normal TCP-mode Realport or competitors TCP redirectors will cost.

Here is a web page explaining how to install RealPort for UDP mode (DDNS names can be used!)
http://digitips.wikispaces.com/Digi+Realport+with+UDP

Here is a web page explaining how to set the WANIA to UDP Sockets.
http://digitips.wikispaces.com/Digi+UDP+Sockets+WAN

I set this up yesterday and have been polling a Modbus/ASCII serial slave on my CPX4 using RealPort and UDP mode. Responses take from 2 to 3 seconds to return … in part because I am polling so slowly that the modem PARKS in between every poll and I wait 30 seconds for a response timeout. I see perhaps 0.25% error / no response timeout but then my SIM doesn’t have the best signal where my product sits.

However, I still believe people need to be realistic – even in the situations where the slave response times out I’d suggest NOT retrying immediately since the retries have a higher than normal probability of failing as well. Missing a few polls a day is better than needlessly paying more for data plans.

Evolution of Data Plan Billing

2008-07-23T12:28:00.000-07:00

Summary: the big three have moved away from unlimited data, towards limited data.

It is interesting - I once (as in last year) had a talk with a potential partner who'd been at some European conference and was convinced the world was on the verge of low-cost (sub-$20/month) unlimited cellular data plans. We were discussing the creation of report-by-exception tools to reduce SCADA costs, and this partner's strong faith in this belief caused them to eventually bail out of the talks, saying "In a year or two, no SCADA company will care about how much cellular data they use."

Yet as of the summer of 2008 the world of cellular data is moving in the opposite direction. Last year the big three (AT&T/Sprint/Verizon) offered "Unlimited Data" for personal users with the Service Terms listing a VERY narrow list of permitted activities - mainly email and web browsing, with many common things like file download/upload, media-streaming prohibited. So when ever one of the big three would cut off a user for moving too much data on an "unlimited plan", the service provider would fall back on the "You are doing prohibitted things, thus impacting our network, thus take your business elsewhere". What a way to cause bad feelings, eh? Note that this change is CONSUMER plans - machine-to-machine have always been limited, priced by the MB/month without rollover, plus with charges for data overages.

Now all three have dropped the price from the $80/month range down to $60/month range ... but added a hard limit of 5GB per month. Isn't free & vigorous market competition wonderful?

Sounds reasonable - 5,000 megabytes of data is a lot, yet this doesn't mean 5GB of data transfer. It means 5GB of metered activity, with many activities I've studied including up to 95% overhead. Thus someone only moving 20-30MB of real data in small packets per month might hit pretty close to their 5GB limit! My experience with normal wide-area-network traffic hints that a real PC user doing simple email and web-browsing once a day would probably move 1-2GB of data before hitting the 5GB total activity limit.

To paraphrase the wireless data service terms for all three:

Data transport is always measured in full kilobytes
Actual transport is always rounded up to next full-kilobyte at "end of session"
Network overhead and resend requests caused by network errors can increase measured kilobytes.
2 of 3 mention always rounding up to nearest kilobyte every hour period.
All warn that you will NOT receive an itemized detail of how your charges are calculated; you will NOT see which services were used or during which time periods the charges were inccurred under.

So if I send a single 50 byte UDP/IP packet, is that a full session and billed as 1024 bytes? Could be under this language since UDP is 'sessionless'.

Hmm, the term session is pretty ambiguous. Perhaps it means per "time you enable your PC-based cellular data card." That seems likely - plus if you left your device on twenty-four hours a day then the once per hour round-up would catch you.

I'm afraid I haven't offered any new answer here, other than to suggest you understand that low-cost unlimited data plans ARE NOT just around the corner ... at best we left them behind last year and I don't foresee them ever returning. I suppose all three now understand that huge new profits are to be made with these 5GB limits, which will cause many "super-salesman" using their cellular data plan daily to spend an extra $50 to $500 in monthly overage charges.

Lower Cost Cellular to Rockwell AB PLC

2008-07-11T09:56:00.000-07:00

I have several customers now working through how to manage cost-effective cellular access to Rockwell PLC such as ControlLogix, CompactLogix, Micrologix 1100 and so on. Unfortunately the most straight forward way to link using Ethernet/IP is fairly costly.

First, a personal recommendation from me – a free tool which I find very useful and think you will too. Today, you can buy 2GB USB flash drives for $15 – if you're old like me, you remember when an entire Windows computer only had 0.020GB of hard drive space! Did you know you can literally install and run many Windows applications from these portable USB drives? This means any Windows computer you plug this USB drive into has your applications, your settings, and your data files. I've used one of these for over a year and it is invaluable - all free open source code too! You can run OpenOffice (which can read/write MSOffice 2003 files and is much faster than the MSOffice 2007 we use at Digi), Firefox web browser, plus a dozen other tools. Take a special look at KeePass, which I use daily from my USB drive to securely hold all of my hundred-plus account names and passwords.

Portableapps.com - What is a Portable App?

Okay, back to work.

Periodic PLC Access from RSLogix

Customers who want to peek into a single PLC at a remote site for an hour or two can use RSLinx to connect to either an IP or DNS name, then see the PLC via cellular. The catch is RSLinx will create from 12MB to 200MB of background traffic per month. So you need to create a new Ethernet Driver (not Ethernet/IP!) JUST for this one-time use, configure in your details, connect and do your work. When you are done you need to turn browsing off, then delete the comm driver. Why not just delete the IP or DNS name? Unfortunately once RSLinx has seen a device, it can be like a bad rash to get rid of it.

Data polling – at Central Office

Customers with an OPC server speaking DF1, CSPv4, or even Ethernet/IP can poll PCCC-type data through a Digi One IAP, which converts the polls into DF1 Radio Modem under UDP. Tests have show using DF1 Radio Modem every few minutes accomplishes the same data movement as Ethernet/IP with only 5% the data cost (or Ethernet/IP uses 2000% more data bytes). One unit of Digi One IAP can poll up to 60 remote IP or DNS names. If your OPC server can encapsulate DF1 Radio Modem directly into UDP/IP, then you won't need the Digi One IAP to act as your host.

Data Polling – at Remote Site

If you have an AB PLC which speaks DF1 Radio Modem directly, then any Digi cellular router can be configured for UDP Sockets, with shuttles UDP data received to the serial port. Make sure you use the latest Digi firmware so it can just return UDP responses to last sender without explicit address configuration.

If your AB PLC doesn't speak DF1 Radio Modem, or you want to use an Ethernet link, then using a Digi cellular router with Python support allows a simple script which accepts DF1 requests and uses a local Ethernet/IP session to query responses from the PLC's PCCC Object. This Python code even runs on a PC under Windows or Linux. As soon as I have a link or web page explaining how to get and use this code, I'll edit this post to add it here.

Quick Data Comms to AB PLC

2008-07-10T14:54:00.000-07:00

One of my readers was asking for a quick way to talk via Ethernet to a Rockwell AB PLC.

You can actually talk to a ControlLogix by only understanding TWO (2) different packets, each with a response, so four packets I guess. The problem is this uses "UCMM" style communications, which the PLC has very limited resources for. Said another way, the CIP Connected Messaging or I/O production both include an inherent allocation of resources, while the UCMM is designed to be used ONLY to setup such pre-allocated resources.

So yes, you can use the information below to create a literal quick-n-dirty solution, and if you talk more than a few times a second you might start to interfer with other communications to the PLC (which is not a good thing!) However you could treat this as a proof of concept, and then work to do the communications more fully per the ODVA specs.

Here is the PDF of how to read/write to the PCCC Object in a Logix Processor here.

Public Internet Risk in Common Tools

2008-05-30T07:06:00.001-07:00

Two months ago a SCADA customer asked me to enable FTP (File-Transfer-Protocol) on a test RTU they'd sent me to put online for them. It was on a DSL link and although I warned them it was a bad idea they said it would be okay because the RTU had username/password protection and the RTU had nothing important on it.

The punchline is that a few days later the customer sent me an email saying they couldn't FTP into the RTU anymore, so couldn't check the log files.

I looked, and the RTU now had 3 TCP sockets open (all the sockets allocated for FTP on this RTU) to some FTP client at an IP address registered in Korea. All 3 were just slowly walking through a dictionary attack of username/passwords (user bill, pass honda14 ... user billk, pass 12tomes ...) No doubt the IP and FTP client belonged to some university student running kiddy-scripts obtained on the Internet. No doubt the kid probably didn't even care if the odd device he or she had never seen before was not a computer (FTP servers always announce what and who they are when you connect). No doubt this attack wasn't costing them anything, as either the university or parents were paying for the Internet connection - or the IP and connection belonged to some patsy whose home computer had been compromised.

So okay, it was not causing any real harm, except it defeated the purpose of enabling FTP since the end user no longer had access to FTP. I suppose you could call it a denial-of-service attack, yet I'm sure that was NOT the intension of the 'attacker'. The student was probably just hoping to be able to post a message on some forum saying 'I hacked a computer at this IP address in the USA, and here is the FTP user name and password I created for you to access.' The fact that the RTU only contained a dozen binary log files would be irrelevant.

Do I have a moral to this story? Hmm, not really - other than industrial users have to understand that what is NOT interesting to them might be interesting to others for very different reasons.

If this user had allowed me to change the FTP port to some random value like TCP port 38207, then it is very likely this particular student would NOT have found it. Since true port-scans are so easy to detect, the student's tool probably had a list of a few dozen TCP ports commonly used by FTP servers, then it would randomly try them over a few weeks at any single target IP.

The same story could be true for web servers on TCP port 80 (or 8000 or 8080). Digi ships our cellular products with the web server enabled on port 80 because that is what customers expect. Sure, they add a username and password, but what will happen to their data plan bill if their 3MB per month plan moves 3-GigaByte because some scripts are trying dictionary attacks on the login of home web page?

I've had to always mention that 3GB part (meaning a $500+ bill for a month) since every time I mention to such users not to leave port 80 setup as a web browser, the industrial customer's answer is invariably '... it'll be okay because the unit has username/password protection and it has nothing worth trying to see on it ... '

Cellular to Wireless Zigbee

2008-05-22T09:34:00.000-07:00

An interesting new market we are moving into is cellular access to wireless mesh (Zigbee as example). In some sense it's a supply-chain dream come true. Imagine you supply a product to customers and EVERYONE (except the customer's IT department) want you to be able to see what the stock level is and auto-schedule deliveries.

The customer benefits because they can treat the product as a 'utility' - turn the tap and there it is.

You benefit because you can minimize emergency truck-rolls - no more angry, panicked customers demanding you send a truck over two-thirds empty because someone forgot to schedule a special delivery because the customer needed to use 60% more product for two days. Of course as a supplier the cost of the truck, fuel, and driver are critical parts of your margin/profit. You desire to only send out full trucks which return gracefully empty!

So we are now working with several of the largest chemical suppliers in the world to enable:

drop in a powered cellular unit at the customer site
drop in powered or battery tank sensors
log levels hourly, for the supplier to upload daily (reduces cellular data charges) The supplier uses this as their 'secret-sauce', their own proprietary value-add to predict when trucks need to roll to maximize efficiency
enable alarm call-out if the levels hit unexpected low-low levels

How this works varies by suppliers. The one I'm working with is using Modbus/TCP to pull up the logs daily. Some other suppliers are having SMTP clients push emails back to the supplier with XML formatted reports. The next supplier I might work will wants the binary logs to be compressed (ZIP'd) and then pushed upstream once a day by FTP, where their accounting system will convert from binary to XML to import and issue bills on product usage per MINUTE.

Of course key to all of this is the wireless drop-in-network concept. The supplier doesn't want to invest thousands of dollars pulling wires through SOMEONE ELSE'S PLANT - especially when the supplier's contract might end in a few months.

Wireless sensors aren't new; cellular data access isn't new; supply-chain systems which auto-detect product levels aren't new. What is new here is the merger of many technologies which reduce infra-structure costs, and thus increase ROI.

PCCC Protected Typed Logical Write with Mask

2008-03-14T07:42:00.000-07:00

Someone asked about the "DF1 Supplement for SLC500" from 1995 which was online at ab.com for a short period, then was pulled - probably because it is a very poor quality optical scan. However, I had it ... then lost it ... then found it stashed away on one of my ftp sites.

So here is the original AB PDF (which I downloaded from ab.com last year) Grab it here while it survives.

The masked write is on page 11 and the first data-word is the mask and all data following has the SAME mask applied. So [0x0001,0x0000] would clear the LSBit and so on. It doesn't offer mask-data pairs - just one mask and N words. Thus this command is mainly for use with 1 word element writes ... unless you have three or four consecutive N-file words you wish having the same mask applied.

Optimizing Modbus for Cellular

2008-03-07T09:27:00.000-08:00

Goal: lower your data costs

How nice it would be if you could take your Ethernet applications and just move them to cellular (or satellite). Well, of course you can ... but you'll pay through the nose for this.

At the moment I'm in the process of creating intelligent cellular gateways accessible by Modbus/UDP (aka Modbus/TCP form in UDP) which support data logging, report-by-exception and other cost-saving goodies.

A few Facts:

You are charged for all IP, TCP, and UDP overhead - the cellular system moves your TCP/IP packet as raw payload encapsulated in mobile-IP or other transports. So to them the 40-52 bytes TCP/IP header as NOT DISTINGUISHED from your data. ( My Blog entry on this )
Thus Modbus/UDP (aka Modbus/TCP in UDP/IP) will save you from 60 to 90% of your data costs. It is a single one-shot request followed by a single one-shot response. In contrast TCP/IP might require up to 400 bytes of socket open & close overhead, plus TCP acknowledge packets.

So if you are concerned about cost, you should first make sure your DATA POLLING can be done with Modbus/UDP - not TCP. No sweat if you need to use Modbus/TCP to reprogram or monitor your RTU or PLC short-term, just make sure your 24/7 repetitive data polling in via UDP. ( Is UDP reliable enough? )

So I've defined a few extensions (read as 'heresy' to many). Full Details are Here at iatips.com:

I allow use of the full Modbus/TCP header - so one can read 500 registers in a single request. This greatly reduces charges for header overhead
I allow returns LESS than data than requested when the context is appropriate. This saves having to pay for data padding, plus not having to poll a status register to see how much logged data is waiting
I allow use of data compression like ZLIB. Bottomline, 'ZIP' compression of small data sucks, but since I can return 500 registers (1K bytes) ZLIB starts showing value.
I allow packing multiple Modbus-ADU below a single header, which (unlike pipelining) signals the gateway to return multiple responses in a single packet.
I am looking at support for simple AES encryption, not as true 'security', but as a good-enough means to support Modbus without making development difficult.

If you are interested in the details, I have more discussion on my wiki site.

Lost My SIM

2008-03-07T09:21:00.000-08:00

Well, I've been quiet for awhile - lost my "free development SIM" as part of Cingular's reorg into AT&T. Thus my collection of PLC with free demo access are off-line. Is interesting to support cellular without a SIM ... not that my boss is ignoring this, but there are 'plans in the works' which involve several companies ... plans which just keep moving out.

However, I am working on Cellular gateways with intelligence now. Goal is to allow a Modbus client/master to come in once per day and upload time-stamped logs; plus if certain events occur use Modbus to call for help.

How Exposed in Public Cellular IP

2007-10-05T10:28:00.000-07:00

One of the concerns about having a PUBLIC IP address for cellular is your exposure to public hackers probing public IPs for services. In theory, you pay for all of these attempts since the mobile IP system encapsulates and transports them all on your behalf.

I'm happy to report that after logging such attempts for many months my cellular devices receive less than 4 probes in any one day and likely under 20 total per month. So many days pass with no probes at all and it appears to stay under 2-3K per month. This compares to perhaps 200 probes per day on my DSL router/firewall.

I am not sure why the difference, although I'd guess it has to do with the high initial latency in contacting a cellular IP. So any "script-kiddie" tool scanning IP address ranges probably is not willing to wait up to 5 seconds for cellular devices on busy towers which need "unparking" to respond.

What kinds of probes are they? Mainly those looking for MS-SQL servers, with a rare access to FTP and the remainder of accesses aimed to seemingly random, unnamed ports - likely associated with trojans or zombie networks.

The Truth about Cellular IP

2007-08-22T06:19:00.000-07:00

I have been very busy analyzing real-world telemetry traffic over cellular IP, and unfortunately I am now 100% convinced that you cannot effectively use most (any?) off-the-shelf "Ethernet" software tool to talk to remote Ethernet devices over cellular IP. Bottom-line is that - unless your host app is custom written to be data cost and time delay sensitive - your data costs will be bloated due to the nature of the tool. Even something as "obvious" as adding compression doesn't solve the problem because telemetry packets tend to be too small for effective compression. For example: an 8-byte Modbus/RTU request becomes 12-bytes after ZIP-style compression. Plus this doesn't reduce the 104-bytes of TCP overhead nor 28-bytes of UDP overhead. None of the cellular providers allow use of RFC-class TCP header compression, since it requires all of the infra-structure to maintain
copies of headers etc.

So I have been working on "reduction" solutions - how to obtain the effect of moving "X" IP packets but only moving "X-minus-a-bunch" of actual IP packets.

Tunneling TCP thru UDP
The most promising and generic form of reduction is to tunnel TCP/IP via UDP/IP over cellular. So the host application talks TCP/IP to a local proxy, which acts as the TCP end-point. All of the TCP SYN, ACK and Keepalive traffic is limited to the local Ethernet. The local proxy then initiates a UDP "session" with a remote proxy over cellular & we instantly see a 60-90% reduction in data costs. The remote proxy initiates a TCP/IP connection to the remote Ethernet device, which again isolated the extra TCP overhead to the remote Ethernet.

The reaction of non-IA network engineers to this idea is predictable and a bit humorous after a while. They immediately say "You cannot do that!!! UDP/IP is unreliable!!! You'll break something!!! You are committing a mortal Sin!!!" But in reality none of the IA protocols leverage the reliability of TCP anyway. For example, Rockwell RSLogix doesn't send a program block to a ControlLogix and blindly assume it was successful after the TCP Acknowledge from the peer is processed. Instead, RSLogix sits (blocks literally) and waits for a successful CIP response on a single CIP Connection. So if the local proxy returns a TCP-ACK to the RSLogix host and the CIP request is lost within the UDP/IP tunnel ... eventually RSLogix times out the CIP connection and the application (and/or user) will restart.

Fortunately, cellular is very reliable - all of my tests sending 10,000 UDP packets rarely even lost 1 packet and I'm not sure if such a rare loss is due to cellular or just my test script hiccupping & dropping a packet. Plus cellular tends to have only very bursty error problems. In other words, you won't lose 1 packet per 10,000; instead you'll lose all packets for 5 minutes or just 5 random packets out of a group of 10 sent. This shotgun-damage tends to confuse TCP/IP state machines to the point that they abort the connection anyway. In truth, in all of my Wireshark/Ethereal trace reviewing I have never seen a single situation where a TCP retry did anything but add data cost; every TCP retransmission just results in a "Duplicate ACK" showing up a few packets below in the trace & a doubling of the cost of that block of data.

So overall, anyone planning to use cellular should first investigate if they can use UDP/IP instead of TCP/IP.

TCP Problem #1 - added cost for pointless ACK
As mentioned above, real-world analysis of telemetry use of TCP shows the TCP ACK isn't useful; but worse, Embedded TCP devices tend to sub-optimize the ACK timing to "speed up" data transmission and recovery. Almost universally moving an IA protocol via TCP/IP results in 4 TCP packets instead of the idealized 3.

Your app sends a TCP request (request data size + 40-52 bytes of overhead)
800-1100 msec later your app receives a TCP ACK without data (another 40-52 bytes of overhead)
10-40 msec later your app receives the protocol response (response data size + 40-52 bytes of overhead)
Within a few msec, your app sends the TCP ACK without data.

So what would have been a 2-packet transaction with only 56 bytes of overhead under UDP/IP, or what should have been a 3-packet transaction with 120-156 bytes of overhead under ideal TCP/IP usually becomes a 4-packet transaction with 160-208 bytes of overhead. Yes, there exists a TCP socket option and the concept of "Delayed ACK Timer" to prevent the first empty TCP ack from being returned over cellular, but few embedded products use this since it adds code complexity, and it slows down overall data communications. At least in the IA world it seems everyone wants their Ethernet Product costing 2-4 times more than their serial product to appear lightning fast. So they ignore the TCP community's decades of hard-earned experience and "hack" their TCP stack to sub-optimize fast local Ethernet performance.

So this is where the instant 60-90% data cost savings of using UDP over TCP comes from. UDP has smaller headers and results in fewer packets being sent. Since the cellular IP system is "encapsulating" your TCP/IP packets in a manner similar to PPP, the entire IP header, TCP or UDP header, and your data is all considered billable payload.

There is also a myth propagated to this day that the TCP ack causes retry to occur more rapidly out in the wide-area-network infrastructure. The rhetoric goes, "If the 3rd and 4th router link is congested and the TCP data packet is lost, then the 3rd router will retransmit ... which is faster ..." Perhaps this was true back in the 1980's, but today the 3rd and 4th router (and all of the other 20 to 30 routers in a cellular end-to-end path) are just tossing IP packets upstream with no awareness of the packet functions. In reality, it is only the TCP state machines within your host machine and within your remote device that have any ability to retransmit anything.

TCP Problem #2 - added cost for premature retry
The TCP RFC includes many dynamic timers that automatically adjust themselves based on real-world performance. This is actually pretty neat. It means if the TCP ACK and response times tend to be longer than normal, then the TCP state-machine slowly increases the delay before retransmission. But I've seen 3 problems with this.

The most effective way to leverage auto-adjust is to include the 12-byte TCP header options that time-stamps all packets. Linux system add this by default and installing one of many PLC engineering tools on your Windows computers causes Windows to also start always using this. The setting generally is global - you either have 40-byte TCP headers or 52-byte TCP headers forever. So for small telemetry packets, this adds a disproportionately large increase in data costs.
Many embedded devices (PLC, RTU and I/O devices) have "hacked" the TCP ACK sub-system to force connection failure to be faster than the standard 3-4 minutes. For example, I worked with one large PLC company which expected TCP sockets failure in less than 1 second, so they forced TCP retransmission in hundreds of msec and without any normal exponential backoff between retries. This is totally unusable over cellular; you will end up with 30% to 90% of your data traffic being premature retries and responses to premature retries. I have literally seen Wireshark/Ethereal traces which are mainly black lines with red text - which is the default color used to show TCP "problems" such as lost-frag, retrans, dup-ack, etc.
The latency in cellular is abnormal by an order of magnitude. Even browsing the internet or doing a telemetry polling test over DSL/cable broadband averages latencies in the 100-150 msec range. This is what a Windows or Linux defines as "slow/bad" - not the 800 to 3500msec of cellular. So even watching a Windows or Linux TCP state machine auto-adjust the retransmission delay over time, you will not see it achieve a 100% effective setting which eliminates wasted TCP retransmissions. The delay seems to top out
at about 1.5 to 1.8 seconds, which is just too close to the actual "normal" latency range. So again, use of UDP/IP frees the use user from data costs associated with TCP legacy assumptions - both the main-stream MIS/IT market variety of assumptions and the misapplied IA vendors "speed-ups".

TCP Problem #3 - uncontrollable SYN/Socket Opens

Given the way all cellular systems "park" inactive cellular data devices, it is exceedingly rare to ever see a host app open a new TCP socket without prematurely retrying/retransmitting the SYN packet. This is because one is virtually guaranteed that it will take about 2.5 seconds for the data device to be given active airwave resources and return the SYN+ACK response. This has NOTHING to do with the "always connected" feature Digi and others claim. The data device (even when parked) is fully connected by IP and fully authenticated by the system - it is "always connected". However, the local cell tower only has finite airwave resources, so any device (cell phone or data device) which is idle from 3 to 45 seconds is "parked" without having any preallocated airwave resources. Literally when the TCP "SYN" shows up, the cell tower has to use the control channel to inform the data device to request airwave resources, and after these are requested and allocated the data device can receive and response to the TCP socket open request.

But that's not the real problem related to TCP Socket Opens ... the real problem is yet another case of IA vendors sub-optimizing TCP behavior for fast local Ethernet performance. For example, I once had a customer who normally paid about $40 per month receive a $2000 bill one month. It turns out they had powered down the remote site for 3+ days and the off-the-shelf 3rd-party host application they used would try to reopen the TCP socket every 5 seconds!!! So Windows would send the initial TCP SYN to start the open, since the remote was off-line Windows would retransmit this TCP SYN a few seconds later. After a total of 5 seconds, the application would ABORT this TCP socket attempt and start a new one. So this host app was pushing 24 billable TCP packets per minute out to a remote site that was powered down. This was nothing the host app vendor documented, nor was it anything a user could configure or over-ride. The user could configure the host app to ONLY poll once per 5 minutes; but the user had no control over this run-away TCP SYN/Open behavior.

Tunneling TCP through UDP effectively decouples the TCP SYN/Open from cellular data charges. The first TCP Syn/Open request to the local proxy would succeed even if the remote IP site is offline. No retries would be required. Even if the host app attempts to retry the data poll every 5 seconds, this is something the UDP proxy can be configured to "resist". If the user truly wants data packets to only move every few minutes, that is something the UDP proxy can easily enforce.

TCP Problem #4 - sub-optimized TCP keepalive

The final problem I'll discuss (but not by any means the "last" problem with TCP) is that many embedded IA devices have relatively fast TCP Keepalives hard-coded to speed up lost-socket detection. While this is an admirable goal, a Rockwell PLC sending out a TCP Keepalive at a fixed 45 second interval can create up to 6MB of monthly traffic by doing this. Siemens S7 PLC seem to issue TCP keepalive every 60 seconds - a bit better, but not by much. Maybe such a heart-beat is useful to know the remote is accessible, but given the reliability of cell phones (when the last time you had a dropped call or no signal ...) you'll obtain a lot of false-alarms if you treat every missed packets as something requiring maintenance's attention.

Again, tunneling TCP through UDP effectively eliminates the automatic, possibly uncontrollable use of TCP Keepalive. If your process can handle you talking to it once an hour, then the cost of TCP socket open and close, as well as any TCP Keepalive is all wasted investment.

Not only this, but the cellular providers do NOT want users who send a simple, rather empty packet every 30 to 60 seconds - this is literally the worst kind of customer, as this forces the cell tower to "waste" one of its very limited airwave resources with almost no income returned to the carrier. From what I hear, carriers either want customers who talk constantly and pay huge monthly fees (say $90 to $350/month); or they want customers who rare talk and pay a small fee (even just $5/mo) but cost the carrier virtually no direct expenses.

Putting this is "restaurant terms":

A cellular data device that talks constantly but pays for a large plan is like a restaurant patron who sits at a table, constantly ordering more food and paying a larger bill.
A cellular data device that rarely talks is like the restaurant patron who comes in once a month, sits at a table, orders a meal, pays and then vacates the table.
A cellular data device that keeps an idle channel open full time but rarely talks is like the restuarant patron who sits at a table in the resturant, reading the paper but rarely ordering food or paying a bill.

In fact, in private chats with carrier account people, I have heard several times that they have been directly to prefer either customers who talk constantly on large plans or those who talk at most once an hour (better once a day) on small plans. Customers planning to talk every few minutes have been defined as bad investments. It may be fair to say that after years of building up the data-plan customer base, the cellular carriers have come to understand that the REAL cost of data plans is not the bulk data bytes moved; it is instead the percentage of time the device consumes (or squats on) 1-of-N scare airwave resources in proportion to the monthly fee they pay.

Report-By-Exception? Maybe not

2007-07-06T07:55:00.000-07:00

One of the PLC vendors I am just starting to deal with was bragging about the power of their report by expection abilities ... how wonderful for lower cellular data costs they claim. Unfortunately they only support TCP/IP - no UDP/IP. I'll cover this application (& name the PLC) more after I have some time to work at it.

However, the big problem with their "report-by-expection" is that the PLC holds the TCP/IP socket open 24/7 and sends a TCP Keepalive every 1 minute. Sending a TCP Keepalive (2 x TCP packet headers with no data) every 60 seconds costs about 3.4MB per month! I have a hard time seeing that as "low cost". They would be better to just poll real data every 4 minutes and eliminate the TCP keepalives!

Hopefully there is a setting in the PLC to change this behavior, which is why I need to look into this more. I just want to point out that just because a system claims to only move data during exceptions ... that does NOT mean it has very low cellular data costs. Don't forget that data costs include things you don't normally see - TCP socket open/close, TCP keepalive, TCP ack, and TCP retransmissions are all things you pay for.

Real World Cellular - ControlLogix PLC

2007-06-12T11:43:00.000-07:00

Summary: Before I listed some real world numbers for Modbus polling. This time I walk through some of the costs and issues of using ODVA Ethernet/IP to talk to a Rockwell ControlLogix PLC.

The Convoluted Path of Wide-Area-Networks:
In general the magic of IP hides reality from us all. We tend to think "now I am browsing Google.com or iatips.com", but we don't really understand how COMPLEX and MIRACULOUS this really is. Your computer is NOT connected to either of these web servers; instead your computer uses the services of a dozen or more other computers/routers to get from "here" to "there". Every single data byte must be forwarded hop-by-hop through all of these cooperative peers.

As example, here is a Trace Route (tracert) of access from a computer within my test lab to a ControlLogix PLC sitting six (6) feet away. I am using public Internet access via a cellular Digi Connect WAN to the Ethernet (ENB) of the ControlLogix. Some of the public IP have "X" entered replacing the digits; you don't need to really know the exact IP value.

My computer has private IP = 10.9.92.1
01 01 ms 10.9.1.1 (Digi's private Intranet)
02 01 ms 10.10.11.10 (Digi's private Intranet)
03 01 ms 10.254.254.2 (Digi's private Intranet)
04 16 ms 66.77.x.x (Digi Co-Host/Internet Link)
05 04 ms 69.8.x.x (Digi Co-Host/Internet Link)
06 64 ms 66.77.x.x (Digi Co-Host/Internet Link)
07 09 ms min-core-02.inet.qwest.net [205.171.128.110]
08 11 ms cer-core-02.inet.qwest.net [67.14.8.18]
09 12 ms cer-brdr-01.inet.qwest.net [205.171.139.62]
10 39 ms qwest-gw.cgcil.ip.att.net [192.205.32.97]
11 35 ms tbr2.cgcil.ip.att.net [12.123.4.254]
12 35 ms tbr2.sl9mo.ip.att.net [12.122.10.46]
13 75 ms tbr2.attga.ip.att.net [12.122.10.137]
14 31 ms 12.122.85.157
15 34 ms 12.86.140.146
16 * Request timed out. (Part of Cellular Infra-Structure)
17 * Request timed out.
18 * Request timed out.
19 * Request timed out.
20 1276 ms mobile-166-XXX-XXX-XXX.mycingular.net [166.XXX.XXX.XXX]
Digi Connect WAN has private local IP = 192.168.196.80 (is 'gateway')
ControlLogix PLC has private local IP = 192.168.196.21

These traces always amaze me - how something so seemingly trivial takes so much effort to really function. Notice how my lab PC has to route through 6 devices to even get out of Digi's company network, then through Qwest (our ISP), through AT&T (my cellular SIM provider), through some unnamed hops of the cell system, and finally be port forwarded to the ControlLogix PLC. The packets may be passing through Minneapolis, Chicago, Detroit, Atlanta, and then finally returning to the PLC sitting right beside me.

Effect of NAT (Network Address Translation)
Now lets look at what happens when RSLinx on my PC opens an ODVA Ethernet/IP socket to the ControlLogix PLC. Every TCP/IP packet requires 4 unique values which define a connection:

Destination IP (target device)
Destination Port (target application within device)
Source IP (return address to originator)
Source Port (likely random port, originator is waiting for responses here)

So we start out with the 4-tuple DST=166.x.x.x : 44818 and SRC=10.9.92.1 : 22256. The 166.x.x.x IP is assigned by my cellular carrier. Port 44818 is ODVA's "well-known" port for Ethernet/IP. 10.9.92.1 is an internal Digi selected private IP. TCP port 22256 is the ephemeral (or random) port selected by RSLinx to listen for responses.

The first NAT effect is the Digi corporate firewall changes the request to be DST=166.x.x.x : 44818 and SRC=66.77.x.x : 22256. My private IP of 10.9.92.1 is meaningless out in the Qwest or AT&T's networks, so something needs to swap this for a "real" world-unique IP leased by Digi. Our corporate NAT interface creates a record (with a lifetime of 5 minutes) that allows any responses to be correctly restored to 10.9.92.1

The second NAT effect is when the Digi Connect WAN forwards to the ControlLogix with another private IP. So the 4-tuple now becomes DST=192.168.196.21 : 44818 and SRC=66.77.x.x : 22256. The ControlLogix thinks IP host 66.77.something is connected to it - not the real host IP of 10.9.92.1. Plus the ControlLogix has NOT CLUE that the RSLinx thinks the ControlLogix as IP of 166.something.

Now, to send a response the ControlLogix issues a TCP/IP packet with the flipped 4-tuple of DST=66.77.x.x : 22256 and SRC=192.168.196.21 : 44818. The Digi Connect WAN restores (undoes) the NAT and changes this to DST=66.77.x.x : 22256 and SRC=166.x.x.x : 44818. After passing back through AT&T and Qwest, Digi's corporate NAT interface restores its own NAT and changes it back to DST=10.9.92.1 : 22256 and SRC=166.x.x.x : 44818.

This understanding of NAT and IP is useful for understanding the capability and limitations of cellular access to certain devices with certain protocols. A future entry will cover setting up RSLinx Classic and using RSLogix 5000 to download over cellular to a L5555 processor.

Unlimited Cellular Data - Revisted

2007-05-31T12:23:00.000-07:00

As I go around dealing with large utility customers and hear feedback from Digi salespeople who survive large accounts, I hear some interesting but disturbing trends rising.

First, there are the people (usually who are new to cellular) who claim any day now the age of cheap, unlimited cellular data plans means all my hard work to understand or offer reduced traffic are wasted effort. I especially hear this coming from European partners.

Then there are the other people ... people I know work with very large, very powerful end users who fail to get the cellular plans they desire. Things I hear:

I have heard of customers who pilot projects and hear promises of unlimited traffic, but when the time comes to sign the contract for 3000+ sites, the carrier decides that they cannot offer unlimited traffic ... period. Hmm, I guess this is the difference showing between the carrier's commissioned sales staff and the business managers who need to keep the cellular system profitable.
I have worked with large customers who do manage to get "unlimited deals" for a modest sized system - say 50 or 100 sites, but the carrier insists on adding the 2 clauses 1) the carrier has the right to artificially slow down the data communications (without detailing what this means) and 2) the carrier has the right to just stop all the customer's data traffic temporarily if the cell system gets busy (again, not details of this defined). Hmm, I have to kind of wonder what kind of control engineer agrees to such "clauses"? You'd think trying to get one's data under control to avoid the need for unlimited data is a wiser design.
I have heard that overall the cellular carriers are starting to rethink the value of machine-to-machine data plans. Unlike DSL or cable, this is NOT an issue of bandwidth; it is an issue of the % of the time the device "hogs" 1 of N slots or resources on the tower forever. Imagine having 10 or 20 such devices squatting there, locking up that tower resource. So it is not even so much an issue of talking once per few minutes verse flat out unlimited traffic. In either case 1 of N finite tower resources is used, so long-term the only "good" data plan may be for a data system using the cellular resource every few hours or a few times a day.

So overall it looks like my efforts to understand and reduce traffic is useful.

I will give Webinar 31-May-2007

2007-05-29T12:00:00.000-07:00

31-May-2007 (Thursday) I'll be giving a webinar related to outdoor use of IP-to-serial devices. While not directly related to my blog theme of "Industrial Protocols over IP", remote devices frequently are in bad locations, where extended temperature ranges, conformal coating, and Class 1 Div 2 certs are desired.

Title: Connect Local and Remote Devices in Hazardous Environment SCADA Applications
Register here: WebEx Link

Date: May 31, 2007
Time: 11:00am CDT (central zone)
Duration: 1 hour
Speakers:

Lynn Linse - Engineer, Digi International
Deb Smith - Business Development, Digi International

What you will learn
This live webinar will illustrate the value of using TCP/IP & UDP/IP based communication over wired and wireless networks to monitor and manage local and remote devices. Topics include:

Overview of requirements for robust, remote, outdoor communication devices
Discussion of extended temperature, conformal coating, hazardous certs
Comparing features of IT-grade network devices to Digi's Haz product line
Extending IP networks to remote serial and Ethernet devices
Moving serial communications through TCP/IP and UDP/IP
Migrating from analog Telco/POTS lines to IP-based broadband
Real-world usage examples
Implementation benefits
- Real-time access to remote sites and process data without site visits
- Reduce repair/replacement costs due to less-than ideal environments
- Options to reduce installation costs with wireless

Please register today and join us May 31st for this informative webinar!
Questions? Contact Deb Smith at deb_smith@digi.com or 952-912-3283.

ODVA, Rockwell, and Modbus Get Smart

2007-05-04T12:53:00.000-07:00

Summary: next week an amazing joining of technologies will take place at ODVA in Ann Arbor MI as ODVA, Rockwell, and Schneider-Electric discuss how to integrate Modbus servers (slaves) into CIP Producer/Consumer systems

One of the fun things about being involved in "multi-vendor" solutions is when you recognize moments of amazing sanity as they occur. One such moment of amazing sanity is occurring next week when ODVA (aka Rockwell / Allen-Bradley) and Modbus supporters (aka Schneider-Electric / Modicon / SquareD / Telemecanique) sit down to discuss how to integrate Modbus devices into the ODVA Ethernet/IP and CIP network systems. Of course there must be some interesting hidden politics behind this move - and I somewhat light-heartedly believe that perhaps French Schneider-Electric sees joining with the Americans (Rockwell/ODVA) as the lesser of two evils when compared to joining with the Germans (Siemens/PNO).

Check out: ODVA Call For Members: Modbus Integration JSIG The kick-off meeting for the Modbus JSIG runs from Thursday, May 10, 11:00 AM to Friday, May 11, 04:00 PM

Side-stepping the marketing fluff and platitudes of a brighter future such meetings evoke, small third-party suppliers and the folks on the plant floor can expect the following benefits. Regardless of the directly stated goals of ODVA, Rockwell, or Schneider-Electric, small vendors will implement solutions that include these abilities:

Vendors making Ethernet Modbus/TCP products will have a simpler "first step" to adding full ODVA/CIP support without the somewhat overwhelming task of 100% conversion of a word-array device model into hundreds (or thousands) of CIP objects.
ControlLogix PLC will be able to connect through Ethernet-to-Serial devices to multi-drops of Modbus/RTU slaves. For example a user with a dozen small Modbus/RTU PID loop controllers will be able to add an Ethernet-to-Serial device to read via Modbus and cyclically produce a small block of word data from each loop controller over Ethernet.
HART, Bluetooth, ZigBee and other new technologies which offer Modbus interfaces will find a instant place as sensors and I/O within CIP and Rockwell systems.
Since Siemens, GE-Fanuc, Omron, Mitsubishi and most major PLC brands offer some method to act as Modbus slaves, users with any of these PLC will be able to integrate them within the CIP Producer/Consumer system.

Looking forward a year or two, I also can foresee some other secondary benefits evolving from this JSIG's work:

I started working with ODVA Ethernet/IP almost 8 years ago and still as-of today the legacy PLC5E, SLC5, and serial MicroLogix (the old PCCC-based PLC) don't have effective inclusion within CIP Producer/Consumer systems. Since the device model of PCCC PLC shares much in common with Modbus PLC, it is a very small enhancement to add a similar support for AB PLC - perhaps AB will actually extend this to future firmware updates to Ethernet-based PLC. In fact, since my Digi One IAP code already allows Modbus masters to query DF1 and CSPv4 slaves as-if Modbus slaves, as soon as Digi adds Ethernet-to-Modbus support per this JSIG's output users of older AB PLC will gain access to CIP Producer/Consumer systems indirectly as honorary Modbus slaves.
Today legacy Modbus and Modbus/TCP systems lack any simple form of multicast producer/consumer exchange. While the IDA protocol offers this, IDA is so many orders of magnitude more complex (and resource hungry) than simple Modbus as to become really an "unrelated" protocol. Any specification that defines a "server interface" naturally implies a corresponding "client interface". So although this ODVA JSIG is not planning to define how Modbus "peers" could use multicast to exchange cyclic data, the end result will be a fairly natural and multi-vendor method to do this. So while I doubt many pure Modbus/TCP products would implement ODVA protocols just to gain this multicast exchange, any products which add the CIP support anyway will naturally add the last few bits of code required to enable pure Modbus-to-Modbus multicast via the ODVA mechanism.
Taking the above point to its natural conclusion means Modbus/TCP masters which implement the Modbus JSIG's "server" function will also gain a mechanism to access CIP Producer/Consumer systems. Even if the ODVA JSIG doesn't cover how to do this, natural methods will be inferred, produced, and copied by vendors to make this a fairly common new product feature.

And lastly, it will interesting to see what response PNO considers to likewise include Modbus slaves more formally into PROFINET. Personally, I have always felt that the simpler PROFINET IO protocol would fit very naturally with multi-drops of Modbus slaves acting like plug in I/O modules ... each slave would be a module within the IO device.

Rockwell and Modbus Data Traffic

2007-04-25T10:09:00.000-07:00

Summary: I compare the data costs for four common off-the-shelf PLC protocols to a remote cellular site. As a rule of thumb, even if you have an Ethernet-based PLC your lowest data costs for SCADA-style periodic polling are obtained using serial protocols via the PLC's serial port. Their modern "Ethernet protocols" are very wasteful of cellular bandwidth.

PLC Protocol Example:
A simple but realistic SCADA scenario is to poll every 15 minutes and read 10 words of data and write 2 words of data. This commonly requires 1 Read command and 1 Write command (I'll ignore the rarely supported Modbus command that reads & writes within a single command.)

While there exists special SCADA protocols and special products that optimize remote traffic, I am not looking at those protocols at the moment. Instead, since cellular and satellite access to remote IP and Ethernet products has enabled people to use off-the-shelf PLC technology, I am looking at the more traditional PLC protocols. These are things which affect users when they apply an Ethernet design to an IP-based wide-area-network system.

I compare these 4 PLC protocols:

AB/DF1 Radio Modem (RM) encapsulated in UDP/IP. DF1 RM is basically DF1 Full-Duplex with no ACK/NAK and is supported by the SLC5 and MicroLogix line.
Modbus/RTU encapsulated in UDP/IP. Modbus/TCP within UDP/IP is roughly the same size.
AB/CSPv4 in TCP/IP as supported by SLC5/05 and PLC5E MSG blocks.
AB/Ethernet/IP as moved by ControlLogix Explicit MSG blocks to PCCC-based remote PLC. Note that Ethernet/IP "I/O Messaging" does NOT work through NAT'd wide-area-networks since the protocol embeds IP information within the data packets and is thus is "broken" by NAT.

The bytes per 15 minutes includes the IP headers, any PLC protocol overhead, and the actual data moved for each update (always 20 + 4 bytes). The MB per month is just defined by 30 days of such polling, or 2880 updates at once per 15 minutes. The 2 serial protocol have been encapsulated into UDP/IP and I assume the remote IP end-point can handle connecting this to the PLC's serial port.

Protocol	Transport	Per 15 Min	MB per month	Relative Cost
Ethernet/IP	TCP/IP	1202 bytes	3.46 MB	100%
AB/CSPv4	TCP/IP	960 bytes	2.76 MB	80%
Modbus/RTU	UDP/IP	166 bytes	0.48 MB	14%
DF1 Radio Modem	UDP/IP	194 bytes	0.56 MB	16%

The two Rockwell "Ethernet" protocols cost a lot more to use in part because they force use of TCP/IP, and therefore suffer the repeated cost of TCP socket opening and closing, plus extra TCP acknowledgment overhead. They also suffer because they both involve connection registration and service functions that needlessly repeat every time the connection is reestablished. While the actual data packets of these protocols are roughly twice the size of the serial encapsulated protocols, the real burden they suffer is all the extra TCP/IP packets exchanged that do NOT directly involve field data update.

Both the serial Modbus/RTU and DF1 Radio Modem benefit that they move no IP packets that don't relate to the field data update - no TCP/IP open or close or acknowledgement; no protocol "service function" overhead. Each moves just 1 read request and 1 read response, plus 1 write request and 1 write response.

Discussion of Other PLC Protocols:

Most other PLC Ethernet protocols will either approach the costs of the AB/CSPv4 - or they won't work at all due to use of direct "Ether-Types" and lack of IP compatibility. Most serial protocols with roughly either match the 2 show here or be twice the cost if protocol ACKs are used by the protocol.

Modbus/ASCII will almost double the cost of Modbus/RTU since each data packet is roughly twice the size. But this wouldn't increase the IP overhead any.

Using DF1 Full-Duplex instead of Radio Modem would effectively double the cost over DF1 RM since DF1 Full-Duplex moves the protocol ACK/NAK, which doubles the IP header overhead also. Using DF1 Half-Duplex would triple or even quadruple the costs since HD not only moves protocol DF1 ACK/NAKs, but the ENQ/EOT polling overhead.

Most other protocols I am aware of - such as Omron Hostlink, GE-Fanuc SNPX, and Siemens PPI - would cost roughly 2 to 3 times more than Modbus/RTU or DF1 RM since they include protocol ACK, while a few even encode many parts of the message as ASCII or BCD form instead of as binary.

Rockwell PLC5E and SLC5 over Cellular

2007-04-19T09:18:00.000-07:00

Summary: Customers ask me "How much will cellular cost" - this blog post walks through some examples of SCADA-style periodic polling of AB/PLC5E or SLC5/05 using the CSPv4 protocol (aka AB/Ethernet to 3rd parties) over TCP port 2222.

Real-World Numbers
For a simple SCADA-style example assume we need to read 10 words of data (20 bytes) and write 2 words (4 bytes) every time period. Obviously there would be simple optimizations to this, such as only writing data which changes or using PLC MSG blocks to push data from PLC to SCADA only when something changes. However my goal in this blog post isn't to "tweak" a solution to minimize cost, but to examine the protocol impact of using Rockwell CSPv4 over IP.

The table below shows the megabyte per month when polling once per second, per 5 seconds, per 1 minute, per 5 minutes, per 15 minutes, and per 1 hour. There are lots of variables considered ... and many more ignored. The traffic ranges from worst-case of 1005.0 MB for TCP/IP with larger header options polled once per second to best case of 0.2 MB for UDP/IP polled once per hour. This assumes the use of the CSPv4 submode 7, with local LSAP addressing and ignores that Rockwell PLC5E and SLC5/05 don't support CSPv4 within UDP/IP. Raw efficiency at moving the data bytes ranges for about 10% for UDP/IP to barely 1% for TCP/IP; which means most of what you are paying for is not related to actual, meaningful field data.

( Click this image to see a larger version )

(is at http://iatips.com/blogimage/rockwell_cspv4_traffic.png)

Notes on the Table

Since this example reads and writes small amounts of data, it assumes a SLC5-style Protected Typed Read with 3-Address Fields and the corresponding SLC5 write.

The smaller 40 byte TCP/IP header has no options attached; the larger 52-byte TCP/IP header includes the RFC 1323 Timestamp and Window Scale TCP options. These appear to be the normal default for Linux and easily becomes enabled under Windows since all applications share a single setting in the Registry.

The two time columns "15 min (Alive)" and "1 hr (Alive) assume a roughly 4 min 45 sec TCP keepalive to prevent the socket from closing. This reduces the traffic by the extra open/close overhead in exchange for billable TCP Keepalive packets. Keep in mind this ALSO requires the PLC to be properly configured to NOT close the idle sockets. By default, my SLC5/05 seems to close the idle connections in a few minutes.

The two time columns "15 min (Cls)" and "1 hr (Cls) assume the socket is closed after the a data polls, and the TCP socket and CSPv4 session must be reopened for teh next poll.

Discussion
Of course the standard costs of using TCP/IP verse UDP/IP apply:

TCP/IP uses larger headers, ranging from 40 to 52 bytes per packet as compared to UDP/IP's smaller 28 byte of header.
TCP/IP involves the TCP Acknowledgments, which may result in separate, billable 40 to 52 byte packets moving frequently without any meaningful field data.
TCP/IP may require reopening a socket, costing 120 to 250 bytes per open, plus closing costing from 160 to 400 bytes. Exact sizes are hard to predict since both opening and closing of sockets tend to be "pushed" and result in excess retransmissions and retries when high network latency is true.
TCP/IP over unknown 3rd party wide-area-network infrastructure requires at least 1 TCP packet to move every 4 minutes 45 seconds to maintain health. This means either a data packet or a TCP Keepalive with data.

It should be clear to see why using UDP/IP over cellular (which is very reliable) is much cheaper than using TCP/IP. There are no socket open, close, acknowledgment, or keepalive costs. Plus field experience has shown that rapid IP retries rarely succeed. For example, a customer polling with UDP every 5 minutes with 3 explicit retries if no answer will likely have the original poll succeed or that poll and all 3 retries fail. Again, cellular is very reliable in that packets almost always make it through unless there is a network or congestion issues and then only time (a few minutes) solves the problem. So such customers have abandoned retries and just ignore 1 failed 5 minute poll and "retry" in 5 minutes.

CSPv4 issues include:

Rockwell PLC and software tools do NOT support use of UDP/IP - my tests with UDP/IP have to be conducted with the Digi One IAP which happily bridges CSPv4 between TCP and UDP (as well as to or from Ethernet/IP and DF1).
CSPv4 requires the exchange of a pair of 28-byte negotiation TCP packets when a new TCP socket is opened to inform the client (master) of a server (slave) assigned session handle. This nearly doubles the overhead of an open-poll-close socket paradigm.
The 28 byte CSPv4 header really contains little useful information; such excess bytes cost nothing tangible under Ethernet but cost cash in the form of requiring larger cell plans over cellar.

In conclusion, CSPv4 will be a rather poor choice for raw, periodic polling of remote AB PLC. ODVA Ethernet/IP (as implemented by all vendors) is even worse.

Your only effective solution at present is to carefully craft a set of MSG blocks to push data from the field in a report-by-exception paradigm. Of course you also must include safe guards within your PLC to prevent rapid, repeated MSG block triggers during system failure that could cost you thousands of dollar ($$$) in a few days.

DF1 Open Source for Visual Basic 2005

2007-04-16T09:11:00.000-07:00

Archie has started a SourceForge project for his AB DF1 code running under VB 2005. He has checked in a full first set of files (unlike many SourceForge projects which get created but NEVER have files :-\ )

http://sourceforge.net/projects/abdf1/

I haven't looked over his code yet, plus all I have is VB 2003 .NET.

Hopefully Microsoft has STOPPED the old VB issue that each new rev of VB is neither 100% forward nor backward compatible ... one always need to "tweak" a few lines to make the port work. I've used VB 1.0, 3.0, 4.0, 5.0, 6.0 and now VB 2003 and none of these have liked old code being pulled forward.

Cellular to Allen-Bradley SLC5/05 on TCP 2222

2007-04-09T12:05:00.000-07:00

The old CSPv4 protocol.

The Rockwell/AB SLC5/05 and PLC5E natively speak an older "unpublished" protocol named CSPv4, although most third party vendors call it either AB/Ethernet or AB/TCP. It moves only on TCP port 2222 - ODVA Ethernet/IP I/O Messaging is only on UDP port 2222, so they don't conflict. The protocol consists (normally) of a 3-part packet:

28-byte header
4 or 15-byte LSAP or end-point addressing packet
PCCC message which is basically what DF1 documents as an Application Packet

In general, the packets are fairly sparse and compressible (if you have the tools to do this). The characteristics of CSPv4 which impact cellular (and wide-area-network) support:

Rockwell tools and PLC only support use of TCP/IP and port 2222; this greatly limits use of CSPv4 in NAT'd networks since the remote NAT router can only forward TCP port 2222 to a single remote PLC.
CSPv4 includes a single TCP packet exchange to "register a session" or connect. If you are polling faster than the PLC will hang-up on you, then this is not important. However, if you poll slow enough that a new TCP/IP socket must be opened for each poll, then even ignoring the TCP socket open/close overhead this nearly doubles your traffic costs.
In tests, a SLC5/05 seems effective at including the TCP ACK response to the host within the CSPv4 data response packet, so you only have to pay for one empty TCP ACK, which is the host's acknowledgment to the PLC for the response.
TCP Keepalive could be an issue, since most hosts fail to issue it and the SLC5/05 I've tested against either doesn't issue TCP keepalives
or does it very frequently.

Interested in Cellular? Setup a DMZ Lab

2007-04-05T06:30:00.000-07:00

Summary: Last post I suggested people interested in cellular data start by learning how to use their home cable router. In this post I suggest the next step, of how to make your life easy at work once you're ready to start testing real cellular or satellite access by IP.

Your Second Step should be to set up a simple, isolated low-speed broadband link at work ... create your own DMZ lab.

Sigh - I waste so much time listening to customers complain about how difficult it is to get the IT department to give them custom firewall permissions. Since modern "Security" wisdom is to block everything until proven safe, I waste more time asking customers complaining that their Modbus/TCP or Rockwell access not working to first talk to their IT group to make sure they aren't blocking unknown binary traffic by default. I waste yet more time when customers struggle for days and finally have to formally get someone in IT to help study the corporate firewall logs to see if any traffic is getting through or not. An interesting epiphany occurs when I suggest they just look into paying roughly $50 per month for a private connection for this. It is surprisingly cheap to do this and makes a lot of people's jobs 200% easier.

So far the feedback from customers has been quite positive, with IT departments over-joyed at the idea (slight exaggeration :-] lessor-of-two-evils may be a better term). This really makes sense; IT is charged with keeping the corporate system working and secure, so when you ask for yet another odd, unknown firewall hole to be opened, you ask them to risk their jobs. Plus trying to keep custom firewall settings updated for 50 different projects is an ongoing headache and ongoing risk for mistakes. I know that Digi's IT group is very satisfied with their policy of not offering custom firewall rules on the corporate LAN but instead helping teams set up such private connections in a safe, isolated manner.

Simple DMZ Lab Design
The simplest lab design is little more than a copy of what you have at home: a computer or two, an 8 or 24-port Ethernet switch, and a simple NAT router to "share" the internet connection with a dozen devices. This allows you to freely set up a few OPC servers and Master PLC to test access to remote cellular and other wide-area-network based systems.

Locate an empty office or lab room for your new network. Perhaps your IT people should pull out or disable the corporate Ethernet in this room. You are going to create a small "DMZ"; a small isolated network that has limited security consequences if you goof up and let a hacker inside. You'll want good security tool installed on your Windows and Linux computer used in here.
Arrange for a low-speed business broadband link with one fixed IP address. 256Kbps is more than enough for general PLC/SCADA testing and should cost in the range of $35 to $50 per month. Yes, just $35-50 per month! I had one customer forced to pay his IT department $100 per month to open one TCP/IP hole in the corporate firewall!! Gee, he could install 2 DSL links for that. Now, be patent when you talk to your carrier, as they are geared to sell the expensive primary access lines used for all corporate traffic including servers. Keep stressing that you want a low-speed secondary line for use with some network testing and eventually you'll locate the low-cost plans you want.
Set up a DNS name for your DMZ lab. Online dynamic DNS providers support user-selected DNS names for static IP addresses. I use dyndns.org for both my dynamic and static IP but there are many out there. You won't need any form of DDNS update client since your IP never changes and you must enter the name manually anyway.
Unless you plan to implement large VPN systems, just buy a nice consumer-grade DSL/Cable Router ... the same kind you use at home is fine. If you plan to set up a serious VPN infra-structure, then you'll just need to bite-the-bullet (& suffer the learning curve) of buying a commercial IT-grade router with VPN server capability built in. Be warned that while many consumer-grade routers mention "VPN Support", they are in fact sub-optimized and documented only for home-office users who connect into a corporate Windows or Cisco VPN server. Normal human beings will find them nearly impossible to set up for anything else!
Do you want more than one public IP address? You need to pay a monthly surcharge which varies greatly per carrier, but could be in the range of $25 per month for 8 IP addresses instead of just 1. Plus you will need a larger IT-grade router since the consumer-grade routers won't support more than 1 public IP address. Most users won't need more than 1 IP address. However, having more than one IP address is helpful if more than 1 team shares the lab; this prevents them from trying to setup conflicting router configurations. Also, a few extra IP are helpful if you want to place a PLC "online" for your customers or sales force to access during customer-site demonstrations in the field.
If you need to access your corporate network from your DMZ lab, then you need to arrange some rules with your IT people. Perhaps the rule is using a notebook computer with 802.11 wireless to the corporate network is Ok as long as the notebook is NEVER connected to the lab's Ethernet. Remember, since your lab has it's own public IP address you can even use FTP or a VPN client to connect "legally" out your corporate network and back into your lab via the Internet.

Fancier DMZ Lab Design
Since Digi is basically a "communication company", the lab I get to use is much fancier. It has 32 public IP addresses and even limited secure access from the corporate LAN. Of course I have to share this lab with other teams, so I'm not owner of 32 IP. As an example, here is how my lab is setup:

Digi's IT group maintains a Cisco PIX router (a mid-range $800 model) that manages the 32 public IP addresses. Actually, this is NOT a complication since this router does not by default firewall any traffic; it merely distributes raw traffic based on public IP to one of many to internal IP addresses in an organized manner. I was lucky enough to get in the lab early and to be assigned 2 of the 32 public IP addresses.
My first IP address receives 100% raw internet traffic at an internal static IP I selected; so an external IP such as 70.x.x.140 forwards to my internal IP of 192.168.20.159. Since the goal of the lab is to avoid burdening IT with TCP/UDP port forwarding chores, one could place a consumer-grade DSL/Cable router at this IP. Placing 2 routes in series is NOT a problem - do a net-trace of how you access www.google.com and you'll see a dozen or more routers in series. Instead of a pure hardware box I have a Ubuntu Linux machine running firewall and router tools at this IP. This is where I forward Modbus/TCP to one PLC and Rockwell Ethernet/IP to another PLC. I prefer the Linux box to the $39 hardware box because it gives me a richer view of traffic in and out, plus I can run an Ethernet sniffer such as WireShark to see a complete trace of the 2-way conversation taking place.
For my second IP address I had Digi IT setup the PIX router to just forward a simple, safe list of Modbus, Rockwell, Digi, and other industrial protocol TCP and UDP ports. I normally have a Digi One IAP (an industrial-protocol aware Ethernet-to-serial device) at this IP address, but I can safely swap in a Windows machine when a test requires use of Windows tools.

Just as your home router does, the main Digi IT-supplied PIX router does out-going NAT to the internet and internal DHCP address assignment. So our DMZ lab has its own internal subnet of 192.168.20.x addresses. Any device in the DMZ lab can access the Internet - with or without going through my Linux firewall/router. Of course the other teams in the lab don't go through my router, but direct to the PIX router.

Since i study cellular usage, I have have a Digi Connect WAN providing an Ethernet-based cellular router in the DMZ lab. So I really have 3 potential routers to use - while it takes a bit of IP experience to not get confused, this allows me to have devices configured to selectively treat any 1 of the 3 routers as "the default gateway/router".

For example, I can have a Master/Client device connect out to a cellular-based Slave/Server using a route such as Master => out PIX+DSL => in Cellular => Slave.
For example, I can have a Master/Client device connect via cellular to a DSL-based Slave/Server using a route such as Master => out Cellular => in DSL+PIX => in Linux-Box => Slave.
In both of this situations I can see BOTH ends of the conversation, which is a huge help in testing, timing, or troubleshooting new applications.

The bonus for having Digi's IT team involved is the PIX router also allows controlled, secure access into the DMZ lab from our corporate LAN. Technically, the PIX runs a set of rules similar to those used by the main corporate firewall out to the Internet and it just treats the 192.168.20.x subnet as a miniature internet. So I can sit at my desk and safely check on equipment running in the DMZ lab. Of course this is limited to equipment treating the PIX as the default router - if you understand IP routing you'll understand why, but at least it lets me log into my Ubuntu Linux router from desk.

Interested in Cellular? Do some homework

2007-03-27T12:01:00.000-07:00

Summary: Unless you are a pro at IP routing, you'll save money and time by learning the basics of IP routing over your home Cable/DSL connection instead of a demo cellular account. Bottom-line ... if you cannot succeed at using your office computer to poll a PLC placed at home via your Cable/DSL connection, then you will NOT succed trying the same trick over satellite or cellular connections.

Homework - Work at Home
When engineers first launch into a cellular data pilot it can be a bit like Christmas with the excitement of new toys, future trends and being "on top of it". However, I encourage anyone interested in using cellular or satellite-based IP systems to do some home work first ... literally "work at home". You'll save lots of cash and avoid many headaches by learning the basics at home first.

Most of you have cable or DSL router/modem at home, so start there. Take a PLC or controller home. If it has an Ethernet port you are all set; however if your device is RS-232 based, then beg, steal, borrow, or purchase a simple Device Server such as the Digi One IAP (fancier, Modbus and Rockwell protocol aware) or the Digi One SP (much cheaper but just a raw Ethernet-to-serial converter). Your goal is to connect from your office computer over the Internet to this device at home ... if you cannot succeed at this, then you won't succeed at cellular access either! But unlike with cellular, all of your trial-and-error over your Cable/DSL Route won't be costing you by the byte.

Just remember that your "Home Cable/DSL Terms and Service Agreement" likely forbids running "servers" so don't go and try to setup an e-commerce shop once you see how easy it is to access your home from the Internet.

Get to Know Your Cable/DSL Router Box
Hopefully you all have an external commercial router box that you either got from your ISP or bought at any big-box store for $39 to $59. If your computer connects directly into your modem or you were fooled into using Microsoft's "Internet Sharing" tool on one computer, save your sanity and go buy a cheap router box! For your $39-59 you get a 4-port switch, a professional stateful-firewall and NAT (more about that later), a wireless access point, and it all consumes maybe 8-10 watts of power so costs you a few $ a year to run. If for no other reason, you just don't want the mindless broadcasts and hacker probes taking a percentage of your home computer's bandwidth. For my VPN testing I have some Linux boxes up exposed like this and they see up to 50 broadcasts per second and a few dozen probes for open Windows and Unix services per hour. There is NO REASON to expose your home PC to this rubbish - use an external router box ... period.

Step 1: Learn how to log onto your Cable/DSL Router.

Under Windows 2000 or newer, open a command window and type the command "ipconfig". You should be shown your computer's current IP Address and the Default Gateway, which is another name for your Cable/DSL router. Most likely the router has an IP such as 192.168.0.1 or 192.168.1.1.
Confirm you can ping your Cable/DSL router with this IP
Open your web browser and browse to the address - as example type the URL "http://192.168.1.1". You should be asked for a user name and password.
Check with your router documentation or go on line to the vendor and read the user guide. For example, at home I have an ActionTec router/wireless access point supplied by Qwest, and when it first came it has no user name and a password of "admin". This is actually not so insecure since by default you can ONLY access this web page from inside your firewall/router. But common sense says changing this name/password is wise.
There is no way I can explain how all Cable/DSL routers work, but once you can log in you should be able to find a status web page which gives your currently assigned external IP address and 2 DNS addresses. This is how the world sees your home system - write this info down. For example, my home Cable/DSL router (as of today) has the temporary (dynamic) IP of 63.228.51.x.

Step 2: Nail Down a fixed DNS name for your Cable/DSL Router.
So at this point, you know how to access your raw "face" exposed on the Internet. Now we want to give ourself a nice, memory-friendly DNS name to represent that face.

As mentioned above, my Qwest IP is dynamic and liable to change at any time. So while I could go to the office and try to point my OPC server or PLC software at 63.228.51.x, I can never be sure how long this will work. In reality it only changes every few months or if I power-cycle my router, but the solution to this problem is very easy so we should solve instead of work-around it.
Sign up with one of the many free online Dynamic DNS providers - I use dyndns.org. The Digi Connect WAN (cellular router) family directly supports this, as do many LinkSys and DLink-class home products. In a nutshell, they allow you to create a domain name such as sillyjoe.gotdns.org or sammy345.dyndns.org and then a client tool on your home system automatically updates this DNS name every time your ISP changes your dynamic IP address.
While the above service is free, you may want to pay the $10 or so per year for a minimum account. This makes the service more tolerant of errors on your part - for example many services automatically delete your free account if it is untouched for 45 days and so on.
If you have a Windows computer, the easiest DDNS update client is just to download the Windows tool recommended by your Dynamic DNS provider. This client automatically monitors the Cable/DSL router's IP as it accesses the internet. If your IP has changed, it correctly updates the DDNS (dynamic DNS servers). I stress the word "correctly" since many external Cable/DSL Router boxes which support DynDns and such services come with bugs which cause your free service to be deleted within hours of setup. So if you chose to use your Cable/DSL Router to maintain your DDNS name, make sure you have the latest firmware upgrade on it!
Within an hour of setup, anyone in the world should be able to ping your new DDNS name and get a response.

Step 3: Learn how to Port-Forward within your Cable/DSL Router.
We are almost ready to try access - but if you point your OPC server at your DDNS name ... nothing will happen since your Cable/DSL Router does NOT understand Modbus or other industrial protocols. Remember, the IP your DDNS name represents is the IP address of your Cable/DSL Router and NOT the IP of your home computer nor is it the IP address of your PLC/controller device.

Log back into your Cable/DSL Router and locate the setup for port forwarding. Some routers call it setup for applications and games. We need to configure the router to FORWARD specific TCP and UDP ports to Ethernet-based devices you have at home. If you don't know what that means, you are in for a tough time using wide-area-network technologies - I suggest you go to any bookstore and buy a book on basic networking that covers what TCP, UDP, IP and NAT are. This is really key to success in this area. You don't need to be an expert, but you do need to understand the basics!
In summary, if you think of the IP address as being synonymous with the main phone number in a building (aka - how to telephone the building), then the TCP and UDP port numbers are synonymous with phone extensions within that building (aka - how to reach a certain department or service). So for example, a Modbus/TCP OPC server will connect to your router using the IP (main phone number) attached to your DDNS name, and then request a connect to TCP port 502 (the service). We need to configure the router to accept and forward the Modbus/TCP traffic to your Modbus PLC. So take the example of a Modbus/TCP device on my local network with the IP 192.168.1.105. We need the router (at say IP 63.228.51.x) to accept any incoming connection on TCP port 502 and forward the packets to the local Ethernet device at IP 192.168.1.105 TCP port 502. Since the PLC has a web server and most ISP block access to home web servers, we'll tell the router to forward TCP port 8080 to local port 80. Depending on the brand of router you have the configuration can get fancier than that - but basically we'll end up with a line in the table looking something like:

Incoming port	Service	Local IP	Local Port
502	TCP	192.168.1.105	502
8080	TCP	192.168.1.105	80

Step 4: Get to Work Learning
That's it - at this point you should be able to use a Modbus/TCP OPC server to poll your PLC indirectly by polling your DDNS name on the standard TCP port 502. Pointing your browser to http://your DDNS name:8080 will pull up your PLC's web pages (the ":8080" tells the web server to use TCP port 8080 instead of default 80).

Of course there is no security offered here - anyone in the world can access your PLC so this is just for educational purposes. Here are some common port numbers to use:

Modbus/TCP uses TCP port 502
Digi Ethernet-to-Serial products use ports like 2101, 2102, etc to access a serial device by raw TCP or UDP sockets.
Digi RealPort uses TCP port 771 (TCP 1027 for SSL/TLS secure connection).
Rockwell AB PLC5E and SLC5/05 use TCP 2222 for the older legacy CSPv4. This is often called AB/Ethernet or AB/TCP by 3rd party vendors
Rockwell ControlLogix and ODVA Ethernet/IP uses TCP port 44818, UDP port 44818, and UDP port 2222. But be warned Rockwell tools are very poorly designed for wide-area network use.
Siemens S7 protocol uses TCP port 102
GE SRTP uses TCP ports 18245 and 18246
GE QuickPanels use TCP port 57176 for configuration

Ok, so now you should be able to take any of your Ethernet or serial device and test to see if you can access remotely. You'll likely need to slow down your tool - increase the response timeouts from a few seconds to 10 seconds for direct Cable/DSL access and 30 to 60 seconds for cellular.

Rockwell PLC and TCP Headers

2007-03-14T06:14:00.000-07:00

I have started running some tests of standard Rockwell protocols querying off-the-shelf Allen-Bradley PLC, with the goal to create a series of "estimators" for traffic. A user would enter the data to poll and the tool will estimates the data byte load contributed by this poll pattern.

The Mystery 17% Cost Increase:
Last night I ran a test polling ten words once a minute from an Allen-Bradley SLC5/05C's N7 file over GSM. This is nothing exotic - I ran similar tests a few months ago and had preconceived ideas of what to expect ... beep ... wrong! In between Then and Now, some unknown application changed my Windows XP system registry, enabling the "RFC 1323 Timestamp and Window Scale TCP options". The end result was an unexpected 16.51% increase in data byte traffic with no perceived value.

I have no clue which tool did this; and unfortunately Windows (at least 2K and XP) use a single global setting for the entire TCP stack. I could change it back ... but would that break this other mystery application? Will this other mystery application just change it back? Will I launch a mini cold-war race as this mystery application tries to keep RFC 1323 enabled and my test tools try to keep it disabled?

The Byte Counts with and without RFC1323:
Here is an exact accounting of the change in byte counts - remember, cellular is basically a mobile-IP tunnel which moves TCP/IP or UDP/IP as pure data payload. So you pay for both the IP and TCP headers, plus any data-less TCP Acknowledge or Keepalive packets.

I'll ignore the opening and closing of the socket, plus TCP Keepalive since I'm polling fairly steady-state once per minute. The PLC includes the TCP ACK in the response, so at least we avoid 1-of-2 data-less TCP Acknowledgments.

	no RFC1323	with RFC1323
Request: IP header	20	20
Request: TCP header	20	32
Request: CSPv4 Packet	42	42
Response: IP header	20	20
Response: TCP header	20	32
Response: CSPv4 Packet	56	56
Client ACK: IP header	20	20
Client ACK: TCP header	20	32
Client ACK: (no data)	0	0

	no RFC1323	with RFC1323
Total Bytes per Poll	218	254
Total Bytes per Hour	13,080	15,240
Total Bytes per Day	313,920	365,760
Total Bytes per Month	9,417,600	10,972,800

So this means a user doing 1 read of 10 words per minute would magically see a 16.51 % increase in data traffic ... just because they (or the IT department or even Microsoft Windows Update) changes a hidden registry setting. This is yet another example of both how hard it is to keep tight control on your cellular data costs; plus adds to my belief that using off-the-shelf host applications over cost sensitive IP networks is a losing battle. At some point you'll need a tool or device which is 100% "under-control" when it come to packet creation.

Windows Registry Details:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts

Tcp1323Opts
Key: Tcpip\Parameters
Value Type: REG_DWORD—number (flags)
Valid Range: 0, 1, 2, 3

0 (disable RFC 1323 options)
1 (window scaling enabled only)
2 (timestamps enabled only)
3 (both options enabled)

Default: No value. The default behavior is as follows: do not use the Timestamp and Window Scale options when initiating TCP connections but use them if the TCP peer that is initiating communication includes them in the SYN segment.

Description: This parameter controls the use of RFC 1323 Timestamp and Window Scale TCP options. Explicit settings for timestamps and window scaling are manipulated with flag bits. Bit 0 controls window scaling, and bit 1 controls timestamps.

Modbus Report-By-Exception over Cellular IP

2007-02-23T11:11:00.000-08:00

Summary: While traditionally serial Modbus has been considered unable to use Report-by-Exception, when combined with IP networks Modbus Report-by-Exception becomes very natural and effective.

Modbus/TCP is inherently peer-to-peer
People using Modbus/TCP over Ethernet or IP are familiar with its ability to function as peer-to-peer. Most PLC with Ethernet ports can function concurrently as a Modbus/TCP slave and Modbus/TCP master. So 2 PLC can very easily connect - with 2 separate Master-Slave TCP connections - and share information. One TCP connection is a Master/Slave connection with the first PLC as Master and second PLC as Slave. The other TCP connection is a Master/Slave connection with the first PLC as Slave and second PLC as Master.

Technically, this is not Report-By-Exception in the true sense of a protocol. However, since the PLC-as-Master communication events can be triggered by field inputs, it has the same effect as writing information only upon exception or when change is relevant.

Modbus/RTU as peer-to-peer
Serial Modbus/RTU is a bit harder to use this peer-to-peer trick with. A device with 2 serial ports can of course have 1 port configured as Master to issue remote reads and writes, while the 2nd port is configured as Slave to answer requests. When connected to a 2 serial port Modbus IP to Serial Bridge (such as the Digi One IAP), the 2-port serial RTU becomes a full Modbus/TCP peer, capable of operating fully peer-to-peer with other PLC and SCADA/OPC applications.

However, vendor's aren't blind to the marketing aspect of "more hardware". While adding a 2nd serial port to a one-port RTU may only cost a few dollars, most likely the 2-port RTU is a much more powerful unit, so the actual end-user cost may go up hundreds of dollars. The same is true of Ethernet; while adding an Ethernet port may only cost a few dollars, user's expectations of Web Pages and fancy functions means the Ethernet-enhanced device price may be $500 or more above that of a simple 1 serial port RTU

Fortunately, the Digi One IAP (as well as PortServer family) allow Modbus/RTU slaves to use Report-By-Exception on the serial port. As long as only one serial slave is on each port, the Digi uses configured knowledge to "split" the single serial port conversation into two traditional Modbus/TCP connections. So traditional remote Modbus/TCP masters can query the serial slave RTU, completely unaware that on occasion the serial slave RTU wakes up a acts as a Master to write data during Exceptions. Somewhere, a traditional remote Modbus/TCP slave will receive Modbus/TCP messages from the serial RTU slave, completely unaware that when not busy reporting exceptions, the remote "Master" is really a passive Modbus/RTU slave.

This feature is ideal in wide-area-network situations were bandwidth is limited or data traffic is billed on volume of bytes moved. For example, many SCADA systems only need to check on remote status every few hours ... for example lift pumps in a storm sewer system do absolutely nothing interesting for weeks or even months in the absence of rain. Even during a normal rain, checking on them every few hours is likely enough ... that is *IF* the remote life pumps can send Report-By-Exception messages during system problems.

For example, we have one customer piloting use of Modbus Report-By-Exception over cellular data network. Their eventual target is to poll the remote sites once per day. They use a simple, single-port Modbus/RTU slave which combines I/O with an LCD and push buttons to make a simple, self contained "RTU" or Remote-Terminal-Unit in the truest sense of the word. Use of Modbus in UDP/IP and Report-By-Exception allows this customer to plan for $12 per month per site bills. If forced to poll continuously with Modbus over TCP/IP, they would need to pay $50 or more per site per month. With hundreds or thousands of sites, that is a huge cost savings and opportunity for better ROI (return on investment).

More Information
Here is a general discussion of how to design Modbus/RTU serial slaves and masters to gracefully handle Report-By-Exception:

http://iatips.wikispaces.com/ModbusRBX

Here is a more focused discussion of the Digi One IAP and PortServer TS1, TS2, TS4, TS8, and TS16 handle serial Modbus/RTU Report-By-Exception

http://digitips.wikispaces.com/Modbus_Report_By_Exception

Cellular IP-Friendly Apps - Response Delays

2007-02-09T09:59:00.000-08:00

Back to my series of entries on creating graceful IP apps

Many newly written Ethernet-enabled applications incorrectly equate "Ethernet = Fast". They overlook that Ethernet is often just a path into other slower IP-based networks. Worse, some well meaning programmers set the response default to 250 milliseconds and limit the user configuration to a maximum of 5 seconds - I'd say so far about 20% of the applications I've had to help customers will limit Ethernet timeouts to 5 seconds or less.

But cellular networks have a high end-to-end latency - especially if the line has been idle for a few minutes. Normal slave response times will be near 2 seconds with round trip delays up between 10 to 12 seconds common each day (see my entry on real world Modbus numbers). Interestingly enough, every cellular "expert" I talk to keeps correcting me that cellular latencies are in the 50 to 100msec range and getting better every new "gen". Well, I guess my Saturn ION can do 400 miles-per-hour also ... if you drop it out of an airplane! Well, regardless of what these "experts" are smoking, my simple tests show otherwise where it really counts ... in actual real world tests run over the Internet to cellular-based IP devices.

Recommendation: IP applications should default to a 3 second response timeout. Applications must allow users to configure this timeout to be lower (perhaps to 250msec) and also higher to at least 60 seconds.

Impact: On Ethernet this should have no direct consequences since the timeout only has affect if the remote is no longer available - in which case the remote is going 'offline' anyway. The minority of users who really want a 250 millisecond timeout can set it manually, while cellular users who want a more reasonable timeout of 15 seconds can also set it also.

For cellular networks, the real problem with premature timeout is the customer has already paid for the request and very likely will also pay for the response - even if the response comes after the application gave up on the response and did a request retry. Assuming the user is polling the remote at a moderate pace to control costs, there is no harm is waiting longer for the response to maximize the value of the traffic paid for.

Another simple example is an application that sends a request, then timeouts twice and retries twice. How will the application react when it receives three responses at the same time? Remember, the first two requests probably were not lost; they still likely reached the remote device and created responses. Their responses may have been just delayed longer than expected. Since serial Modbus doesn't include enough information in a response to match it up to a request, this can cause serious misoperation of the system. Protocols including a sequence number should handle this more gracefully, but it will still be a waste of money.

We have also seen protocols which treat unexpected responses as a reason to abort and reset the communication channel, which further adds to cost. For example, we had one super headache with a big-name seller of "energy curtailment" systems. The end user insisted a 5 second timeout was the maximum they could tolerate (ie: wishful thinking - set a 5 second timeout regardless of reality). So lets just see what happens when we hit one of the rare but expected latencies over 10 seconds.

SCADA software sends out request sequence 74
5 seconds later, SCADA times out 74 and sends out 75
5 seconds later, SCADA times out 75 and sends out 76
1 second later - since TCP/IP is reliable - all three responses return
SCADA is expecting response 76, but sees 74 ... Oh, big problem ... need to reset comm subsystem
SCADA sends reset to remote RTU, expects response 1 but ... da da ... sees response 75 since they never flushed the old info and TCP/IP is reliable.
SCADA sends a 2nd reset to remote RTU, expects response 1 but sees response 76 since they never flushed the old info and TCP/IP is reliable
At this point, I hope you see that there are still 2 responses to the comms reset in the receive queue!

Anyway, whenever this reset "temper-tantrum" occurred it would take 10 to 15 minutes to get the connection back up. Of course one problem was the stupid customer unwilling to set the correct timeout, but the SCADA software was defective since it wasn't smart enough to just discard old responses with timed out sequence numbers. In the above example, life would have been fine and dandy had the SCADA system just discarded responses 74 and 75 since it expected 76.

Cellular and DNP3

2007-02-07T08:14:00.000-08:00

I was just at the Distributech show earlier this week - the show for power utilities. Lots of interest in cellular access. I know both OSI and Itron have successfully tested their software against our cellular product.

I have a DNP3 RTU up on my public cellular device, but need to confirm details of how the public can access it. I also had a discussion with the primary provider of DNP3 source code in the world and we will be looking at putting a DNP3 slave simulator up via cellular. It would be really userful if this could expose some of the statistical & diagnostic info managed by the simulator. This would help software vendors fine tune their software to handle the variable latency of cellular.

Do Users Really Want Industrial Ethernet?

2007-02-01T11:35:00.000-08:00

(For those impatent to read this to the end - I'm not saying don't use Ethernet ... I am just saying be careful you understand what your customers expect and what functionality they will assume you include *for free* when you add Ethernet)

My last post created some interesting feedback. But I want to emphasize a topic from that post more fully. For the last 15 years I've been involved in the "multi-vendor interface" business - linking multiple vendors' equipment by data comms. First I worked in RS-232 and 485, then fiber optics, then Ethernet, and now by virtually every technology that moves TCP/IP.

From time to time I am contacted by some pretty desperate customers - for example I had one customer who had piloted some Ethernet-based temperature sensors. Things worked fine in the lab with their lab computer, so they bought 50 ... only to find out they couldn't use them. It seems these sensors really were "just Ethernet" - they talked by Ethernet broadcast and direct MAC-layer packets. They didn't support TCP/IP and therefore could NOT be routed by any standard network infrastructure. The user could not talk to any of the sensors they had intended to install in panels around the plant because the "Computer Room" wasn't on the same physical Ethernet segment as the "floor". There was no way to broadcast or unicast MAC-level between the systems. This customer hoped I knew of some magic box to act as gateway between TCP/IP nodes and pure Ethernet nodes; I didn't.

So this brings me back to the concept of the true cost to implement "Ethernet". Customers who ask for Ethernet are not really asking for Ethernet hardware or an Ethernet media bus. They have the expectation that they can interface your "Ethernet Devices" with the wide variety of other equipment they have - including WiFi, routed Ethernet, fiber optics, wide-area networks, and so on. They also expect (at least in a future firmware rev) web pages for configuration, SNMP for remote management, strong encryption, and so on.

So the term "Ethernet" has taken on a life of its own - remember when 802.11 was called "Wireless Ethernet". Well, there is absolutely NOTHING Ethernet about 802.11, yet it was a useful PR move to link the two. No doubt it helped spread the acceptance of WiFi as we now call it. Interestingly enough, the current PCI verse PCI-Express adapters you buy for a PC are using the same PR trick - linking a new, unknown technology to an old established technology that merely accomplish the same function by very different means. Maybe Sony should have called Beta-Max VHS-Max instead ... but then I'm showing my age by even knowing that a consumer-oriented video standard other than VHS even existed.

But back to Ethernet. If you are a small device maker and have yet to start making Ethernet-based products, just be aware that customers who ask for "Ethernet products" aren't really asking for ... err, products with Ethernet. They are asking for products which integrate into (at a minimum) the wide family of TCP/IP based technologies out there. I am not even talking about should you support Modbus/TCP or ODVA Ethernet/IP or ProfiNet yet. I am just saying customers will expect your "Ethernet products" to be able to hold a raw TCP/IP or UDP/IP conversation with all of the other equipment they are investing in daily.

So the cost to add an Ethernet port is just a small part of your cost to "add Ethernet". That is why companies like Digi can sell Ethernet-to-Serial converters or sell "async Ethernet driver chips" like the Digi Connect ME which links to your CPU's serial UART. These devices of course cost more than $9.95 or even the cost of a few new hardware chips, but that higher cost is paying for TCP/IP, web servers, SNMP servers, strong AES encryption and all of the other things your customers expect when the buy "Ethernet products".

So to digress a bit, I suffer this "Oh, don't worry ... it's Ethernet" on a daily basis. So far I have to say at least 95% of the off-the-shelf software applications I test supporting TCP/IP don't work well with technologies other than direct Ethernet. This includes problems not only when extremely different media like satellite or cellular, but even when WiFi is used. So that is part of my mission in this blog - what you want is NOT to Ethernet-enable your products. Instead you need to "IP-enable" your products by way of an Ethernet interface.

Do you need a special Industrial Ethernet hardware?

2007-01-29T15:21:00.000-08:00

This is a bit off topic, but someone asked me about “Industrial-Grade” Ethernet designs. Well, I’m not a hardware expert but have better than average exposure to issues of grounding and surge management. Plus I’ve been involved in many detailed discussions about how things may break Ethernet devices with engineers from Rockwell and other industrial suppliers. Bottomline is standard Ethernet designs are quite robust - the main thing I see hurting careless designers is PCB via or tracks violating the 1500v isolation of the Ethernet magnetics. This means a customer actually testing high-voltage noise or surges could see arcing across the isolation barrier which causes product reboots or even failure.

The $9.95 Mindset:
Any detailed discussion about “special Ethernet for Industry” first starts with the fact that customers can buy 10/100 computer Ethernet adapters from any big-box store for $9.95. So users have this perception that the increment for Ethernet is small & cheap. While they may not expect you to sell Ethernet products for $10 more than serial products, they won’t be happy to hear that your Ethernet product is $300 more. I will tie this together below, but the bottom line is the closer you can match your Ethernet hardware to the market norm, the lower your over all costs will be.

Who Pays for Extra Software Work?
Of course, that $9.95 computer Ethernet card doesn’t include:

Microsoft’s ROI on TCP/IP and network stack work
the OPC server cost to add Ethernet drivers in place of serial drivers
tool vendors need (ie: your need) to rewrite serial-based tools to become network-based tools.

Plus don’t forget that the customers are NOT really just asking for “Ethernet” – Customer X will want web pages for configuration, Customer Y will want SNMP for remote management, and Customer Z will want strong 128-bit AES encryption suitable for general US-Government usage. So unlike the $9.95 PC card, you must hope your customers help pay for a whole lot of added, ongoing engineer work.

What is the Market Supply Sweet-Spot?
Go online and look at the cost of hard-drives – a 300GB (300,000MB) drive is in the $75 range, while an old 20MB drive (Meg, not Gig) costs about $140. We all understand this oddity – there is high demand for 300GB drives and virtually no demand for old legacy 20MB drives needed for repair. Even trying to buy a 40GB (Gig) drive today is hard. The market has what people call a “sweet spot” – a range of product features and capacity which is the cheapest and easiest to buy. Product builders trying to use components that are better than (or even worse-than) the market sweet-spot have disproportionally higher costs than builders using components in sync with the market sweet-spot.

The same thing happens for Ethernet components – for example buying magnetics rated at 1500v isolation (normal IEEE commercial spec) is very cheap while trying to source magnetics with 2500v isolation can cost an order of magnitude more. So while your company could define a number of electrical improvements for an “Industrial Ethernet” interface, you have to weigh this against the added cost and supply headaches of buying against the grain – of ignoring the gigantic “sweet-spot” for commercial-grade Ethernet components that enable creation of that $9.95 PC Ethernet adapter.

What is Your Manufacturing Sweet-Spot?
Just as the world market has a sweet-spot, so does your own in-house production; just ask your purchasing department. Adding Ethernet is NOT just the cost of adding a few new chips - the NIC, MAC/PHY, magnetics, and RJ45 connector. You may need to upgrade your whole basic hardware design away from a simple 8-bit CPU with 64KByte of memory to a 16 or 32-bit CPU with several MByte of memory. For example, Digi’s basic Device Server platform has a 32-bit CPU, 4MB flash, and 8MB RAM. Few of our products really need this much horsepower, but putting for example 8MB of RAM into all products is cheaper given purchasing logistics and reliability of supply than buying a mix of 2, 4, and 8MB chips. In fact, today we are looking at the cost tradeoff in shifting the basic design from 8MB to 16, 32, or even 64MB. Yes, 16MB (or 64MB) will cost more than 8MB, but given some products need 16MB (or 64MB) there are both tangible and intangible benefits to moving a larger volume of products up the curve to retain supply-chain advantages. This is especially true of FLASH and RAM chips which frequently suffer feast-and-famine availability cycles.

All small companies quickly learn – often the hard way – that during market shortages, it is the small volume purchases that get last delivery. During a chip famine low-volume purchasers will NOT be able to buy sufficient chips at any price to maintain their production. The higher your volume of a part, the lower is your price and perhaps more importantly the more reliable is your supply. So when you start to add Ethernet products and reduce sales of non-Ethernet products, you may find you need to upgrade the CPU design of some your non-Ethernet products to gain or retain reliability of parts supply.

How Robust is Commercial-Grade Ethernet?
So far I have been saying that trying to create special Ethernet hardware for industry may be costly and not very cost-effective. Worse, your average commercial-grade Ethernet is already very robust when compared to RS-232, RS-485 or USB serial. Ethernet uses a transformer-isolated signal with differential pairs, plus has nice, low-level, hardware-supported error detection. Given the high signal frequency, low signal voltage and isolation transformer, trying to add extra surge protection greatly complicates product ground design and weakens the signal, shortening the supported cable length below the 100m length customers have etched within their minds. So trying to boost your Ethernet spec for an industrial design gives questionable gain for the extra cost and lost profits. Plus your customers won’t likely perceive a market differentiation that they are willing to pay for if you say you have better isolation, etc.

A Note on Shielded Ethernet Cables:
Many industrial users start out assuming STP (Shielded Twisted Pair) is better than UTP (Unshielded Twisted Pair) for Ethernet. Oddly enough, STP has proven a bit like ABS brakes in private automobiles; despite lots of hoopla about saving lives when the US government forced ABS brakes into cars, insurance industry records continue to show it has had no measurable impact in real world road deaths. It seems while an expert driver can be helped immensely by ABS, your average idiot or careless driver still reacts to skidding situations in ways ABS brakes cannot fix.

The same appears true for STP cables and Ethernet. I have seem many discussions where industrial users tried STP cables and found the system only works reliably when they lay temporary UTP cables across the weld-shop floor! I suspect the main problem is traditional IT groups have used and measured STP success in terms of preventing Ethernet cable emissions affecting other equipment. This is not the same as using STP to prevent external interference from affecting the Ethernet signal. So ignoring the issues old truisms of a floating shield is worse than no shield and a shield grounded at both ends and creating a ground loop is worse than no shield, it appears that only experts and a very detailed system design results in STP Ethernet working better than UTP. My recommendation is to use optical fiber whenever you really worry about noise interfering with UTP Ethernet.

Vibration and RJ45:
Field tests of RJ45 connectors have shown them very bad in areas of high vibration. This is actually very easy to see for yourself - take any RJ45 connector with pins facing down and wiggle it up and down. What happens? That little finger-catch / lock acts as a pivot point and you are actually scrubbing the gold-flash contacts of the connector against the socket contacts. Metal-against-Metal; quess the result. Tests on industrial robot arms have shown even high-quality gold plated RJ45 connectors self-destruct in months or even weeks. If you expect vibration, better look for alternative connectors - such as any of the many (way too many) IP67 locking designs.

Industry and CAT 5, 5e, and 6:
Another insteresting twist to the commercial evolution of Ethernet is tests of bulk cable shows that CAT 5 is the likely the best for industrial use where the noise rejection properties of the twisted pair (differential signal) is desired. This is because - so I have been told - one of the tradeoffs IEEE allowed for CAT 5e and 6 is to allow less consistancy within the wires of a pair. After all, few Ethernet systems ever see serious external interference, so things which improved speed outweighed things which reduced noise rejection in abnormally high noise conditions. Several large automation companies tried to bring up the idea of a CAT 5i with IEEE which emphasised better noise rejection and special jacket plastic, however ... it appears it went no where. If the big computer, networking, and cable vendors don't see the value, it cannot happen through IEEE.

Been quiet 2 weeks - had some domain fun

2007-01-29T08:57:00.000-08:00

Sorry I have not posted in 2 weeks - I was trying to move iatip.com between hosters. This took 2 weeks longer than it should have and while the DNS info was in limbo I didn't want to risk breaking the site with partial updates. I am trying a new supplier with better web stats and email SPAM filtering at the SMTP level - I don't want to even see the 300-400 junk email a day I get from "known SPAM servers" on black-lists.

Hint: if you have a personal or private domain hosted by a 3rd party, don't have the same firm manage your domain record. It should have only taken 48-hours to change a DNS record, but my web hoster was trying to save me from myself (ie: my moving my DNS record away from an active 6-year old host account). I suppose there was also some self-interest in my hoster erroring on the side of caution.

With the various scams, general lack of IP knowledge in most customers, and the waiting periods to RESTORE incorrectly changed DNS info I don't blame my hoster for being careful, however I wish they'd have sent me email notifications that they wanted confirmation. I wasted the first 6 days doing 2 online requests to change. It was only after the 2nd request produced no result in 72 hours that I started calling them by phone to push the issue.

Cellular Costs - bytes you pay for each month

2007-01-12T12:44:00.000-08:00

Sadly, we are about to enter one of the dark-arts of cellular usage ... what are you actually billed for. Given the 50 page voice cell phone bill my family gets each month, one would NOT think the cell phone companies lacked the ability to explain - let alone document - what they charge data users for! It is not that one cannot get a verbal answer from cellular providers' engineers; one can get too many different answers.

However, there are some facts we can know.

An example: Modbus/RTU via TCP/IP, one poll per 10 minutes
Let us build up an example. Start with a customer named Joe who plans to poll 10 words of data every 10 minutes, or 4320 polls per month. Under Modbus/RTU this would be 8 bytes in the request and 25 bytes in the response. So Joe starts with a the wonderful view that he'll only be moving 143K per month and maybe one of those $3.95/month plans for half-a-meg will fit nicely.

Sorry to throw some cold water on Joe's euphoria, but Joe still must pay for the TCP and IP header overhead. After all, the cell data network is in effect "tunneling" his TCP/IP and Modbus/RTU and so treats even the TCP and IP headers a billable payload. So Joe needs to consider that 4320 round-trip polls per month results in 8640 TCP/IP data packets and potentially another 8640 TCP acknowledge packets. Perhaps half of these TCP acknowledge packets will be merged with the TCP/IP data packet returning the Modbus/RTU response ... but then again maybe they won't. So to keep it simple and budget safe, Joe should assume worst case and that all 8640 TCP acknowledgements travel alone. Assuming each IP header is 20 bytes and each TCP header is another 20 bytes (they may be 28 is you use Linux), this amounts to another 17,280 times 40 bytes or 691K bytes (0.7MB) JUST for the theoretical TCP/IP overhead. Joe is up to 834K per month now - clearly a 1MB/mo or larger plan is required.

Ok, wait a second ... now why did I say "theoretical TCP/IP overhead"? Because in reality Joe will end up moving more TCP/IP traffic than the 4320 polls strictly require. The first extra overhead will be from premature TCP retransmissions. The high variable latency of cellular means Joe will see from 2% to 10% retransmissions, and since cellular is very reliable, each transmission will result in duplicate TCP acknowledgements as well. Sticking to worst case, budget-safe assumptions Joe should budget about 10% or 100K per month for premature TCP retransmissions. So now Joe is up to 934K per month.

However, there is yet another overhead Joe should budget for - TCP Keepalive probes to detect lose or death of the TCP socket. Without this, one end of the connection could go away and the other end would never know and never recover the socket resource. Since wide-area-networking is involved, Joe also needs to assume at least one intermediate device will abort and discard the TCP context if idle more than 5 minutes. Given Joe polls every 10 minutes, he'll need at least one TCP keepalive exchange between each poll. Each TCP keepalive exchange consists of another 40 plus 40 bytes, so we are talking 4320 x 80 bytes or another 346K of billable traffic. This puts Joe up to 1.28MB of billable traffic to move 143K of Modbus traffic.

Now, why not close and reopen the socket? Yes, that is an option but each TCP close and reopen generates about 320 bytes - not including TCP retransmissions. So Joe can either pay for 346K worth of TCP Keepalive or 1.38M of TCP socket thrashing; which would be 1.28MB and 2.32MB per month respectively.

So Joe is up near 1.5MB per month just to move his 10 registers of data once per 10 minutes, and this doesn't include any time he checks the web interface of his cellular device for status (say another 200-500K per access), nor does it include any on-demand HMI data access screens which trigger other Modbus/RTU polls. These could easily create many MB of traffic per month and requires carefully, mindful behavior by Joe and his colleges to control costs. One careless person can easily drive the cellular bill up by hundreds of dollars in a month!

Summary:

Raw Modbus/RTU data = 140K per month
Basic TCP/IP headers to move and acknowledge data = 691K per month
Estimated 10% premature retransmission = 100K per month
One TCP Keepalive exchange between 10 minute polls = 346K per month
Overall, Joe should expect at least 1.5MB per month and I'd suggest he budget for 3MB or even 5MB. This puts him up into the $20/month cell plan range.

WiFi 802.11b Modbus Bridge - TCP to serial

2007-01-11T11:42:00.000-08:00

- Want to query a movable Modbus slave on a conveyor?
- Want to roll a Modbus Master HMI around on a cart or AGV?
- Want to query your outdoor oxygen tanks without running a cable through the wall?

I was a bit surprised, yet pleased, to see our embedded team selected to port the Modbus Bridge function into the full line of low-end Digi Connect products. The latest 82001220_H2.bin firmware for the small, low-cost Digi Connect Wi-SP includes Digi's basic Modbus bridge functionality which allows network Modbus masters to share serial Modbus RTU and Modbus ASCII slaves. It is similar to the function of the Digi One IA and a subset of the Digi One IAP.

Protocol and transport combinations supported:

Modbus/TCP
Modbus/TCP via UDP/IP
Modbus/RTU via serial, TCP/IP or UDP/IP
Modbus/ASCII via serial, TCP/IP or UDP/IP

So for example, a Modbus/TCP OPC server could query a portable Modbus/RTU serial slave; such as a machine cell moved around as production mixes require. Or a serial Modbus/RTU master on a repair cart could wirelessly query network-based Modbus PLC and test devices.

Here is the full datasheet for the Digi Connect Wi-SP, but feature summary:

Field selectable RS-232, RS-422, RS-485
Flexible 9-30vdc power supply (serial NOT isolated from power ground)
Standard 802.11b with WPA2/802.11i security
SSL/TLS strong security support
Digi RealPort support for comm-port redirection
Software development tools available for custom firmware development

Real Numbers (Part 2) - Modbus/RTU over cellular

2007-01-06T14:44:00.000-08:00

My previous post covered Modbus/RTU polls once per 30 seconds. Here is a second set of results for 24 hours with polling once every 60 seconds.

Of 1440 polls, only 3 failed responses with a 30 sec timeout.
Fastest poll was 451 msec round-trip (1/2 sec)
Slowest poll was 10,407 msec round-trip (10.5 sec)
Statistical average of successful polls was 1748 msec
In chart below, white dots are round-trip times and black line is the moving average over a 15-minute period.
Notice a few interesting trends; such as the very fast response around 11am and again around 3am the next morning.

(Click the image to see a large version)
(Click here to download ZIP file with Excel and OpenOffice spreadsheet of times)

Real Numbers - Modbus/RTU over cellular

2007-01-05T08:51:00.000-08:00

I'm working on a fuller set of numbers, but here is a real-world example of Modbus/RTU poll-response times over cellular.

I am running a Modbus/TCP poller (ModScan32), which polls a Digi PortServer TS4H by Ethernet, and the TS4H in turn is using Modbus/RTU-in-TCP/IP via the Digi corporate backbone to poll a Digi Connect WAN on Cingular GSM with a Twido PLC on the serial port.

Why use the TS4H here? Why not go directly from a Windows computer to cellular? Well, three reasons:
1) The Digi TCP/IP stack is much more graceful over cellular than Windows' TCP/IP stack - the Windows stack retries too hard and wastes bandwidth that some patience would save. With Windows you'll commonly see 2 to 5 percent retransmissions which - given how cellular is VERY reliable - ends up doing nothing but create duplicated TCP acknowledgments you must pay for. This is actually a weakness in the design of TCP/IP; which proponents claim is SOLVED by TCP/IP as-is. TCP allows hosts to auto-adjust timing behavior to match real-world performance. Unfortunately, the standard TCP algorithms keep timing too close to the "average" behavior which wastes CASH over cellular links with high and variable latency.
2) I am testing a slave timeout algorithm in the Modbus Bridge code for the Digi One IAP and TSx family related to "stale" responses arriving after the slave timeout. This is a common weakness in Modbus/RTU hosts which assume either a timely response or NO response.
3) The Digi Modbus Bridge keeps nice timing statistics such as min/max/average round-trip delay and most Windows tools do NOT.

So for example, my Master polls once per 30 seconds. This means the GSM modem in the Digi Connect WAN maintains a constant data slot allocation with the cell tower. After 370 polls, 352 have had a round-trip of 2500 msec or less and 18 polls have had a round-trip above 2500 msec (ie: I have seen 18 timed out requests with a "stale response" arriving AFTER the timeout period - behavior I am investigating).

The Digi PortServer TS4H telnet trace includes this info:
01:38:15 IA INFO: mbrtu:s02 complete rsp min:467 avg:1565 max:9142 msec

This means the fastest poll took only 467 msec, the slowest took 9142 msec (nearly 10 seconds!) and the moving average round-trip time is about 1.5 seconds. So the minimum round-trip was only one-third of the average, while the maximum round-trip was nearly six-times the average. Since every poll is exactly the same, one would NEVER see such variance on a direct RS-232 or RS-485 serial line. I'd be surprised if a PLC would have even a +/- 10% variance in response times. This is one of the problems with using off-the-shelf tools with cellular - the vendors have just NOT designed the tools for such variation in response performance.

As a side note, the polls being less than 40-50 seconds is of interest here because the cell tower (with 2.5G GSM/CDMA) will take away the data slots from the modem if it has been idle too long - the time varies but is often in the 40-50 seconds. When this happens, the modem is still connected to the tower but requires some control traffic to be reassigned bandwidth to move data. So using a poll rate slower than this idle time would shift the average round-trip time up. Once cellular system make the move to 3G this "idle period" will drop to a few seconds only, meaning telemetry systems may perform much WORSE in the new faster networks.

Rockwell Bridging - Ethernet to DF1

2007-01-03T10:16:00.000-08:00

Question: We have a MicroLogix 1500 with only 1 serial port. What Digi product can we use to enable Ethernet access from RSLinx or a HMI display?

The Digi One IAP allows bridging AB protocols:
- Ethernet/IP Master (such as ControlLogix) can query DF1 PLC
- CSPv4 Master (such as RSLinx, PLC5E or SLC5/05) can query DF1 PLC
- DF1 encapsulated in TCP/Ip (such as OPC server) can query serial DF1 PLC
- Modbus Master (TCP, RTU, or ASCII) can query DF1 PLC as-if a Modbus slave.

All of these can function concurrently, as the serial port is moving pure DF1.

The Digi One IAP cannot support DH485 because (like ProfiBus) the token rotation is too fast to be encapsulated over Ethernet successfully.

The general Rockwell Bridging is discussed in this PDF file:
http://ftp1.digi.com/support/documentation/90000636_a_doiap_ra_bridge.pdf

Quick comparison of Digi One IAP to the 1761-NET-ENI:

Product	Digi One IAP	1761-NET-ENI
Ethernet/IP to DF1 FullDuplex	YES	YES
CSPv4 (PLC5E protocol) to DF1 FullDuplex	YES	no
Modbus to DF1 FullDuplex	YES	no
DF1 encapsulated in TCP/IP or UDP/IP	YES	no
DF1 by port redirection	YES	YES
Supports DF1 Radio Modem	YES (next release "H")	no
Maximum active Masters/Peers	64	4
Configuration by Web	YES	no
Configuration by Telnet or SSH	YES	no

Basically, the Digi One IAP does everything the 1761-NET-ENI does (plus much more) except:

The Digi One IAP does NOT handle CIP encapsulated within DF1, which is required only for RSLinx to CompactLogix RS-232 port
The Digi One IAP does NOT have emails triggered by PLC MSG blocks
RSLinx won't talk Ethernet/IP through the Digi One IAP - it will talk "Ethernet Driver" fine. This is *NOT* related to existance of an EDS file. RSLinx talks via the 1761-NET-ENI because it is hard-coded to treat the ENI special. There is no EDS file information which RSLinx examines to function with the ENI.

Cellular IP-Friendly Apps - Never Block on TCP Sockets

2006-12-28T08:13:00.000-08:00

Traditionally programmers have assumed that a TCP packet will either make it to the remote peer error-free or the TCP socket will be detected as failed. However, this has proven a disastrous assumption in the world of cellular networks.

Cellular networks seem to suffer a kind of burst-error mode where whole groups of TCP packets get lost or delayed, while another group makes it through. This seems to confuse the TCP state machines within OS which are optimized for the more rare, single-packet loss of Ethernet. We have Ethereal traces where one can see the application send a TCP packet, the OS retries once, a collect of old stale TCP acknowledgements return from the remote - then nothing. Eleven hours later there have been no more TCP retries, no TCP keepalive, no response from the remote, and no TCP stack error from the OS to abort the application block. The host application is still hung, blocking waiting for either a response or socket failure which never come.

So is this a bug in the OS? Does it matter? It is your application and "our" customer who pays the price. For example, Digi had to go through our RealPort driver and literally add an OS timer to abort every TCP socket call if it did not return in 60 seconds. Yes, this sounds like a royal pain but it was the only way to avoid this failure every few weeks when running across cellular IP.

Recommendation: applications must NEVER block on a socket waiting for a response or socket failure. Applications must always use an OS or external timer to abort socket functions that take longer than 1 minute. Sadly, running in non-blocking mode is NOT enough since at times it will be the API call which fails to return regardless of the block/non-block setting. So even using API calls with explicit timeouts is not safe.

IP-Encapsulation of Modbus/RTU

2006-12-20T09:49:00.000-08:00

Summary: Modbus/RTU can easily be encapsulated within TCP/IP ... as long as there exists some mechanism to keep full Modbus/RTU messages packed within a single TCP segment or UDP packet.

Most industrial users have learned to be wary of expecting Modbus/RTU to work over error-correcting modems (especially radio) unless you use special modems which are Modbus/RTU aware. So it is wise to be wary of moving Modbus/RTU over IP without some special settings or features in the IP devices involved. Fortunately all Digi devices (and most competitors' devices) have such features or settings.

Bullet-Proof Solution: Modbus Bridges
The safest and most flexible way to move Modbus over IP is to use devices which fully understand the Modbus protocol and dialects of Modbus/TCP, Modbus/RTU, and Modbus/ASCII. This allows multiple Masters to share the slave(s), plus Modbus/TCP masters can query Modbus/RTU slaves and the bridge handles the protocol conversions.
More detailed information of this topic is this application note
Setting up Digi One IAP for Modbus Bridging. The basic information in this application note applies to the following Digi products with Modbus Bridge ability:

Digi One IA, Digi One IAP
PortServer TS1 to 4 MEI including TS H MEI(adds Ext Temp -35 to +74DegC), TS Hcc MEI(adds Conformal Coating), and coming soon TS Haz MEI which adds Div 1 Cls II certs
PortServer TS8 and TS16
Digi Connect WAN IA)
(Coming in next firmware release: Digi Connect SP, Wi-SP, ME, Wi-ME, EM, Wi-EM)

Effective Solution: TCP (or UDP) Sockets Profile
If you don't really require the multi-master or protocol bridging features, then any Digi device server can be used. By default the Digi serial "TCP Socket Profile" will break all messages into TCP segments of from 4 to 64 bytes - not what you want for Modbus/RTU. This default behavior creates the lowest latency for normal data without the timing fussiness of Modbus/RTU. However, all you need to do is enable the option checkbox feature "Send data only under any of the following conditions" and then the sub-option "Send after the following number of idle milleseconds". Entering a time such as 25 milliseconds causes the Digi device server to continue collecting data and delays creation of the TCP segment packet until no more serial data is seen for 25 milliseconds. This is a very nice fit to the Modbus/RTU "3.5 character idle time" end-of-message condition. Why not use 5 msec? Well, experience has shown me the 25 (or even 100 msec for cellular) is a more robust value.

So an example solution using TCP Socket Profile would be to use an OPC server such as Kepware which can put most of its serial protocols into a TCP/IP socket. These should naturally put a full Modbus/RTU request into a single TCP segment - the host application is "defective" if it causes more than one TCP segement to be used; it means the host application vendor doesn't know what they are doing. Since the Digi device server receives the entire request as a single TCP segment, the full Modbus/RTU request will move out of the serial port as a single continuous stream of bytes. With the correct settings, when the Modbus/RTU response returns the Digi device server

Mixing Modbus and Rockwell on Ethernet

2006-12-15T11:43:00.001-08:00

Both Modbus and PCCC-based protocols like DF1 or CSPv4 (AB/Ethernet) have been around for years. Yet if one looks at the similarities between the two, one quickly sees that the act of reading 10 words from an N7 data file is exactly the same as reading 10 words from Modbus 4x00001. The Digi One IAP leveraged this to become the world's first off-the-shelf transparent protocol bridge. It freely accepts Modbus or Rockwell requests and bridges them to the appropriate form for the slave to understand.

Here is an example system:

The ControlLogix can poll the Modbus/TCP and DF1 PLC
The Modbus/TCP PLC can poll the ControlLogix and DF1 PLC
The DF1 PLC can poll the ControlLogix and Modbus/TCP PLC.

So how does this work? Take a look at the messages to read the first 48 bits of bit memory:

Modbus/TCP is 001E00000006010100000030
Modbus/RTU is 0101000000303C1E
Modbus/ASCII is :010100000030CE(CR)(NL)
DF1 Full-Duplex is 100201000F000019A206038500001003DE06
CSPv4 is 0107000E00 … 010500000F000019A2060385000
PCCC-Ethernet/IP is 6F002800 … 0000010000000F000019A2060385000

Notice the bold, underlined text patterns? This is the heart of how a normal Modbus Bridge or 1761NetENI function. Modbus/TCP, Modbus/RTU, and Modbus/ASCII may have different bytes, but they all move the exact same Modbus command; a Modbus bridge doesn't need to understand the Modbus command, just be able to unpack and repack each form. Similarly DF1, CSPv4, and PCCC-in-Ethernet/IP have different bytes, but they all move the same PCCC command; a PCCC bridge doesn't need to understand the PCCC command, just be able to unpack and repack each form.

The Digi One IAP takes this one step further - since each of these bold, underlines commands is accomplishing the same thing - namely reading the first 48-bits of bit memory - the Digi One IAP can take either command and mechanically create the other. So given the Modbus command 010100000030, it can create the PCCC command 0F000019A20603850000. Given the core PCCC command 0F000019A20603850000 it can create the Modbus command 010100000030. So this how a Modbus/TCP master can query a ControlLogix with PCCC-enabled. the Modbus/TCP master thinks it is polling another Modbus device. The ControlLogix thinks it is being polled by another Ethernet/IP device.

Here are links to other related information:

Digi One IAP product page
Application Note for Modbus master polling Rockwell devices.
Excel spreadsheet for Modbus master polling Rockwell devices.
Application Note for Rockwell master polling Modbus devices.
Excel spreadsheet for Rockwell master polling Modbus devices.
PDF presentation of various ways to mix Modbus and Rockwell devices

Rockwell AB PLC via Cellular

2006-12-15T08:50:00.000-08:00

So far we have succeeded in getting several Rockwell/Allen-Bradley PLC up on Cellular with the Digi Connect WAN, which is a cellular router for GSM or CDMA with local Ethernet and serial port.

In Summary:

Serial DF1: You can access serial MicroLogix PLC such the MicroLogix 1200 on the remote Digi Connect WAN's serial port. You either need to have an OPC server which can directly encapsulate DF1 protocols into TCP/IP or to use Digi RealPort to create redirected COM ports for RSLinx. Ideally, using the newer DF1 Radio Modem protocol can cut your data costs in half, but DF1 Full-Duplex or Half-Duplex can also be used. DH485 won't work via cellular due to the high latency. You must slow the PLC (ACK) timeout setting down to 30 seconds, so you cannot use a MicroLogix 1000 since it doesn't allow this parameter to be adjusted. DF1 Radio Modem has no DF1 (ACK) or (NAK), which is why it costs less to use.
CSPv4 or AB/Ethernet: You can access legacy PLC such as SLC5/05 and PLC5E by enabling TCP port forwarding of port 2222 on the Digi Connect WAN. Under RSLinx you enter the IP or DNS name for your Digi Connect WAN in the "Ethernet Driver", then right click the driver to slow down the timeouts from default of 3 seconds to a cellular-friendly 30-seconds. For a bit of fun, open this link in your browser and you will access the web pages of my SLC5/05 through Cingular/GSM cellular - http://digiwan.gotdns.org:8080/. But please don't leave this page open since you'll impact other people trying to look at my cellular PLC.
Ethernet/IP: You can access ControlLogix and other newer PLC supporting Ethernet/IP by enabling TCP port forwarding of port 44818 on the Digi Connect WAN. Under RSLinx you enter the IP or DNS name for your Digi Connect WAN in the "Remote Devices via Linx Gateway" Driver, then right click the driver to slow down the timeouts from default of 3 seconds to a cellular-friendly 30-seconds. You cannot use the RSLinx Ethernet/IP driver since it relies on UDP broadcast which cannot move across wide-area-networks.

If you want more detailed instructions, I have an application note online here:

90000772_A_Cell_AB.pdf

Simulating Multi-drop Across routed IP

2006-12-11T10:50:00.000-08:00

Summary: the UDP Sockets profile in Digi device servers can be used to simulate multi-drop behavior in routed IP or wide-area networks. An application note is linked to this entry.

Ever wished your Ethernet could mimic an RS-485 network? Or are you trying to replace an old, expensive multi-point analog modem system with newer IP-based technologies such as cellular, satellite, or aDSL links?

On a local Ethernet subnet a UDP broadcast can be used to simulate multi-drop ... however IT departments and anyone thinking of the future knows IP broadcast is something not to be used lightly. IP broadcast loads every device on the network and examples of high broadcast load killing or crippling important embedded devices are common.

The preferred method on a local Ethernet is the use of Class D IP (aka IP addresses in the 224.x.x.x to 239.x.x.x range). However, details of IP assignment, IP collision, and the risk of turning switches into hubs make this a risky and confusing technology. Most heavy users of ODVA Ethernet/IP can cite a few cases where enabling high multi-cast traffic killed other third-party products (notably security or video devices) which had treated all multicast traffic as broadcast to be examined by software.

Plus we are talking about wide-area-networks and use of cellular or satellite technology. Routed IP networks won't move broadcast or multicast traffic unless active proxies exist at each end to encapsulate the broadcast/multicast traffic into TCP/IP.

Fortunately, the Digi One IAP (and most Digi device servers) include the ability to use a form of repeated UDP/IP unicast to simulate multicast to up to 64 remote peers. I have customers using this to move Modbus/RTU and AB DF1 Half-Duplex through routed private wide-area-network. Here is an application note which explains how to set this up.

(For now it is a Word 2003 document - but it can be opened by Open Office Writer v2.0 if you don't have Word. I'll shortly turn it into a Acrobat PDF)

http://iatips.com/blogimage/90000xxx_A_UDP_Multidrop.doc

Rockwell Protocol Documents

2006-12-07T09:00:00.000-08:00

A friend just pointed out this public web page to me: How to Communicate with Rockwell Automation Products. While have seen many of this documents before, a few of them were new for me. It includes information on:

How to talk to ControlLogix tag data via Ethernet/IP
How to understand ControlLogix data structure packing when read raw
The DF1 serial protocol specification
How to encapsulate CIP over DF1 (ie: talk to serial port of Compact/ControlLogix)
How to use Ethernet/IP explicit messaging to ControlLogix
How to use Ethernet/IP I/O messaging with ControlLogix

In addition, I see www.ab.com has added a new DF1 supplement to its Knowledge Base. Since you have to login giving you a direct link is pointless, but it is called "DF1 Protocol supplement 17706516". It compares PLC5 and SLC5 communications, covers some useful commands such as 0x0AB "Protected Typed Logical Write with Mask" to write individual bits, and new data file types not covered in the latest 1996 version of the DF1 specification.

While we are discussing new Rockwell protocol information, you should also review and be aware of the new "DF1 Radio Modem" protocol. I don't think there is a form specification, but you can find a file in the http://www.ab.com/ Knowledge Base that describes the simple differences between it and DF1 Full-Duplex. In summary, DF1 Radio Modem *IS* DF1 Full-Duplex without the protocol ACK/NAK. It is designed for use in radio systems where the powering up of slave modems just to ACK something they will respond to anyway just slows down overall polling. I'm also finding it ideal for cellular IP networks since it literally cuts your data usage by 50 to 60% to NOT be moving 2-byte DF1 ACKs within TCP/IP which also includes a TCP acknowledgements. SInce DF1 includes a TNS or transaction number, there is no problem with mishandling lost or delayed messages.

The main catch today is that I think neither RSLinx nor ControlLogix support DF1 Radio Modem - it is mainly a MicroLogix and SLC5 family resource. However the next release of the Digi One IAP will include the ability to bridge to DF1 Radio Modem from Ethernet/IP, CSPv4 (AB/Ethernet) and DF1 Full-Duplex.

Cellular IP-Friendly Apps - It Costs to Talk

2006-12-06T06:59:00.000-08:00

Summary: All communication must be "under control". All data sent into the cellular system costs money; even if the remote cellular device is powered off, the customer still pays for data set to it.

As a follow-on to the discussion of Retrying TCP Socket Opens, applications must allow the user to both understand and limit all aspects of protocol usage and retry. Users must be allowed to limit and predict a reasonable worst case traffic cost. For example, some protocols include large blocks of initial connection negotiation, which means talking once per minute over an continuously open socket could result in much less cost than talking once per 10 minutes over a socket opened just for one transaction. I have seen applications that allow users to set a maximum desired retry setting - then not always follow that setting and do retries anyway in certain fault conditions.

Recommendation: application-writers must step back and examine every place within the application they create traffic and confirm users have the ability to limit the traffic created.

Example and Numbers: now most of you will be saying "Yah, dahh - so obvious why is this even mentioned?". Well, I'll give you an all too typical example of how this affects real customers. A customer (call him Joe) running a pilot on cellular data access calls to complain his costs are higher than expected. He says he's just polling 3 Modbus registers every 5 minutes. Being no dummy, Joe has already calculated that each request should be 12 bytes of data (One Modbus/TCP function 3 read) and each response should be 17 bytes of data (One Modbus/TCP response with 4 registers since he is reading 4x00003, 4x00004 and 4x00006 so one assumes 4x00005 comes along for the ride). One poll each 5 minutes works out to be 8640 poll per 30 days, so he had hoped to see only about a quarter-megabyte of traffic a month. Yet Joe was seeing data bill for 6 to 10 MB of traffic a month. This means his $20 per month 5MB plan was costing him closer to $60 per month with data overages.

First, Joe overlooked the fact that he has to pay for not only his Modbus data, but also the TCP and IP overhead used to move it. Standard Windows-generated TCP headers are 20 bytes and so are the IP headers. Linux tends to defaults to use TCP time-stamps and thus creates 28-byte TCP headers. So each request is NOT 12 bytes, but 52-60 bytes ... plus the TCP Acknowledge frame will add an additional 40-48 bytes. Yes, YOU pay for the TCP Acknowledgements as well! With headers and TCP Acknowledgement, his responses will be 97-113 bytes not 17. So right off the bat, I can see that he has been under-estimating his monthly traffic. Since he is using Windows, he should be seeing at least 1.6MB of traffic and never 0.25MB.

So I vist Joe and do a network trace of his OPC server traffic. We see that OPC is issuing 3 Modbus polls every 5 minutes - not 1. Hmmm, of course Joe's first reaction is "Heck no - I'm not polling 3 - just 1" but the proof is there as colored pixels on my notebook display. We decode the polls and see the OPC server is polling 3 blocks of 32-registers each. After decoding the Modbus/TCP bytes we learn the exact registers being polled and Joe eventually discovers why these are being polled:

One block of 32 registers is fetching his 3 desired value of 4x00003, 4x00004 and 4x00006. Reading the fine print in the OPC manual we see that the OPC server decided this was a "scattered poll" of 2 separate memory areas so it bumped the size up to 32 registers. So just for this one poll, his monthly budget is up to 2.8MB instead of 0.25MB
A second block was caused by Joe programming an HMI display to pop-up if a certain alarm condition where true in the field. This was a demo he'd done to impress a customer, but Joe hadn't thought to disable it nor had realized the exact "cost" of such a feature. So the OPC server needs a single register off somewhere else in the PLC memory to satisfy the HMI's alarm/event function. We don't know why this is polled as 32 registers instead of 1 - it is not a "scattered poll" as defined by the OPC vendor's documentation. Perhaps his HMI or OPC server software has a bug in it. Since this is Modbus/TCP (not serial) it is unlikely anyone else has noticed or cared that the application is moving 62 bytes of extra data in every poll. After all, Ethernet is fast and costs nothing to use. It is possible the programmers at the OPC vendor just decided there was no reason to ever poll less than 32 registers when using fast, free "Ethernet".
The final block was being caused by Joe's boss leaving open an HMI display in another room that wasn't supposed to be left open - human error (or is it?). Joe learned instanty how important it was for him to properly configure the HMI display settings which timed out displays - either closing the window or just stopping the supporting data polls. He had done that for the normal "user display", but had been lazy and not put such settings into the various diagnostic displays users weren't expected to use!

So now his 6 to 10MB of traffic a month begins to make sense. Each distinct poll is creating nearly 3MB of traffic per month, and his traffic is influenced by which HMI displays users open. Multiple 3MB by 3 polls and you roughly 9MB per month.

What has been learned here?

With the overhead of TCP/IP, Joe learned that he had to pay for over 4 times more traffic than his raw Modbus byte calculates had led him to believe.
Joe learned that he should be looking at using UDP/IP instead of TCP/IP for his Modbus/TCP since this would cut 40-60% off his bill instantly. Modbus doesn't really require the TCP Acknowledgement and my own tests of UDP/IP over cellular shows it to be about 99.99% reliable - or put another way, I only see about 1 packet lost per 10,000 sent.
Joe learned how to review his OPC server's data statistics page. His OPC server had been (indirectly) giving him the answer as to why his data usage was so high. While his OPC server never totaled up the data bytes to include TCP/IP overhead, it was able to show him the 36 polls per hour he was moving instead of his expected 12 (one per 5 minutes).
Joe learned that perhaps he needs to look for a new OPC supplier, since his present vendor just doesn't seem to see the big picture of IP-enabled protocols; that Ethernet is not the only media using TCP/IP. Increasingly people expect TCP/IP to move through diverse media which is not always "fast and free" like Ethernet. Joe's present OPC supplier didn't give him the ability to reduce the poll block size below 32 registers when the OPC system thought "Ethernet" was being used.
Joe learned he had to be more aggressive in his HMI display design. He couldn't assume users would only look at certain displays and not leave open displays unexpectedly. Joe needed to actively set every possible display to automatically close or stop generating new polls. In fact, after review he discovered that most of his displays had no need for "real-time" update and he could just set them to display the data once as read without any refresh. Users always had to the option to manually redisplay the page.
Joe learned that maybe just reading data from the RTU program directly was not such a wise idea. His RTU had the ability to copy and repack data into special polling areas to eliminate "scattered polls". In fact, in the above example we traced at Joe's site, all of the data in those 3 polls could have easily fit within a single 13 register block. So Joe is reviewing his RTU program design to repack ALL data of interest - even data supporting rare HMI displays - into a dedicated memory area. While Joe had previously hoped to avoid this work, he now sees the potential dollar saving or cost penalty his company could face if he avoided this work.

So really in summary I have to say Your data polling needs to be UNDER CONTROL, as in being controlled. You need both the tools and the investment in effort to define as exactly as possible each and every data poll.

TCP/IP Encapsulation Limited by Distance?

2006-11-17T14:29:00.000-08:00

Summary: serial tunneling or TCP encapsulation is NOT directly affected by distance. However, it is affected by "hops" or how many routers and segments it goes through. To reduce the effect "hops" have on your TCP/IP Encapsulation, you need to set the correct options in your Digi Device Server. These are NOT standard defaults since what works best for Wide-Area-Network is not best for local direct Ethernet links.

What is serial TCP/IP encapsulation? It is also called serial bridging or serial tunneling. Think of it as the modern IP equivalent to old short-haul modems or leased line modems. At each end you have a "modem", you connect a serial cable to each, and you create a virtual serial link running from end to end.

During a webinar I gave, a traffic-industry user asked if serial TCP encapsulation is limited by distance. If he serial bridges between 2 intersections of a road, things work fine. However, if he tries the same serial bridge between an intersection and the home office, then serial bridging does NOT work. So he was wondering "what is the distance limit for serial bridging or encapsulation?"

The simple answer is "There is no distance limitation". However, the longer the distance you move your serial encapsulation, the more routers and network segments (hops) your traffic moves through. The more hops your traffic moves through, the more variable latency (or delay or jitter) is introduced between consecutive network packets. This affects the timing and different protocols and software implementations react differently to it.

Let me just throw some hypothetical numbers together. Let's say the device sends data as a block of 100 bytes; the receiving device will loop and collect this data. Of course the receiving device cannot just wait for 100 bytes - what if the line breaks? It could sit there forever waiting for the end of the message. So various timers are coded to enable the receiving device to understand failure and abort receiving. Let's say the receiving device waits at most 20 milliseconds for the next byte. On a direct serial link this is very common - once the sending device starts sending data it is very unlikely that even a 2 or 3 millisecond gap will appear between bytes.

Enter serial encapsulation - either by radio or Ethernet or any IP-based media. All of these technologies are packet-based and most include some form of error-retry. So now the serial bytes collect in a buffer up to some point, then a chunk of them move together as one packet. Ideally, the full message moves as a single packet. However, if the message becomes split between 2 or more packets it is possible a gap will be detected by the receiving device. So for illustration we'll say the 100 bytes is split into 4 x 25-byte packets. On a single-hop network, there is much less opportunity for timing delays to open gaps in the final serial data. This diagram shows a small gap between the 25th and 26th bytes:

But running the serial encapsulation through many network hops greatly increases the accumulated delays added to each packet. So each hop has the opportunity to increase the latency and lag. This diagram shows a much larger gaps that may occur when the packets create the serial traffic at the remote end:

How to solve this problem? On the Digi Connect products, you'll be using one of these serial port profiles: TCP Sockets or Serial Bridge. Under the Advanced Serial Settings you need to enable the check box labeled: [ ] Send data only under any of the following conditions. If you do not check this option, the Digi Device Server purposely fragments the serial data into many TCP segments to provide more realistic end-to-end performance on direct Ethernet links. However, since you want to move data through a wide-area network, you are less concerned with raw throughput than in preventing message fragmentation. You may also be interested in creating fewer TCP/IP packets to send more serial data. Changing this setting accomplishes both of these things.

You now have 2 options to define when TCP packets are moved:

"Send when data is present on the serial line" allows you to define an end-of-message pattern such as a carriage-return (\r or \r\n, etc).
"Send after the following number of idle milliseconds" allows you to define an idle or quiet time to wait before sending data. This second option is generally safest and I find a value of 10 or 25 milliseconds to be ideal with most automated devices.

Note: do NOT change the setting in the "Send after the following number of bytes" field! This is rarely useful and it does NOT mean (when unchecked) that the Digi Device Server must see 1024 bytes before it sends anything. I have had too many users change this to 1 and then wonder why they have a huge amount of network traffic!

Application Pitfalls: Serial DF1 over WAN

2006-11-10T08:31:00.000-08:00

Digi's wireless group was asking me why AB/DF1 didn't always work over radio when per the specification, DF1 has a nice end-of-message pattern. One would think moving serial DF1 through radio or cellular-IP would be natural and painless.

However, the problem I see watching Windows applications use the serial API (via the PortMon utility) is that they ASSUME a small delay or gap between the (DLE)(ACK) bytes and the response from the slave.

So the application uses the incorrect algorithm:

Read 2 Bytes
Ask Windows to notify application when more data comes
Loop, reading buffered data and waiting until full response seen

The problem with this algorithm is it assumes the response will NOT have been received by the time the application sees the 2 byte (DLE)(ACK). Because we're dealing with radio or IP system that make effort to packetize data there is a high probability that the (DLE)(ACK) and the slave response arrived at the same time within the same packet without any noticable gap. Therefore, Windows NEVER notifies the application of that more data has arrived ... because no more ever comes! The full response has already been received and buffered. The application makes the false assumption that measurable time will occur between step #1 and the start of loop #3.

The correct algorithm would be:

Read 2 Bytes
Loop, reading buffered data and waiting until full response seen

This works because it handles both the situation of no response, a response already received and fully buffered, and a response trickling in over time.

See Also:

Download the Rockwell Allen-Bradley DF1 specification: this used to be open to download as a PDF ... it is still free but now you need to log into http://www.ab.com/ and find "DF1 Protocol and Command Set Reference Manual - Publication 1770-6.5.16"
Download PortMon http://www.microsoft.com/technet/sysinternals/utilities/PortMon.mspx

City-wide WiFi - it's not Ethernet

2006-11-09T06:15:00.000-08:00

One of my customers is struggling to IP-enable a few dozen Ethernet PLC via one of these new fangled city-wide WiFi systems that are all the rage now. Looked good on paper, but they can only keep the PLCs online for about 20 minutes at a time.

Why? Is the WiFi system defective? Of course not ... it is just behaving more like Wide-Area-Network than Local-Area-Network. I am not directly involved in this, but I'd wager the problem is neither the WiFi nor the PLC. The problem is the host software making Ethenet LAN assumptions about the system. I should offer to go out and do some latency tests; I'd wager the system has high and variable latency more like satellite or cellular.

So industrial control application developers beware, migrating your Ethernet-enabled, LAN-friendly applications to be true IP-enabled, WAN-friendly applications will become more important every time another city annouces the installation of a city-wide wireless infrastructure.

Cellular-IP Friendly Apps - Retrying Socket Opens

2006-11-06T14:54:00.000-08:00

Most industrial applications allow the user to set a slow poll rate – such as one poll per 5 minutes. This allows a user to budget a cell plan at 5MB per month and be quite assured of not going over. Unfortunately, this steady-state poll rate is unrelated to initial TCP/IP socket connection opens!

If the remote device is powered down or the TCP socket open fails for any reason, most applications will attempt to reopen the TCP socket continuously. On Ethernet this may make sense; the more frequently the open is retried, the sooner the failed connection will recover. Most Ethernet-based applications will retry opening a TCP socket every 5 to 30 seconds forever. However, for cellular you are paying for all traffic entering the cellular system. It is not Cingular or Sprint or Verizon's fault your remote device is off-line. You will be billed for each and every TCP retry. I have literally seen applications create up to 1000 MB of traffic each day attempting to reopen a TCP socket to an unreachable remote IP. On a 5 MB per month plan, this 1GB of overage could easily cost you $1000 or more for the month!

Recommendation: all applications must include a user-settable option to delay attempts to reopen TCP sockets. This value can default to no-delay, but users must be able to set a delay of at least 1 hour between retries. This enables the user to define and stay within their data usage budget regardless of success or failure of the TCP connection.

Impact: On Ethernet this should have no direct consequences since the recommended default is no delay. Cellular users must adjust this retry delay to match their data traffic expectations and their cell plan budget.

For example, an application polling 10 Modbus registers per 5 minutes via TCP/IP creates about 198 bytes per poll. This works out to 2376 bytes per hour or a little under 2 MB per month. This is a very safe poll rate when paying for a 5 MB per month plan.

Therefore the desired TCP reconnect scenario should also create no more than 2400 bytes per hour. Consider that a 20-second timeout under Windows creates at least 120 bytes of traffic to an off-line remote. Windows sends a 40-byte [SYN] packet and retries the same 40-byte [SYN] in roughly 3 and then 8 seconds from the previous [SYN] packet. Increasing the timeout to 30 or more seconds creates a fourth 40-byte [SYN] packet sent about 18 seconds after the third. So forcing an application to only attempt one connection per 5 minutes will create from 1440 to 1920 bytes of traffic per hour. This will not break our budgeted cell plan.

Modbus Bid Spec Suggestions

2006-10-31T07:01:00.000-08:00

A large customer asked me for advice on bid-specing the use of Modbus/TCP. They are expecting HVAC and other non-production systems to provide "gateways" with Modbus/TCP to simplify central HMI and data collection. Experienced field people know this is not quite as easy as it sounds.

So I stepped back and put myself in their shoes - if I were trying to design a new assembly plant and I hoped to monitor HVAC, chemical tank farm, and such by Modbus/TCP, then what issues would hinder this? What details NOT included in the http://www.modbus-ida.org/ protocol specification would complicate true interoperability? I have many experiences of integration problems with Modbus masters and slaves from 2 different vendors not quite talking as expected. Usually the customer ends up PAYING one vendor or the other to change; or the customer has to buy a 3rd party box to fix the difference of opinion.

So how to avoid this up front? Here is the list I created:

The required interface is Modbus/TCP running on standard Ethernet II frames and following the published specification at http://www.modbus-ida.org/.

All devices must support at least 100M half-duplex Ethernet and if auto-negotiate is supported, they must be manually configurable to force 100M Half-Duplex.

If the supplied product uses serial Modbus/RTU or Modbus/ASCII, then vendor must supply a configured, tested, and powered Modbus Ethernet-to-Serial bridge (such as the Digi One IA, model 70001862) to bridge this to Modbus/TCP on Ethernet.

All data must be available and/or mirrored within the Modbus 4x or "Holding Register" memory area. The other areas can be optionally supported, but all 0x, 1x, and 3x data must be readable in the 4x memory area. For digital writes, support of single-bit writes (function 5) to the 0x area are acceptable. Products that require access to the 1x and 3x area to operate are not acceptable; access to 1x/3x area must be optional.

Modbus 32-bit longs and floating points must be available in Modicon 984 Compatibility format, which means as two consecutive 16-bit big-endian registers, with the low word in the first register. Other forms (Daniels/Enron or high-word first) can exist but must be optional.

All gateways or converters bridging non-Modbus data to Modbus must not provide stale data and must not require special "status registers" be monitored to confirm data validity. If the source device of the non-Modbus data is unavailable or the data is out-of-date, then Modbus/TCP requests must return an exception such as 0x0B until the source data is valid again.

Software tools must function properly with slaves only supporting Modbus functions 3 and 16. Requiring diagnostic function 8 support is not acceptable.

Software tools must be configurable to write a single register as either function 6 or 16.

Software tools must be configurable to limit reads and write to user selectable limits; for example, the software must accept being limited to reading 1 register per transaction and writing 1 register per transaction.

Software tools must allow setting to the Modbus/TCP "Unit Id" to a value other than zero. This is required for Ethernet-to-Serial bridging.

Software tools must use the Modbus/TCP sequence number and modify it between polls. The tool must not leave it set as 0 or 1 all the time.

To support future wide-area-network usage, all "Masters" must permit TCP socket opens to take up to 30 seconds.

To support future wide-area-network usage, all "Masters" must permit slave timeouts be set to at least 30 seconds.

To support future wide-area-network usage, all serial slave devices must have a configurable "gap" or intercharecter delay timeout. The Modbus spec's "3.5 character times" is problematic when dealing with radio & other error-correcting media.

All devices must be capable of transport via wireless bridging by common Ethernet radio systems such as 802.11 bridges and more traditional 900Mhz line-of-sight bridges.

Now, will all vendors be able to meet all of these requirements? Probably not since many of them are not required per the Modbus-IDA specifications. However, at least this brings the issues up front to be addressed during the bid award phase. If custom firmware modifications are required, it can be addressed up front and not during factory acceptance testing.

Cellular-IP Friendly Apps - Socket Open

2006-10-23T08:20:00.000-07:00

Most applications attempt to open a TCP socket using the OS/Windows default timeout. This results in an unpredictable timeout. I looked through Microsoft's VS.NET documentation looking for the "How long?" answer ... and never found an answer. I suspect it depends on your version and service-pack levels. I did a web search to discover the truth and found people claiming Windows timed out in 2 seconds, 5 seconds, 10 seconds, 20 seconds, 20-30 seconds, and even one claiming 5 minutes. Sadly, most of these people were looking for a way to force Windows to use a connection timeout of 1 second or less - which will prevent their applications from working on normal wide-area networks.

Such short connection attempts are not suitable for cellular network where the first response packet from an idle remote tends to complete in 3-4 seconds during average conditions. Therefore even a 5 second timeout is too close to the norm to be suitable.

Recommendation: all applications must use an explicit, predictable timeout during a TCP socket open request. This value can be user-settable higher or lower, but for cellular should default to 20 seconds and be settable to at least 60 seconds for satellite.

Impact: On Ethernet this should have limited direct consequences since the timeout only has affect if the remote is not available. If having your application wait 20 seconds for an inaccessable remote is a problem, then enable a user setting to select either "local-area-network" or "wide-area-network" mode and adjust the default connection timeout as appropriate.

In a best case scenario, failing to wait long enough to open a TCP socket when the network is sluggish could prevent connecting for many minutes as sockets succeed to open, but the OS aborts the open before the successful response can come back from the remote. Keep in mind that over cellular the end user is paying for at least 120 bytes of data for every open attempt, and that TCP retransmissions likely make this 160 or 200 bytes.

In a worst case scenario, this aborting of opened sockets on a remote with limited resources risks tying up all resources with past failed opens. Remember, just because your OS timed out the open does not mean the remote device didn't allocate the connection resource and send a successful response. The lack of resources blocks new attempts by the application to reconnect until TCP keepalive or some other mechanism detects the broken sockets and frees up the resources.

Be warned that under cellular - as if in defiance of traditional faith in the reliability of the TCP state machine - TCP sockets break in rare occasions in ways that common OS will fail to detect! During cellular network hiccups, I have seen machines "hang" for 11 hours waiting for a TCP Acknowledgement that never comes! This is with TCP Keepalive enabled for 5 minutes even.

Some Visual Studio discussion: Just out of curiosity, I did some snooping around inside the Visual Studio .NET documentation. I didn't find a good answer, so cannot explain how to solve this problem.

Here is example VB.NET code to opean a TCP socket:

Dim tcpClient As New TcpClient
Dim ipAddress As IPAddress = dns.GetHostEntry("www.digi.com").AddressList(0)
TcpClient.Connect(ipAddress, 11003)

Notice we cannot ask the OS to wait "longer" or "shorter". The documentation says "The Connect method will block until it either connects or fails." A connection timeout results in a SocketException failure being thrown. The TcpClient class has ReceiveTimeout and SendTimeout properties, but these only relate to reads and writes on the connection and have no impact on the initial connection open. Suggestions to use the System.Net.Sockets.Socket class instead aren't helpful since this class also doesn't offer a simple connection timeout mechanism.

The only wait to define a predictable TCP socket connection timeout appears to be use an asynchronous design with BeginConnect and some form of external timer to call EndConnect at the desired timeout.

To rephrase myself, I am not saying the default Windows connection timeout is incorrect - I am saying evidence is that you cannot predict what timeout your customer will see if you don't explicitly define one. So while your application running on your computer may default to a nice 20 seconds timeout, what happens if your customer runs the same application on an older computer and sees a 3 second or 5 second timeout? The answer is they won't be able to reliably connect to cellular or satellite remote IPs, and either won't buy your product again or will call Tech Support.

Cellular-IP Friendly Applications - Intro

2006-10-20T06:58:00.000-07:00

In theory, host applications using TCP/IP on Ethernet should work over wide-area networks which support TCP/IP. Unfortunately, most host applications are written and tested for Ethernet, not generic IP. When you move into cellular IP or satellite, the high and variable latency introduced causes many host applications to either fail or generate an order of magnitute more traffic than than they should.

For example, here is a chart of 1000 Modbus/RTU polls over cellular TCP/IP. There is a random delay between polls of 30 seconds to 30 minutes. The patterns are rather striking: most polls complete in between 1 to 2 seconds, but there is clearly some systematic "aliasing" causing responses to complete in 2.8, 3.8, and 10.8 seconds.

After years of troubleshooting customers systems, I have been creating a running document and commentary on Bad things host apps do. I will be publishing these things over time in this blog. But to summarize:

The default OS timeout on opening TCP Sockets may be too short.
Attempting to open TCP sockets to unresponsive remotes must be a controlled process, since retries cost money.
Since all packets and retries cost money, all aspects of the implemented protocol must be controlled and adjustable.
OS stack calls may not return if the OS fails to detect a response or socket failure.
Responses from the remote could take 15 to 60 seconds.
TCP segment fragmentation and reassembly is exaggerated; can have many seconds of delay between fragments.
TCP sockets idle longer than 5 minutes often go away without error or detection.
Every byte your application sends (or resends) costs your customer money.

Siemens PLC via Cellular

2006-10-12T07:36:00.000-07:00

We succeeded in getting a Siemens S7-226 with CP243 and PPI serial up on the Digi Connect WAN, which is a cellular router for GSM or CDMA with local Ethernet and serial port.

In Summary:

To talk to S7-226 by serial PPI, you need a newer Siemens PC-to-PPI cable - the older one doesn't work. I am not sure why, but that is what we found. Using Digi RealPort we enabled a redirected COM port to the remote Digi Connect WAN's serial port, which is connected to the PPI port of the S7-200. We then defined a radio modem port within MicroWin using that COM port. Although a 30 second timeout would be ideal, MicroWin only gives options for 1, 10 or 100 seconds of timeout. You should probably select the 100 seconds to minimize your comm costs. Now MicroWin or Step7 can freely connect to and reprogram the S7-200. The high end-to-end latency of the cellular IP networks makes the performance pretty sluggish when compared to direct serial, but it works.
To talk to S7-315 by serial MPI, you need the special Siemens PC-to-MPI cable. Just as with the S7-200, we set up a redirected Digi RealPort, however we did NOT need to fool Step7 into thinking this was a radio modem. It just worked fine as is when given longer timeout settings.
To talk to CP243 by S7 protocol over ISODE on Ethernet, we enabled TCP port forwarding of port 102 on the Digi Connect WAN to the CP243 module. The CP243 is configured to treat the Digi Connect WAN as its Gateway IP. This also worked fine as is when given longer timeout settings.

Using Python to query Modbus slaves

2006-10-09T06:54:00.000-07:00

I use Python ( http://www.python.org/ ) at lot in my testing. It is a language designed to make a programmer's life easy and the computer sweat - in other words, it is an ideal tool for test scripts and maybe a bad tool for "constant use" tools.

Stock python has no serial support. For serial, you'll need some serial tool like pyserial - this hides details of OS and allows Linux (or Windows) style serial calls on either OS. A web search of pyserial will turn up a download site - such as http://pyserial.sourceforge.net/ . The "Vaults of parnassus" is another nice source for Python tools including pyserial. http://py.vaults.ca/~x/parnassus/apyllo.py/

Creating binary messages is not hard in Python, but a bit ugly. You use lots of "chr(x)" function to build up a binary string and to parse a binary response lots of "ord()". Other than that, look at the spec at http://www.blogger.com/www.modus-ida.org for details of the actual protocol.

CRC-16 for Modbus (or DF1):
Here is my CRC16 routine including a few test cases (written with no regard for CPU speed, since that is not why one uses Python).

crc16.py as a ZIP file

Better not try to use "unlimited data"

2006-10-06T14:00:00.000-07:00

When a potential customer starts talking to me about cellular data access to their telemetry devices, I start the discussion with the basic monthly costs of cellular data. Business cellular plans make you pay for what you use; every byte you send potentially costs you many. Of course, the natural reaction from potential customers is "Oh, that's no problem ... I'll just sign up for one of those 'unlimited plans' I see advertised all the time". When I point out these plans are available only for consumers, the natural reaction is to say "Oh, I just won't tell them this is for business ..."

I'll put on hold a moment the debate of "do unlimited data plans really exist?" and get back to cellular data access to telemetry devices. Today (and perhaps forever) cellular data access only makes sense if you have your data access well defined and under-control. If you poll X words of data every Y minutes, you will be able to select a monthly data plan that fits within a planned budget. If you connect to remote equipment for limited diagnostic maintenance and you understand that the cellular overage charges could cost you X dollars per hour, you will be able to manage your monthly bills. However, if you approach cellular data access to telemetry devices by saying you need to poll as much data as fast as you can, then this is NOT the correct technology for you. You are better to look at the various long-range Ethernet line-of-sight radios.

So back to the question of "do unlimited data plans really exist?" Hmm, unlimited - sounds nice, doesn't it. Yet an Internet search for "+unlimited +internet +cancelled" shows a growing collection of frustrated people with DSL or cable broadband, wireless PDAs, voice-over-ip (VoIP), and cellular plans who have had their "unlimited services" cancelled because they (ta-da) moved too much data. It seems unlimited doesn't really mean unlimited. I could provide links to such information, but the sites tends to be full of wild ranting language, plus I don't want to single out just a few companies. Do the search above and you'll find examples for any type of service you desire.

While I can empathize with the ranters who've found out that unlimited just means "without a predefined limit", as a network professional I understand the basis for these service cancellations. It would be nice if the marketing hype-sters could be honest enough to stop using the term "unlimited", but then no user would sign up for an honest broadband service stupid enough to define limits when competitors are shouting about "unlimited plans".

All IP-based broadband systems consist of a series of hops or links, each with a predefined maximum data throughput. All commercial broadband services try to handle as many customers as they can sign up. Therefore the performance a user sees is merely a function of how many other users are active at that instant, how much data they are trying to push through at that instant, and what is the limiting throughput of the system bottlenecks. As a business person seeking to make money, would you prefer to keep 100 users paying $80 monthly to each move 100MB of data per month (10,000MB/month), or prefer to keep the one user paying $80 monthly to move 10GB (10,000MB) of data per month? While this is an extreme example, as soon as a few of the 100MB/month users complain to the broadband service about their high-speed internet seeming pretty slow speed, the solution is obvious to the business-minded. Canceling the "unlimited service" of the one user moving 10,000MB/month will effectively double the performance of the other 100 users with no added expenses and a mere loss of $80 per month of income. Failing to cancel the "unlimited service" of the one heavy user risks causing 10 or 20 of the other 100 light users to change services with a potential monthly income loss of hundreds or even thousands of dollars. I am not saying this is honest to cancel "unlimited service" based on high usage. I am just saying it is understandable and makes business sense.

How is this cancellation legal? Easy - just read the huge terms of service contract you agree to when you sign up for unlimited data service. To generalize some typical clauses in an unlimited cellular service plan:

You agree to only use it for internet web browsing and email checking
You agree to not download or upload files
You agree to not use streaming media or peer-to-peer file sharing
You agree to not run any application servers or data services
You agree to not use the service as a replacement for a wired data circuit
You agree to not use the service as a backup for a wired data circuit
You agree that the service provider can cancel the service without notice if your usage impacts the operation of the service or other users of the service

In other words, you agree to use the cellular data service as a typical consumer with a notebook PC or PDA who spends at most an hour or two daily accessing the internet. I hope by now you can see how difficult it will be to fool any cellular service provider for long that your telemetry data system was just a normal consumer.

Cellular data - like a DSL or Cable Modem, but ...

2006-09-29T11:59:00.000-07:00

Most potential users assume cellular data modems are just a newer form of the old Hayes AT-style analog dial-up modems - you know, your old ATDT {phone-number} interface. While there are a few old, rapidly obsoleting standards which work like this, all modern cellular data systems are IP-based. They operate much like your home Cable or DSL modem. Your cellular data modem links and authenticates to your ISP (or "carrier") and is assigned an IP address - just like your home Cable/DSL modem. There are no phone numbers to dial; your modem is "connected" as long as it is powered up. Your charges relate only to IP packets moved without any concept of minutes connected.

But ... marketing hype-aside, current and near-term cellular systems are NOT broadband as you or I would use that term. Ever used a cell phone and had a call drop? A voice garbled beyond recognition? No signal now even when you had one an instant ago? The same variability applies to cellular data networks. Cell towers do their best to share the air waves with all users; thus the speed and latency in your data movement is deeply impacted by both location and what other cellular users are doing.

In my world it is even worse ... this meaning people who use IP networks primarily for telemetry or data collection with small, cyclic polls for data. While you'd think this would be a natural fit for cellular data networks, it turns out to be fairly abnormal when viewed in the context of how the standards have been evolving. Every new advance in cellular data networks is aimed at pumping up throughput for web browsing, email exchange, or image/music downloads. New advances are aimed at your average human user with a PDA, notebook computer, or iPod; users who pay lots of cash per month for a single account with little concern for Return-On-Investment (ROI). Parents of teenagers know this best - what is the ROI of the $50 to $200 monthly the average "wired" teenager seems to spend? My youngest daughter spent an amazing three years during high school working part-time at a noisy place with video games and a big dancing mouse to pay for her cell phone usage.

In old-fashioned network terms, these are connection-oriented paradigms where larger latency & overhead is invested to initiate the connection as a tradeoff to having less cost to move each fragment of a large amount of data. Without going into too much detail, understand that traditional digital cellular voice calls can be viewed as small streaming media sessions. Each cell phone negotiates a strict series of repeated tiny time-periods during which it can send the next small digitized and compressed portion of your conversation. You can visualize this as an impossible juggling act where a few dozen assistants arranged in a star around an old juggling pro set up a rhythm and tempo that passes their pins in and out without a hiccup. Each pin must be launched at the exact correct instant to find an empty hand waiting on the other side. (Note that CDMA uses a different paradigm than these GSM "slots", but the need to share a limited resource is the same.)

The 2G/2.5G cellular data networks (slightly past state-of-the-art) functioned by just mimicking a voice call but substituting small chunks of pure data for the compressed slices of voice. Later advances to these standards allowed the tower and data modems to negotiate more or fewer of the tiny time-periods per second to better match the actual data throughput. Thus your perceived data network throughput can grow or shrink as the tower as fewer or more voice calls to handle. But breaking data up into a whole series of contiguous tiny time periods separated by overhead is very inefficient. So a key evolution in the latest standards is the ability of the tower to in effect negotiate consolidating a block of contiguous tiny time periods into fewer, longer time period.

Ok, enough over-simplified details - back to why telemetry data is abnormal in this world of "Cellular Broadband". Lets say every 15 minutes I send a Modbus/RTU poll that consists of 8 bytes of data into my 400kbps to 3mbps "Broadband" connection ... well, I'll get my blazingly fast 100 bytes response back in from 2 to 5 seconds. Hmm, sounds a lot like the performance of an old 600bps (600 baud) radio modem! So cellular data networks have fairly large latency when small infrequent amounts of data are moved. This goes back to the "juggler" paradigm I mentioned before. If your cellular device has no data to send after some number of seconds, the cell tower asks it to "stop juggling" - your device gives back its allocated tiny time-period(s) to be reused by other voice or data devices. After all, there are just so many of these periods to be shared by all users. So the process of the tower and data modem reallocating these tiny time-period(s) is the primary source of this large, variable latency.

Web browsers won't notice this latency since once the web page request is sent up, the page content just flows rapidly down in multiple TCP/IP streams without the explicit poll/response behavior of a telemetry session. Since the data exchange between the tower and cell modem is active and heavy, the tower does its best to bump up and allocate as much bandwidth to the cell modem as it can spare. This is where the elusive "Broadband" performance is able to peek out.

Today, your data modem needs to send data at least every 40 seconds to avoid this latency, but with 3G this will drop to every 3 seconds - which would be cost prohibitive. You say just get an unlimited plan? Sorry, no such thing, but that will be my next blog entry.

Want more details? See Also:
http://www.gsmworld.com/index.shtml
http://electronics.howstuffworks.com/cell-phone.htm/printable

Introduction

2006-09-29T11:46:00.000-07:00

Hello and welcome to my blog. I am scheduled to update this at least weekly, although at times I may be more prolific.

I have worked in pushing industrial protocols through Ethernet and IP for nearly 2 decades. This means using TCP/IP or UDP/IP to communicate between devices on a local or even world-wide scale. At present my job involves use of cellular networks, so most of the blog will focus on "adventures" in fooling poorly-written Ethernet applications to talk to a remote IP address.

Topics to cover:

How to write cellular/satellite friendly host applications
How to work around poorly written host applications
Over time I'll slowly discuss various common industrial protocols and how well they do or don't work to remote IP addresses.