Rockwell PLC5E and SLC5 over Cellular

Summary: Customers ask me "How much will cellular cost" - this blog post walks through some examples of SCADA-style periodic polling of AB/PLC5E or SLC5/05 using the CSPv4 protocol (aka AB/Ethernet to 3rd parties) over TCP port 2222.

Real-World Numbers
For a simple SCADA-style example assume we need to read 10 words of data (20 bytes) and write 2 words (4 bytes) every time period. Obviously there would be simple optimizations to this, such as only writing data which changes or using PLC MSG blocks to push data from PLC to SCADA only when something changes. However my goal in this blog post isn't to "tweak" a solution to minimize cost, but to examine the protocol impact of using Rockwell CSPv4 over IP.

The table below shows the megabyte per month when polling once per second, per 5 seconds, per 1 minute, per 5 minutes, per 15 minutes, and per 1 hour. There are lots of variables considered ... and many more ignored. The traffic ranges from worst-case of 1005.0 MB for TCP/IP with larger header options polled once per second to best case of 0.2 MB for UDP/IP polled once per hour. This assumes the use of the CSPv4 submode 7, with local LSAP addressing and ignores that Rockwell PLC5E and SLC5/05 don't support CSPv4 within UDP/IP. Raw efficiency at moving the data bytes ranges for about 10% for UDP/IP to barely 1% for TCP/IP; which means most of what you are paying for is not related to actual, meaningful field data.

( Click this image to see a larger version )

(is at http://iatips.com/blogimage/rockwell_cspv4_traffic.png)

Notes on the Table

Since this example reads and writes small amounts of data, it assumes a SLC5-style Protected Typed Read with 3-Address Fields and the corresponding SLC5 write.

The smaller 40 byte TCP/IP header has no options attached; the larger 52-byte TCP/IP header includes the RFC 1323 Timestamp and Window Scale TCP options. These appear to be the normal default for Linux and easily becomes enabled under Windows since all applications share a single setting in the Registry.

The two time columns "15 min (Alive)" and "1 hr (Alive) assume a roughly 4 min 45 sec TCP keepalive to prevent the socket from closing. This reduces the traffic by the extra open/close overhead in exchange for billable TCP Keepalive packets. Keep in mind this ALSO requires the PLC to be properly configured to NOT close the idle sockets. By default, my SLC5/05 seems to close the idle connections in a few minutes.

The two time columns "15 min (Cls)" and "1 hr (Cls) assume the socket is closed after the a data polls, and the TCP socket and CSPv4 session must be reopened for teh next poll.

Discussion
Of course the standard costs of using TCP/IP verse UDP/IP apply:

• TCP/IP uses larger headers, ranging from 40 to 52 bytes per packet as compared to UDP/IP's smaller 28 byte of header.
• TCP/IP involves the TCP Acknowledgments, which may result in separate, billable 40 to 52 byte packets moving frequently without any meaningful field data.
• TCP/IP may require reopening a socket, costing 120 to 250 bytes per open, plus closing costing from 160 to 400 bytes. Exact sizes are hard to predict since both opening and closing of sockets tend to be "pushed" and result in excess retransmissions and retries when high network latency is true.
• TCP/IP over unknown 3rd party wide-area-network infrastructure requires at least 1 TCP packet to move every 4 minutes 45 seconds to maintain health. This means either a data packet or a TCP Keepalive with data.

It should be clear to see why using UDP/IP over cellular (which is very reliable) is much cheaper than using TCP/IP. There are no socket open, close, acknowledgment, or keepalive costs. Plus field experience has shown that rapid IP retries rarely succeed. For example, a customer polling with UDP every 5 minutes with 3 explicit retries if no answer will likely have the original poll succeed or that poll and all 3 retries fail. Again, cellular is very reliable in that packets almost always make it through unless there is a network or congestion issues and then only time (a few minutes) solves the problem. So such customers have abandoned retries and just ignore 1 failed 5 minute poll and "retry" in 5 minutes.

CSPv4 issues include:

• Rockwell PLC and software tools do NOT support use of UDP/IP - my tests with UDP/IP have to be conducted with the Digi One IAP which happily bridges CSPv4 between TCP and UDP (as well as to or from Ethernet/IP and DF1).
• CSPv4 requires the exchange of a pair of 28-byte negotiation TCP packets when a new TCP socket is opened to inform the client (master) of a server (slave) assigned session handle. This nearly doubles the overhead of an open-poll-close socket paradigm.
  
• The 28 byte CSPv4 header really contains little useful information; such excess bytes cost nothing tangible under Ethernet but cost cash in the form of requiring larger cell plans over cellar.

In conclusion, CSPv4 will be a rather poor choice for raw, periodic polling of remote AB PLC. ODVA Ethernet/IP (as implemented by all vendors) is even worse.

Your only effective solution at present is to carefully craft a set of MSG blocks to push data from the field in a report-by-exception paradigm. Of course you also must include safe guards within your PLC to prevent rapid, repeated MSG block triggers during system failure that could cost you thousands of dollar ($$$) in a few days.

DF1 Open Source for Visual Basic 2005

Archie has started a SourceForge project for his AB DF1 code running under VB 2005. He has checked in a full first set of files (unlike many SourceForge projects which get created but NEVER have files :-\ )

http://sourceforge.net/projects/abdf1/

I haven't looked over his code yet, plus all I have is VB 2003 .NET.

Hopefully Microsoft has STOPPED the old VB issue that each new rev of VB is neither 100% forward nor backward compatible ... one always need to "tweak" a few lines to make the port work. I've used VB 1.0, 3.0, 4.0, 5.0, 6.0 and now VB 2003 and none of these have liked old code being pulled forward.

Cellular to Allen-Bradley SLC5/05 on TCP 2222

The old CSPv4 protocol.

The Rockwell/AB SLC5/05 and PLC5E natively speak an older "unpublished" protocol named CSPv4, although most third party vendors call it either AB/Ethernet or AB/TCP. It moves only on TCP port 2222 - ODVA Ethernet/IP I/O Messaging is only on UDP port 2222, so they don't conflict. The protocol consists (normally) of a 3-part packet:

• 28-byte header
• 4 or 15-byte LSAP or end-point addressing packet
• PCCC message which is basically what DF1 documents as an Application Packet

In general, the packets are fairly sparse and compressible (if you have the tools to do this). The characteristics of CSPv4 which impact cellular (and wide-area-network) support:

• Rockwell tools and PLC only support use of TCP/IP and port 2222; this greatly limits use of CSPv4 in NAT'd networks since the remote NAT router can only forward TCP port 2222 to a single remote PLC.
• CSPv4 includes a single TCP packet exchange to "register a session" or connect. If you are polling faster than the PLC will hang-up on you, then this is not important. However, if you poll slow enough that a new TCP/IP socket must be opened for each poll, then even ignoring the TCP socket open/close overhead this nearly doubles your traffic costs.
• In tests, a SLC5/05 seems effective at including the TCP ACK response to the host within the CSPv4 data response packet, so you only have to pay for one empty TCP ACK, which is the host's acknowledgment to the PLC for the response.
• TCP Keepalive could be an issue, since most hosts fail to issue it and the SLC5/05 I've tested against either doesn't issue TCP keepalives
  or does it very frequently.

Interested in Cellular? Setup a DMZ Lab

Summary: Last post I suggested people interested in cellular data start by learning how to use their home cable router. In this post I suggest the next step, of how to make your life easy at work once you're ready to start testing real cellular or satellite access by IP.

Your Second Step should be to set up a simple, isolated low-speed broadband link at work ... create your own DMZ lab.

Sigh - I waste so much time listening to customers complain about how difficult it is to get the IT department to give them custom firewall permissions. Since modern "Security" wisdom is to block everything until proven safe, I waste more time asking customers complaining that their Modbus/TCP or Rockwell access not working to first talk to their IT group to make sure they aren't blocking unknown binary traffic by default. I waste yet more time when customers struggle for days and finally have to formally get someone in IT to help study the corporate firewall logs to see if any traffic is getting through or not. An interesting epiphany occurs when I suggest they just look into paying roughly $50 per month for a private connection for this. It is surprisingly cheap to do this and makes a lot of people's jobs 200% easier.

So far the feedback from customers has been quite positive, with IT departments over-joyed at the idea (slight exaggeration :-] lessor-of-two-evils may be a better term). This really makes sense; IT is charged with keeping the corporate system working and secure, so when you ask for yet another odd, unknown firewall hole to be opened, you ask them to risk their jobs. Plus trying to keep custom firewall settings updated for 50 different projects is an ongoing headache and ongoing risk for mistakes. I know that Digi's IT group is very satisfied with their policy of not offering custom firewall rules on the corporate LAN but instead helping teams set up such private connections in a safe, isolated manner.

Simple DMZ Lab Design
The simplest lab design is little more than a copy of what you have at home: a computer or two, an 8 or 24-port Ethernet switch, and a simple NAT router to "share" the internet connection with a dozen devices. This allows you to freely set up a few OPC servers and Master PLC to test access to remote cellular and other wide-area-network based systems.

• Locate an empty office or lab room for your new network. Perhaps your IT people should pull out or disable the corporate Ethernet in this room. You are going to create a small "DMZ"; a small isolated network that has limited security consequences if you goof up and let a hacker inside. You'll want good security tool installed on your Windows and Linux computer used in here.
• Arrange for a low-speed business broadband link with one fixed IP address. 256Kbps is more than enough for general PLC/SCADA testing and should cost in the range of $35 to $50 per month. Yes, just $35-50 per month! I had one customer forced to pay his IT department $100 per month to open one TCP/IP hole in the corporate firewall!! Gee, he could install 2 DSL links for that. Now, be patent when you talk to your carrier, as they are geared to sell the expensive primary access lines used for all corporate traffic including servers. Keep stressing that you want a low-speed secondary line for use with some network testing and eventually you'll locate the low-cost plans you want.
  
• Set up a DNS name for your DMZ lab. Online dynamic DNS providers support user-selected DNS names for static IP addresses. I use dyndns.org for both my dynamic and static IP but there are many out there. You won't need any form of DDNS update client since your IP never changes and you must enter the name manually anyway.
• Unless you plan to implement large VPN systems, just buy a nice consumer-grade DSL/Cable Router ... the same kind you use at home is fine. If you plan to set up a serious VPN infra-structure, then you'll just need to bite-the-bullet (& suffer the learning curve) of buying a commercial IT-grade router with VPN server capability built in. Be warned that while many consumer-grade routers mention "VPN Support", they are in fact sub-optimized and documented only for home-office users who connect into a corporate Windows or Cisco VPN server. Normal human beings will find them nearly impossible to set up for anything else!
  
• Do you want more than one public IP address? You need to pay a monthly surcharge which varies greatly per carrier, but could be in the range of $25 per month for 8 IP addresses instead of just 1. Plus you will need a larger IT-grade router since the consumer-grade routers won't support more than 1 public IP address. Most users won't need more than 1 IP address. However, having more than one IP address is helpful if more than 1 team shares the lab; this prevents them from trying to setup conflicting router configurations. Also, a few extra IP are helpful if you want to place a PLC "online" for your customers or sales force to access during customer-site demonstrations in the field.
  
• If you need to access your corporate network from your DMZ lab, then you need to arrange some rules with your IT people. Perhaps the rule is using a notebook computer with 802.11 wireless to the corporate network is Ok as long as the notebook is NEVER connected to the lab's Ethernet. Remember, since your lab has it's own public IP address you can even use FTP or a VPN client to connect "legally" out your corporate network and back into your lab via the Internet.
  

Fancier DMZ Lab Design
Since Digi is basically a "communication company", the lab I get to use is much fancier. It has 32 public IP addresses and even limited secure access from the corporate LAN. Of course I have to share this lab with other teams, so I'm not owner of 32 IP. As an example, here is how my lab is setup:

• Digi's IT group maintains a Cisco PIX router (a mid-range $800 model) that manages the 32 public IP addresses. Actually, this is NOT a complication since this router does not by default firewall any traffic; it merely distributes raw traffic based on public IP to one of many to internal IP addresses in an organized manner. I was lucky enough to get in the lab early and to be assigned 2 of the 32 public IP addresses.
  
• My first IP address receives 100% raw internet traffic at an internal static IP I selected; so an external IP such as 70.x.x.140 forwards to my internal IP of 192.168.20.159. Since the goal of the lab is to avoid burdening IT with TCP/UDP port forwarding chores, one could place a consumer-grade DSL/Cable router at this IP. Placing 2 routes in series is NOT a problem - do a net-trace of how you access www.google.com and you'll see a dozen or more routers in series. Instead of a pure hardware box I have a Ubuntu Linux machine running firewall and router tools at this IP. This is where I forward Modbus/TCP to one PLC and Rockwell Ethernet/IP to another PLC. I prefer the Linux box to the $39 hardware box because it gives me a richer view of traffic in and out, plus I can run an Ethernet sniffer such as WireShark to see a complete trace of the 2-way conversation taking place.
  
• For my second IP address I had Digi IT setup the PIX router to just forward a simple, safe list of Modbus, Rockwell, Digi, and other industrial protocol TCP and UDP ports. I normally have a Digi One IAP (an industrial-protocol aware Ethernet-to-serial device) at this IP address, but I can safely swap in a Windows machine when a test requires use of Windows tools.
  

Just as your home router does, the main Digi IT-supplied PIX router does out-going NAT to the internet and internal DHCP address assignment. So our DMZ lab has its own internal subnet of 192.168.20.x addresses. Any device in the DMZ lab can access the Internet - with or without going through my Linux firewall/router. Of course the other teams in the lab don't go through my router, but direct to the PIX router.

Since i study cellular usage, I have have a Digi Connect WAN providing an Ethernet-based cellular router in the DMZ lab. So I really have 3 potential routers to use - while it takes a bit of IP experience to not get confused, this allows me to have devices configured to selectively treat any 1 of the 3 routers as "the default gateway/router".

• For example, I can have a Master/Client device connect out to a cellular-based Slave/Server using a route such as Master => out PIX+DSL => in Cellular => Slave.
• For example, I can have a Master/Client device connect via cellular to a DSL-based Slave/Server using a route such as Master => out Cellular => in DSL+PIX => in Linux-Box => Slave.
• In both of this situations I can see BOTH ends of the conversation, which is a huge help in testing, timing, or troubleshooting new applications.
  

The bonus for having Digi's IT team involved is the PIX router also allows controlled, secure access into the DMZ lab from our corporate LAN. Technically, the PIX runs a set of rules similar to those used by the main corporate firewall out to the Internet and it just treats the 192.168.20.x subnet as a miniature internet. So I can sit at my desk and safely check on equipment running in the DMZ lab. Of course this is limited to equipment treating the PIX as the default router - if you understand IP routing you'll understand why, but at least it lets me log into my Ubuntu Linux router from desk.

Interested in Cellular? Do some homework

Summary: Unless you are a pro at IP routing, you'll save money and time by learning the basics of IP routing over your home Cable/DSL connection instead of a demo cellular account. Bottom-line ... if you cannot succeed at using your office computer to poll a PLC placed at home via your Cable/DSL connection, then you will NOT succed trying the same trick over satellite or cellular connections.

Homework - Work at Home
When engineers first launch into a cellular data pilot it can be a bit like Christmas with the excitement of new toys, future trends and being "on top of it". However, I encourage anyone interested in using cellular or satellite-based IP systems to do some home work first ... literally "work at home". You'll save lots of cash and avoid many headaches by learning the basics at home first.

Most of you have cable or DSL router/modem at home, so start there. Take a PLC or controller home. If it has an Ethernet port you are all set; however if your device is RS-232 based, then beg, steal, borrow, or purchase a simple Device Server such as the Digi One IAP (fancier, Modbus and Rockwell protocol aware) or the Digi One SP (much cheaper but just a raw Ethernet-to-serial converter). Your goal is to connect from your office computer over the Internet to this device at home ... if you cannot succeed at this, then you won't succeed at cellular access either! But unlike with cellular, all of your trial-and-error over your Cable/DSL Route won't be costing you by the byte.

Just remember that your "Home Cable/DSL Terms and Service Agreement" likely forbids running "servers" so don't go and try to setup an e-commerce shop once you see how easy it is to access your home from the Internet.

Get to Know Your Cable/DSL Router Box
Hopefully you all have an external commercial router box that you either got from your ISP or bought at any big-box store for $39 to $59. If your computer connects directly into your modem or you were fooled into using Microsoft's "Internet Sharing" tool on one computer, save your sanity and go buy a cheap router box! For your $39-59 you get a 4-port switch, a professional stateful-firewall and NAT (more about that later), a wireless access point, and it all consumes maybe 8-10 watts of power so costs you a few $ a year to run. If for no other reason, you just don't want the mindless broadcasts and hacker probes taking a percentage of your home computer's bandwidth. For my VPN testing I have some Linux boxes up exposed like this and they see up to 50 broadcasts per second and a few dozen probes for open Windows and Unix services per hour. There is NO REASON to expose your home PC to this rubbish - use an external router box ... period.

Step 1: Learn how to log onto your Cable/DSL Router.

• Under Windows 2000 or newer, open a command window and type the command "ipconfig". You should be shown your computer's current IP Address and the Default Gateway, which is another name for your Cable/DSL router. Most likely the router has an IP such as 192.168.0.1 or 192.168.1.1.
  
• Confirm you can ping your Cable/DSL router with this IP
  
• Open your web browser and browse to the address - as example type the URL "http://192.168.1.1". You should be asked for a user name and password.
• Check with your router documentation or go on line to the vendor and read the user guide. For example, at home I have an ActionTec router/wireless access point supplied by Qwest, and when it first came it has no user name and a password of "admin". This is actually not so insecure since by default you can ONLY access this web page from inside your firewall/router. But common sense says changing this name/password is wise.
• There is no way I can explain how all Cable/DSL routers work, but once you can log in you should be able to find a status web page which gives your currently assigned external IP address and 2 DNS addresses. This is how the world sees your home system - write this info down. For example, my home Cable/DSL router (as of today) has the temporary (dynamic) IP of 63.228.51.x.

Step 2: Nail Down a fixed DNS name for your Cable/DSL Router.
So at this point, you know how to access your raw "face" exposed on the Internet. Now we want to give ourself a nice, memory-friendly DNS name to represent that face.

• As mentioned above, my Qwest IP is dynamic and liable to change at any time. So while I could go to the office and try to point my OPC server or PLC software at 63.228.51.x, I can never be sure how long this will work. In reality it only changes every few months or if I power-cycle my router, but the solution to this problem is very easy so we should solve instead of work-around it.
  
• Sign up with one of the many free online Dynamic DNS providers - I use dyndns.org. The Digi Connect WAN (cellular router) family directly supports this, as do many LinkSys and DLink-class home products. In a nutshell, they allow you to create a domain name such as sillyjoe.gotdns.org or sammy345.dyndns.org and then a client tool on your home system automatically updates this DNS name every time your ISP changes your dynamic IP address.
• While the above service is free, you may want to pay the $10 or so per year for a minimum account. This makes the service more tolerant of errors on your part - for example many services automatically delete your free account if it is untouched for 45 days and so on.
  
• If you have a Windows computer, the easiest DDNS update client is just to download the Windows tool recommended by your Dynamic DNS provider. This client automatically monitors the Cable/DSL router's IP as it accesses the internet. If your IP has changed, it correctly updates the DDNS (dynamic DNS servers). I stress the word "correctly" since many external Cable/DSL Router boxes which support DynDns and such services come with bugs which cause your free service to be deleted within hours of setup. So if you chose to use your Cable/DSL Router to maintain your DDNS name, make sure you have the latest firmware upgrade on it!
  
• Within an hour of setup, anyone in the world should be able to ping your new DDNS name and get a response.
  

Step 3: Learn how to Port-Forward within your Cable/DSL Router.
We are almost ready to try access - but if you point your OPC server at your DDNS name ... nothing will happen since your Cable/DSL Router does NOT understand Modbus or other industrial protocols. Remember, the IP your DDNS name represents is the IP address of your Cable/DSL Router and NOT the IP of your home computer nor is it the IP address of your PLC/controller device.

• Log back into your Cable/DSL Router and locate the setup for port forwarding. Some routers call it setup for applications and games. We need to configure the router to FORWARD specific TCP and UDP ports to Ethernet-based devices you have at home. If you don't know what that means, you are in for a tough time using wide-area-network technologies - I suggest you go to any bookstore and buy a book on basic networking that covers what TCP, UDP, IP and NAT are. This is really key to success in this area. You don't need to be an expert, but you do need to understand the basics!
• In summary, if you think of the IP address as being synonymous with the main phone number in a building (aka - how to telephone the building), then the TCP and UDP port numbers are synonymous with phone extensions within that building (aka - how to reach a certain department or service). So for example, a Modbus/TCP OPC server will connect to your router using the IP (main phone number) attached to your DDNS name, and then request a connect to TCP port 502 (the service). We need to configure the router to accept and forward the Modbus/TCP traffic to your Modbus PLC. So take the example of a Modbus/TCP device on my local network with the IP 192.168.1.105. We need the router (at say IP 63.228.51.x) to accept any incoming connection on TCP port 502 and forward the packets to the local Ethernet device at IP 192.168.1.105 TCP port 502. Since the PLC has a web server and most ISP block access to home web servers, we'll tell the router to forward TCP port 8080 to local port 80. Depending on the brand of router you have the configuration can get fancier than that - but basically we'll end up with a line in the table looking something like:

Incoming port	Service	Local IP	Local Port
502	TCP	192.168.1.105	502
8080	TCP	192.168.1.105	80

Step 4: Get to Work Learning
That's it - at this point you should be able to use a Modbus/TCP OPC server to poll your PLC indirectly by polling your DDNS name on the standard TCP port 502. Pointing your browser to http://your DDNS name:8080 will pull up your PLC's web pages (the ":8080" tells the web server to use TCP port 8080 instead of default 80).

Of course there is no security offered here - anyone in the world can access your PLC so this is just for educational purposes. Here are some common port numbers to use:

• Modbus/TCP uses TCP port 502
• Digi Ethernet-to-Serial products use ports like 2101, 2102, etc to access a serial device by raw TCP or UDP sockets.
• Digi RealPort uses TCP port 771 (TCP 1027 for SSL/TLS secure connection).
• Rockwell AB PLC5E and SLC5/05 use TCP 2222 for the older legacy CSPv4. This is often called AB/Ethernet or AB/TCP by 3rd party vendors
• Rockwell ControlLogix and ODVA Ethernet/IP uses TCP port 44818, UDP port 44818, and UDP port 2222. But be warned Rockwell tools are very poorly designed for wide-area network use.
• Siemens S7 protocol uses TCP port 102
• GE SRTP uses TCP ports 18245 and 18246
• GE QuickPanels use TCP port 57176 for configuration
  

Ok, so now you should be able to take any of your Ethernet or serial device and test to see if you can access remotely. You'll likely need to slow down your tool - increase the response timeouts from a few seconds to 10 seconds for direct Cable/DSL access and 30 to 60 seconds for cellular.

Rockwell PLC and TCP Headers

I have started running some tests of standard Rockwell protocols querying off-the-shelf Allen-Bradley PLC, with the goal to create a series of "estimators" for traffic. A user would enter the data to poll and the tool will estimates the data byte load contributed by this poll pattern.

The Mystery 17% Cost Increase:
Last night I ran a test polling ten words once a minute from an Allen-Bradley SLC5/05C's N7 file over GSM. This is nothing exotic - I ran similar tests a few months ago and had preconceived ideas of what to expect ... beep ... wrong! In between Then and Now, some unknown application changed my Windows XP system registry, enabling the "RFC 1323 Timestamp and Window Scale TCP options". The end result was an unexpected 16.51% increase in data byte traffic with no perceived value.

I have no clue which tool did this; and unfortunately Windows (at least 2K and XP) use a single global setting for the entire TCP stack. I could change it back ... but would that break this other mystery application? Will this other mystery application just change it back? Will I launch a mini cold-war race as this mystery application tries to keep RFC 1323 enabled and my test tools try to keep it disabled?

The Byte Counts with and without RFC1323:
Here is an exact accounting of the change in byte counts - remember, cellular is basically a mobile-IP tunnel which moves TCP/IP or UDP/IP as pure data payload. So you pay for both the IP and TCP headers, plus any data-less TCP Acknowledge or Keepalive packets.

I'll ignore the opening and closing of the socket, plus TCP Keepalive since I'm polling fairly steady-state once per minute. The PLC includes the TCP ACK in the response, so at least we avoid 1-of-2 data-less TCP Acknowledgments.

	no RFC1323	with RFC1323
Request: IP header	20	20
Request: TCP header	20	32
Request: CSPv4 Packet	42	42
Response: IP header	20	20
Response: TCP header	20	32
Response: CSPv4 Packet	56	56
Client ACK: IP header	20	20
Client ACK: TCP header	20	32
Client ACK: (no data)	0	0

	no RFC1323	with RFC1323
Total Bytes per Poll	218	254
Total Bytes per Hour	13,080	15,240
Total Bytes per Day	313,920	365,760
Total Bytes per Month	9,417,600	10,972,800

So this means a user doing 1 read of 10 words per minute would magically see a 16.51 % increase in data traffic ... just because they (or the IT department or even Microsoft Windows Update) changes a hidden registry setting. This is yet another example of both how hard it is to keep tight control on your cellular data costs; plus adds to my belief that using off-the-shelf host applications over cost sensitive IP networks is a losing battle. At some point you'll need a tool or device which is 100% "under-control" when it come to packet creation.

Windows Registry Details:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Tcp1323Opts

Tcp1323Opts
Key: Tcpip\Parameters
Value Type: REG_DWORD—number (flags)
Valid Range: 0, 1, 2, 3

• 0 (disable RFC 1323 options)
• 1 (window scaling enabled only)
• 2 (timestamps enabled only)
• 3 (both options enabled)

Default: No value. The default behavior is as follows: do not use the Timestamp and Window Scale options when initiating TCP connections but use them if the TCP peer that is initiating communication includes them in the SYN segment.

Description: This parameter controls the use of RFC 1323 Timestamp and Window Scale TCP options. Explicit settings for timestamps and window scaling are manipulated with flag bits. Bit 0 controls window scaling, and bit 1 controls timestamps.

Modbus Report-By-Exception over Cellular IP

Summary: While traditionally serial Modbus has been considered unable to use Report-by-Exception, when combined with IP networks Modbus Report-by-Exception becomes very natural and effective.

Modbus/TCP is inherently peer-to-peer
People using Modbus/TCP over Ethernet or IP are familiar with its ability to function as peer-to-peer. Most PLC with Ethernet ports can function concurrently as a Modbus/TCP slave and Modbus/TCP master. So 2 PLC can very easily connect - with 2 separate Master-Slave TCP connections - and share information. One TCP connection is a Master/Slave connection with the first PLC as Master and second PLC as Slave. The other TCP connection is a Master/Slave connection with the first PLC as Slave and second PLC as Master.

Technically, this is not Report-By-Exception in the true sense of a protocol. However, since the PLC-as-Master communication events can be triggered by field inputs, it has the same effect as writing information only upon exception or when change is relevant.

Modbus/RTU as peer-to-peer
Serial Modbus/RTU is a bit harder to use this peer-to-peer trick with. A device with 2 serial ports can of course have 1 port configured as Master to issue remote reads and writes, while the 2nd port is configured as Slave to answer requests. When connected to a 2 serial port Modbus IP to Serial Bridge (such as the Digi One IAP), the 2-port serial RTU becomes a full Modbus/TCP peer, capable of operating fully peer-to-peer with other PLC and SCADA/OPC applications.

However, vendor's aren't blind to the marketing aspect of "more hardware". While adding a 2nd serial port to a one-port RTU may only cost a few dollars, most likely the 2-port RTU is a much more powerful unit, so the actual end-user cost may go up hundreds of dollars. The same is true of Ethernet; while adding an Ethernet port may only cost a few dollars, user's expectations of Web Pages and fancy functions means the Ethernet-enhanced device price may be $500 or more above that of a simple 1 serial port RTU

Fortunately, the Digi One IAP (as well as PortServer family) allow Modbus/RTU slaves to use Report-By-Exception on the serial port. As long as only one serial slave is on each port, the Digi uses configured knowledge to "split" the single serial port conversation into two traditional Modbus/TCP connections. So traditional remote Modbus/TCP masters can query the serial slave RTU, completely unaware that on occasion the serial slave RTU wakes up a acts as a Master to write data during Exceptions. Somewhere, a traditional remote Modbus/TCP slave will receive Modbus/TCP messages from the serial RTU slave, completely unaware that when not busy reporting exceptions, the remote "Master" is really a passive Modbus/RTU slave.

This feature is ideal in wide-area-network situations were bandwidth is limited or data traffic is billed on volume of bytes moved. For example, many SCADA systems only need to check on remote status every few hours ... for example lift pumps in a storm sewer system do absolutely nothing interesting for weeks or even months in the absence of rain. Even during a normal rain, checking on them every few hours is likely enough ... that is *IF* the remote life pumps can send Report-By-Exception messages during system problems.

For example, we have one customer piloting use of Modbus Report-By-Exception over cellular data network. Their eventual target is to poll the remote sites once per day. They use a simple, single-port Modbus/RTU slave which combines I/O with an LCD and push buttons to make a simple, self contained "RTU" or Remote-Terminal-Unit in the truest sense of the word. Use of Modbus in UDP/IP and Report-By-Exception allows this customer to plan for $12 per month per site bills. If forced to poll continuously with Modbus over TCP/IP, they would need to pay $50 or more per site per month. With hundreds or thousands of sites, that is a huge cost savings and opportunity for better ROI (return on investment).

More Information
Here is a general discussion of how to design Modbus/RTU serial slaves and masters to gracefully handle Report-By-Exception:

• http://iatips.wikispaces.com/ModbusRBX

Here is a more focused discussion of the Digi One IAP and PortServer TS1, TS2, TS4, TS8, and TS16 handle serial Modbus/RTU Report-By-Exception

• http://digitips.wikispaces.com/Modbus_Report_By_Exception

Cellular IP-Friendly Apps - Response Delays

Back to my series of entries on creating graceful IP apps

Many newly written Ethernet-enabled applications incorrectly equate "Ethernet = Fast". They overlook that Ethernet is often just a path into other slower IP-based networks. Worse, some well meaning programmers set the response default to 250 milliseconds and limit the user configuration to a maximum of 5 seconds - I'd say so far about 20% of the applications I've had to help customers will limit Ethernet timeouts to 5 seconds or less.

But cellular networks have a high end-to-end latency - especially if the line has been idle for a few minutes. Normal slave response times will be near 2 seconds with round trip delays up between 10 to 12 seconds common each day (see my entry on real world Modbus numbers). Interestingly enough, every cellular "expert" I talk to keeps correcting me that cellular latencies are in the 50 to 100msec range and getting better every new "gen". Well, I guess my Saturn ION can do 400 miles-per-hour also ... if you drop it out of an airplane! Well, regardless of what these "experts" are smoking, my simple tests show otherwise where it really counts ... in actual real world tests run over the Internet to cellular-based IP devices.

Recommendation: IP applications should default to a 3 second response timeout. Applications must allow users to configure this timeout to be lower (perhaps to 250msec) and also higher to at least 60 seconds.

Impact: On Ethernet this should have no direct consequences since the timeout only has affect if the remote is no longer available - in which case the remote is going 'offline' anyway. The minority of users who really want a 250 millisecond timeout can set it manually, while cellular users who want a more reasonable timeout of 15 seconds can also set it also.

For cellular networks, the real problem with premature timeout is the customer has already paid for the request and very likely will also pay for the response - even if the response comes after the application gave up on the response and did a request retry. Assuming the user is polling the remote at a moderate pace to control costs, there is no harm is waiting longer for the response to maximize the value of the traffic paid for.

Another simple example is an application that sends a request, then timeouts twice and retries twice. How will the application react when it receives three responses at the same time? Remember, the first two requests probably were not lost; they still likely reached the remote device and created responses. Their responses may have been just delayed longer than expected. Since serial Modbus doesn't include enough information in a response to match it up to a request, this can cause serious misoperation of the system. Protocols including a sequence number should handle this more gracefully, but it will still be a waste of money.

We have also seen protocols which treat unexpected responses as a reason to abort and reset the communication channel, which further adds to cost. For example, we had one super headache with a big-name seller of "energy curtailment" systems. The end user insisted a 5 second timeout was the maximum they could tolerate (ie: wishful thinking - set a 5 second timeout regardless of reality). So lets just see what happens when we hit one of the rare but expected latencies over 10 seconds.

1. SCADA software sends out request sequence 74
2. 5 seconds later, SCADA times out 74 and sends out 75
3. 5 seconds later, SCADA times out 75 and sends out 76
4. 1 second later - since TCP/IP is reliable - all three responses return
5. SCADA is expecting response 76, but sees 74 ... Oh, big problem ... need to reset comm subsystem
6. SCADA sends reset to remote RTU, expects response 1 but ... da da ... sees response 75 since they never flushed the old info and TCP/IP is reliable.
7. SCADA sends a 2nd reset to remote RTU, expects response 1 but sees response 76 since they never flushed the old info and TCP/IP is reliable
8. At this point, I hope you see that there are still 2 responses to the comms reset in the receive queue!

Anyway, whenever this reset "temper-tantrum" occurred it would take 10 to 15 minutes to get the connection back up. Of course one problem was the stupid customer unwilling to set the correct timeout, but the SCADA software was defective since it wasn't smart enough to just discard old responses with timed out sequence numbers. In the above example, life would have been fine and dandy had the SCADA system just discarded responses 74 and 75 since it expected 76.

Cellular and DNP3

I was just at the Distributech show earlier this week - the show for power utilities. Lots of interest in cellular access. I know both OSI and Itron have successfully tested their software against our cellular product.

I have a DNP3 RTU up on my public cellular device, but need to confirm details of how the public can access it. I also had a discussion with the primary provider of DNP3 source code in the world and we will be looking at putting a DNP3 slave simulator up via cellular. It would be really userful if this could expose some of the statistical & diagnostic info managed by the simulator. This would help software vendors fine tune their software to handle the variable latency of cellular.

Do Users Really Want Industrial Ethernet?

(For those impatent to read this to the end - I'm not saying don't use Ethernet ... I am just saying be careful you understand what your customers expect and what functionality they will assume you include *for free* when you add Ethernet)

My last post created some interesting feedback. But I want to emphasize a topic from that post more fully. For the last 15 years I've been involved in the "multi-vendor interface" business - linking multiple vendors' equipment by data comms. First I worked in RS-232 and 485, then fiber optics, then Ethernet, and now by virtually every technology that moves TCP/IP.

From time to time I am contacted by some pretty desperate customers - for example I had one customer who had piloted some Ethernet-based temperature sensors. Things worked fine in the lab with their lab computer, so they bought 50 ... only to find out they couldn't use them. It seems these sensors really were "just Ethernet" - they talked by Ethernet broadcast and direct MAC-layer packets. They didn't support TCP/IP and therefore could NOT be routed by any standard network infrastructure. The user could not talk to any of the sensors they had intended to install in panels around the plant because the "Computer Room" wasn't on the same physical Ethernet segment as the "floor". There was no way to broadcast or unicast MAC-level between the systems. This customer hoped I knew of some magic box to act as gateway between TCP/IP nodes and pure Ethernet nodes; I didn't.

So this brings me back to the concept of the true cost to implement "Ethernet". Customers who ask for Ethernet are not really asking for Ethernet hardware or an Ethernet media bus. They have the expectation that they can interface your "Ethernet Devices" with the wide variety of other equipment they have - including WiFi, routed Ethernet, fiber optics, wide-area networks, and so on. They also expect (at least in a future firmware rev) web pages for configuration, SNMP for remote management, strong encryption, and so on.

So the term "Ethernet" has taken on a life of its own - remember when 802.11 was called "Wireless Ethernet". Well, there is absolutely NOTHING Ethernet about 802.11, yet it was a useful PR move to link the two. No doubt it helped spread the acceptance of WiFi as we now call it. Interestingly enough, the current PCI verse PCI-Express adapters you buy for a PC are using the same PR trick - linking a new, unknown technology to an old established technology that merely accomplish the same function by very different means. Maybe Sony should have called Beta-Max VHS-Max instead ... but then I'm showing my age by even knowing that a consumer-oriented video standard other than VHS even existed.

But back to Ethernet. If you are a small device maker and have yet to start making Ethernet-based products, just be aware that customers who ask for "Ethernet products" aren't really asking for ... err, products with Ethernet. They are asking for products which integrate into (at a minimum) the wide family of TCP/IP based technologies out there. I am not even talking about should you support Modbus/TCP or ODVA Ethernet/IP or ProfiNet yet. I am just saying customers will expect your "Ethernet products" to be able to hold a raw TCP/IP or UDP/IP conversation with all of the other equipment they are investing in daily.

So the cost to add an Ethernet port is just a small part of your cost to "add Ethernet". That is why companies like Digi can sell Ethernet-to-Serial converters or sell "async Ethernet driver chips" like the Digi Connect ME which links to your CPU's serial UART. These devices of course cost more than $9.95 or even the cost of a few new hardware chips, but that higher cost is paying for TCP/IP, web servers, SNMP servers, strong AES encryption and all of the other things your customers expect when the buy "Ethernet products".

So to digress a bit, I suffer this "Oh, don't worry ... it's Ethernet" on a daily basis. So far I have to say at least 95% of the off-the-shelf software applications I test supporting TCP/IP don't work well with technologies other than direct Ethernet. This includes problems not only when extremely different media like satellite or cellular, but even when WiFi is used. So that is part of my mission in this blog - what you want is NOT to Ethernet-enable your products. Instead you need to "IP-enable" your products by way of an Ethernet interface.