Report-By-Exception? Maybe not

One of the PLC vendors I am just starting to deal with was bragging about the power of their report by expection abilities ... how wonderful for lower cellular data costs they claim. Unfortunately they only support TCP/IP - no UDP/IP. I'll cover this application (& name the PLC) more after I have some time to work at it.

However, the big problem with their "report-by-expection" is that the PLC holds the TCP/IP socket open 24/7 and sends a TCP Keepalive every 1 minute. Sending a TCP Keepalive (2 x TCP packet headers with no data) every 60 seconds costs about 3.4MB per month! I have a hard time seeing that as "low cost". They would be better to just poll real data every 4 minutes and eliminate the TCP keepalives!

Hopefully there is a setting in the PLC to change this behavior, which is why I need to look into this more. I just want to point out that just because a system claims to only move data during exceptions ... that does NOT mean it has very low cellular data costs. Don't forget that data costs include things you don't normally see - TCP socket open/close, TCP keepalive, TCP ack, and TCP retransmissions are all things you pay for.

Real World Cellular - ControlLogix PLC

Summary: Before I listed some real world numbers for Modbus polling. This time I walk through some of the costs and issues of using ODVA Ethernet/IP to talk to a Rockwell ControlLogix PLC.

The Convoluted Path of Wide-Area-Networks:
In general the magic of IP hides reality from us all. We tend to think "now I am browsing Google.com or iatips.com", but we don't really understand how COMPLEX and MIRACULOUS this really is. Your computer is NOT connected to either of these web servers; instead your computer uses the services of a dozen or more other computers/routers to get from "here" to "there". Every single data byte must be forwarded hop-by-hop through all of these cooperative peers.

As example, here is a Trace Route (tracert) of access from a computer within my test lab to a ControlLogix PLC sitting six (6) feet away. I am using public Internet access via a cellular Digi Connect WAN to the Ethernet (ENB) of the ControlLogix. Some of the public IP have "X" entered replacing the digits; you don't need to really know the exact IP value.

My computer has private IP = 10.9.92.1
01 01 ms 10.9.1.1 (Digi's private Intranet)
02 01 ms 10.10.11.10 (Digi's private Intranet)
03 01 ms 10.254.254.2 (Digi's private Intranet)
04 16 ms 66.77.x.x (Digi Co-Host/Internet Link)
05 04 ms 69.8.x.x (Digi Co-Host/Internet Link)
06 64 ms 66.77.x.x (Digi Co-Host/Internet Link)
07 09 ms min-core-02.inet.qwest.net [205.171.128.110]
08 11 ms cer-core-02.inet.qwest.net [67.14.8.18]
09 12 ms cer-brdr-01.inet.qwest.net [205.171.139.62]
10 39 ms qwest-gw.cgcil.ip.att.net [192.205.32.97]
11 35 ms tbr2.cgcil.ip.att.net [12.123.4.254]
12 35 ms tbr2.sl9mo.ip.att.net [12.122.10.46]
13 75 ms tbr2.attga.ip.att.net [12.122.10.137]
14 31 ms 12.122.85.157
15 34 ms 12.86.140.146
16 * Request timed out. (Part of Cellular Infra-Structure)
17 * Request timed out.
18 * Request timed out.
19 * Request timed out.
20 1276 ms mobile-166-XXX-XXX-XXX.mycingular.net [166.XXX.XXX.XXX]
Digi Connect WAN has private local IP = 192.168.196.80 (is 'gateway')
ControlLogix PLC has private local IP = 192.168.196.21

These traces always amaze me - how something so seemingly trivial takes so much effort to really function. Notice how my lab PC has to route through 6 devices to even get out of Digi's company network, then through Qwest (our ISP), through AT&T (my cellular SIM provider), through some unnamed hops of the cell system, and finally be port forwarded to the ControlLogix PLC. The packets may be passing through Minneapolis, Chicago, Detroit, Atlanta, and then finally returning to the PLC sitting right beside me.

Effect of NAT (Network Address Translation)
Now lets look at what happens when RSLinx on my PC opens an ODVA Ethernet/IP socket to the ControlLogix PLC. Every TCP/IP packet requires 4 unique values which define a connection:

1. Destination IP (target device)
2. Destination Port (target application within device)
3. Source IP (return address to originator)
4. Source Port (likely random port, originator is waiting for responses here)

So we start out with the 4-tuple DST=166.x.x.x : 44818 and SRC=10.9.92.1 : 22256. The 166.x.x.x IP is assigned by my cellular carrier. Port 44818 is ODVA's "well-known" port for Ethernet/IP. 10.9.92.1 is an internal Digi selected private IP. TCP port 22256 is the ephemeral (or random) port selected by RSLinx to listen for responses.

The first NAT effect is the Digi corporate firewall changes the request to be DST=166.x.x.x : 44818 and SRC=66.77.x.x : 22256. My private IP of 10.9.92.1 is meaningless out in the Qwest or AT&T's networks, so something needs to swap this for a "real" world-unique IP leased by Digi. Our corporate NAT interface creates a record (with a lifetime of 5 minutes) that allows any responses to be correctly restored to 10.9.92.1

The second NAT effect is when the Digi Connect WAN forwards to the ControlLogix with another private IP. So the 4-tuple now becomes DST=192.168.196.21 : 44818 and SRC=66.77.x.x : 22256. The ControlLogix thinks IP host 66.77.something is connected to it - not the real host IP of 10.9.92.1. Plus the ControlLogix has NOT CLUE that the RSLinx thinks the ControlLogix as IP of 166.something.

Now, to send a response the ControlLogix issues a TCP/IP packet with the flipped 4-tuple of DST=66.77.x.x : 22256 and SRC=192.168.196.21 : 44818. The Digi Connect WAN restores (undoes) the NAT and changes this to DST=66.77.x.x : 22256 and SRC=166.x.x.x : 44818. After passing back through AT&T and Qwest, Digi's corporate NAT interface restores its own NAT and changes it back to DST=10.9.92.1 : 22256 and SRC=166.x.x.x : 44818.

This understanding of NAT and IP is useful for understanding the capability and limitations of cellular access to certain devices with certain protocols. A future entry will cover setting up RSLinx Classic and using RSLogix 5000 to download over cellular to a L5555 processor.

Unlimited Cellular Data - Revisted

As I go around dealing with large utility customers and hear feedback from Digi salespeople who survive large accounts, I hear some interesting but disturbing trends rising.

First, there are the people (usually who are new to cellular) who claim any day now the age of cheap, unlimited cellular data plans means all my hard work to understand or offer reduced traffic are wasted effort. I especially hear this coming from European partners.

Then there are the other people ... people I know work with very large, very powerful end users who fail to get the cellular plans they desire. Things I hear:

• I have heard of customers who pilot projects and hear promises of unlimited traffic, but when the time comes to sign the contract for 3000+ sites, the carrier decides that they cannot offer unlimited traffic ... period. Hmm, I guess this is the difference showing between the carrier's commissioned sales staff and the business managers who need to keep the cellular system profitable.
• I have worked with large customers who do manage to get "unlimited deals" for a modest sized system - say 50 or 100 sites, but the carrier insists on adding the 2 clauses 1) the carrier has the right to artificially slow down the data communications (without detailing what this means) and 2) the carrier has the right to just stop all the customer's data traffic temporarily if the cell system gets busy (again, not details of this defined). Hmm, I have to kind of wonder what kind of control engineer agrees to such "clauses"? You'd think trying to get one's data under control to avoid the need for unlimited data is a wiser design.
• I have heard that overall the cellular carriers are starting to rethink the value of machine-to-machine data plans. Unlike DSL or cable, this is NOT an issue of bandwidth; it is an issue of the % of the time the device "hogs" 1 of N slots or resources on the tower forever. Imagine having 10 or 20 such devices squatting there, locking up that tower resource. So it is not even so much an issue of talking once per few minutes verse flat out unlimited traffic. In either case 1 of N finite tower resources is used, so long-term the only "good" data plan may be for a data system using the cellular resource every few hours or a few times a day.

So overall it looks like my efforts to understand and reduce traffic is useful.

I will give Webinar 31-May-2007

31-May-2007 (Thursday) I'll be giving a webinar related to outdoor use of IP-to-serial devices. While not directly related to my blog theme of "Industrial Protocols over IP", remote devices frequently are in bad locations, where extended temperature ranges, conformal coating, and Class 1 Div 2 certs are desired.

Title: Connect Local and Remote Devices in Hazardous Environment SCADA Applications
Register here: WebEx Link

Date: May 31, 2007
Time: 11:00am CDT (central zone)
Duration: 1 hour
Speakers:

• Lynn Linse - Engineer, Digi International
• Deb Smith - Business Development, Digi International

What you will learn
This live webinar will illustrate the value of using TCP/IP & UDP/IP based communication over wired and wireless networks to monitor and manage local and remote devices. Topics include:

• Overview of requirements for robust, remote, outdoor communication devices
• Discussion of extended temperature, conformal coating, hazardous certs
• Comparing features of IT-grade network devices to Digi's Haz product line
• Extending IP networks to remote serial and Ethernet devices
• Moving serial communications through TCP/IP and UDP/IP
• Migrating from analog Telco/POTS lines to IP-based broadband
• Real-world usage examples
• Implementation benefits
  - Real-time access to remote sites and process data without site visits
  - Reduce repair/replacement costs due to less-than ideal environments
  - Options to reduce installation costs with wireless

Please register today and join us May 31st for this informative webinar!
Questions? Contact Deb Smith at deb_smith@digi.com or 952-912-3283.

ODVA, Rockwell, and Modbus Get Smart

Summary: next week an amazing joining of technologies will take place at ODVA in Ann Arbor MI as ODVA, Rockwell, and Schneider-Electric discuss how to integrate Modbus servers (slaves) into CIP Producer/Consumer systems

One of the fun things about being involved in "multi-vendor" solutions is when you recognize moments of amazing sanity as they occur. One such moment of amazing sanity is occurring next week when ODVA (aka Rockwell / Allen-Bradley) and Modbus supporters (aka Schneider-Electric / Modicon / SquareD / Telemecanique) sit down to discuss how to integrate Modbus devices into the ODVA Ethernet/IP and CIP network systems. Of course there must be some interesting hidden politics behind this move - and I somewhat light-heartedly believe that perhaps French Schneider-Electric sees joining with the Americans (Rockwell/ODVA) as the lesser of two evils when compared to joining with the Germans (Siemens/PNO).

Check out: ODVA Call For Members: Modbus Integration JSIG The kick-off meeting for the Modbus JSIG runs from Thursday, May 10, 11:00 AM to Friday, May 11, 04:00 PM

Side-stepping the marketing fluff and platitudes of a brighter future such meetings evoke, small third-party suppliers and the folks on the plant floor can expect the following benefits. Regardless of the directly stated goals of ODVA, Rockwell, or Schneider-Electric, small vendors will implement solutions that include these abilities:

• Vendors making Ethernet Modbus/TCP products will have a simpler "first step" to adding full ODVA/CIP support without the somewhat overwhelming task of 100% conversion of a word-array device model into hundreds (or thousands) of CIP objects.
• ControlLogix PLC will be able to connect through Ethernet-to-Serial devices to multi-drops of Modbus/RTU slaves. For example a user with a dozen small Modbus/RTU PID loop controllers will be able to add an Ethernet-to-Serial device to read via Modbus and cyclically produce a small block of word data from each loop controller over Ethernet.
• HART, Bluetooth, ZigBee and other new technologies which offer Modbus interfaces will find a instant place as sensors and I/O within CIP and Rockwell systems.
• Since Siemens, GE-Fanuc, Omron, Mitsubishi and most major PLC brands offer some method to act as Modbus slaves, users with any of these PLC will be able to integrate them within the CIP Producer/Consumer system.
  

Looking forward a year or two, I also can foresee some other secondary benefits evolving from this JSIG's work:

• I started working with ODVA Ethernet/IP almost 8 years ago and still as-of today the legacy PLC5E, SLC5, and serial MicroLogix (the old PCCC-based PLC) don't have effective inclusion within CIP Producer/Consumer systems. Since the device model of PCCC PLC shares much in common with Modbus PLC, it is a very small enhancement to add a similar support for AB PLC - perhaps AB will actually extend this to future firmware updates to Ethernet-based PLC. In fact, since my Digi One IAP code already allows Modbus masters to query DF1 and CSPv4 slaves as-if Modbus slaves, as soon as Digi adds Ethernet-to-Modbus support per this JSIG's output users of older AB PLC will gain access to CIP Producer/Consumer systems indirectly as honorary Modbus slaves.
• Today legacy Modbus and Modbus/TCP systems lack any simple form of multicast producer/consumer exchange. While the IDA protocol offers this, IDA is so many orders of magnitude more complex (and resource hungry) than simple Modbus as to become really an "unrelated" protocol. Any specification that defines a "server interface" naturally implies a corresponding "client interface". So although this ODVA JSIG is not planning to define how Modbus "peers" could use multicast to exchange cyclic data, the end result will be a fairly natural and multi-vendor method to do this. So while I doubt many pure Modbus/TCP products would implement ODVA protocols just to gain this multicast exchange, any products which add the CIP support anyway will naturally add the last few bits of code required to enable pure Modbus-to-Modbus multicast via the ODVA mechanism.
• Taking the above point to its natural conclusion means Modbus/TCP masters which implement the Modbus JSIG's "server" function will also gain a mechanism to access CIP Producer/Consumer systems. Even if the ODVA JSIG doesn't cover how to do this, natural methods will be inferred, produced, and copied by vendors to make this a fairly common new product feature.

And lastly, it will interesting to see what response PNO considers to likewise include Modbus slaves more formally into PROFINET. Personally, I have always felt that the simpler PROFINET IO protocol would fit very naturally with multi-drops of Modbus slaves acting like plug in I/O modules ... each slave would be a module within the IO device.

Rockwell and Modbus Data Traffic

Summary: I compare the data costs for four common off-the-shelf PLC protocols to a remote cellular site. As a rule of thumb, even if you have an Ethernet-based PLC your lowest data costs for SCADA-style periodic polling are obtained using serial protocols via the PLC's serial port. Their modern "Ethernet protocols" are very wasteful of cellular bandwidth.

PLC Protocol Example:
A simple but realistic SCADA scenario is to poll every 15 minutes and read 10 words of data and write 2 words of data. This commonly requires 1 Read command and 1 Write command (I'll ignore the rarely supported Modbus command that reads & writes within a single command.)

While there exists special SCADA protocols and special products that optimize remote traffic, I am not looking at those protocols at the moment. Instead, since cellular and satellite access to remote IP and Ethernet products has enabled people to use off-the-shelf PLC technology, I am looking at the more traditional PLC protocols. These are things which affect users when they apply an Ethernet design to an IP-based wide-area-network system.

I compare these 4 PLC protocols:

• AB/DF1 Radio Modem (RM) encapsulated in UDP/IP. DF1 RM is basically DF1 Full-Duplex with no ACK/NAK and is supported by the SLC5 and MicroLogix line.
• Modbus/RTU encapsulated in UDP/IP. Modbus/TCP within UDP/IP is roughly the same size.
  
• AB/CSPv4 in TCP/IP as supported by SLC5/05 and PLC5E MSG blocks.
  
• AB/Ethernet/IP as moved by ControlLogix Explicit MSG blocks to PCCC-based remote PLC. Note that Ethernet/IP "I/O Messaging" does NOT work through NAT'd wide-area-networks since the protocol embeds IP information within the data packets and is thus is "broken" by NAT.

The bytes per 15 minutes includes the IP headers, any PLC protocol overhead, and the actual data moved for each update (always 20 + 4 bytes). The MB per month is just defined by 30 days of such polling, or 2880 updates at once per 15 minutes. The 2 serial protocol have been encapsulated into UDP/IP and I assume the remote IP end-point can handle connecting this to the PLC's serial port.

Protocol	Transport	Per 15 Min	MB per month	Relative Cost
Ethernet/IP	TCP/IP	1202 bytes	3.46 MB	100%
AB/CSPv4	TCP/IP	960 bytes	2.76 MB	80%
Modbus/RTU	UDP/IP	166 bytes	0.48 MB	14%
DF1 Radio Modem	UDP/IP	194 bytes	0.56 MB	16%

The two Rockwell "Ethernet" protocols cost a lot more to use in part because they force use of TCP/IP, and therefore suffer the repeated cost of TCP socket opening and closing, plus extra TCP acknowledgment overhead. They also suffer because they both involve connection registration and service functions that needlessly repeat every time the connection is reestablished. While the actual data packets of these protocols are roughly twice the size of the serial encapsulated protocols, the real burden they suffer is all the extra TCP/IP packets exchanged that do NOT directly involve field data update.

Both the serial Modbus/RTU and DF1 Radio Modem benefit that they move no IP packets that don't relate to the field data update - no TCP/IP open or close or acknowledgement; no protocol "service function" overhead. Each moves just 1 read request and 1 read response, plus 1 write request and 1 write response.

Discussion of Other PLC Protocols:

Most other PLC Ethernet protocols will either approach the costs of the AB/CSPv4 - or they won't work at all due to use of direct "Ether-Types" and lack of IP compatibility. Most serial protocols with roughly either match the 2 show here or be twice the cost if protocol ACKs are used by the protocol.

Modbus/ASCII will almost double the cost of Modbus/RTU since each data packet is roughly twice the size. But this wouldn't increase the IP overhead any.

Using DF1 Full-Duplex instead of Radio Modem would effectively double the cost over DF1 RM since DF1 Full-Duplex moves the protocol ACK/NAK, which doubles the IP header overhead also. Using DF1 Half-Duplex would triple or even quadruple the costs since HD not only moves protocol DF1 ACK/NAKs, but the ENQ/EOT polling overhead.

Most other protocols I am aware of - such as Omron Hostlink, GE-Fanuc SNPX, and Siemens PPI - would cost roughly 2 to 3 times more than Modbus/RTU or DF1 RM since they include protocol ACK, while a few even encode many parts of the message as ASCII or BCD form instead of as binary.

Rockwell PLC5E and SLC5 over Cellular

Summary: Customers ask me "How much will cellular cost" - this blog post walks through some examples of SCADA-style periodic polling of AB/PLC5E or SLC5/05 using the CSPv4 protocol (aka AB/Ethernet to 3rd parties) over TCP port 2222.

Real-World Numbers
For a simple SCADA-style example assume we need to read 10 words of data (20 bytes) and write 2 words (4 bytes) every time period. Obviously there would be simple optimizations to this, such as only writing data which changes or using PLC MSG blocks to push data from PLC to SCADA only when something changes. However my goal in this blog post isn't to "tweak" a solution to minimize cost, but to examine the protocol impact of using Rockwell CSPv4 over IP.

The table below shows the megabyte per month when polling once per second, per 5 seconds, per 1 minute, per 5 minutes, per 15 minutes, and per 1 hour. There are lots of variables considered ... and many more ignored. The traffic ranges from worst-case of 1005.0 MB for TCP/IP with larger header options polled once per second to best case of 0.2 MB for UDP/IP polled once per hour. This assumes the use of the CSPv4 submode 7, with local LSAP addressing and ignores that Rockwell PLC5E and SLC5/05 don't support CSPv4 within UDP/IP. Raw efficiency at moving the data bytes ranges for about 10% for UDP/IP to barely 1% for TCP/IP; which means most of what you are paying for is not related to actual, meaningful field data.

( Click this image to see a larger version )

(is at http://iatips.com/blogimage/rockwell_cspv4_traffic.png)

Notes on the Table

Since this example reads and writes small amounts of data, it assumes a SLC5-style Protected Typed Read with 3-Address Fields and the corresponding SLC5 write.

The smaller 40 byte TCP/IP header has no options attached; the larger 52-byte TCP/IP header includes the RFC 1323 Timestamp and Window Scale TCP options. These appear to be the normal default for Linux and easily becomes enabled under Windows since all applications share a single setting in the Registry.

The two time columns "15 min (Alive)" and "1 hr (Alive) assume a roughly 4 min 45 sec TCP keepalive to prevent the socket from closing. This reduces the traffic by the extra open/close overhead in exchange for billable TCP Keepalive packets. Keep in mind this ALSO requires the PLC to be properly configured to NOT close the idle sockets. By default, my SLC5/05 seems to close the idle connections in a few minutes.

The two time columns "15 min (Cls)" and "1 hr (Cls) assume the socket is closed after the a data polls, and the TCP socket and CSPv4 session must be reopened for teh next poll.

Discussion
Of course the standard costs of using TCP/IP verse UDP/IP apply:

• TCP/IP uses larger headers, ranging from 40 to 52 bytes per packet as compared to UDP/IP's smaller 28 byte of header.
• TCP/IP involves the TCP Acknowledgments, which may result in separate, billable 40 to 52 byte packets moving frequently without any meaningful field data.
• TCP/IP may require reopening a socket, costing 120 to 250 bytes per open, plus closing costing from 160 to 400 bytes. Exact sizes are hard to predict since both opening and closing of sockets tend to be "pushed" and result in excess retransmissions and retries when high network latency is true.
• TCP/IP over unknown 3rd party wide-area-network infrastructure requires at least 1 TCP packet to move every 4 minutes 45 seconds to maintain health. This means either a data packet or a TCP Keepalive with data.

It should be clear to see why using UDP/IP over cellular (which is very reliable) is much cheaper than using TCP/IP. There are no socket open, close, acknowledgment, or keepalive costs. Plus field experience has shown that rapid IP retries rarely succeed. For example, a customer polling with UDP every 5 minutes with 3 explicit retries if no answer will likely have the original poll succeed or that poll and all 3 retries fail. Again, cellular is very reliable in that packets almost always make it through unless there is a network or congestion issues and then only time (a few minutes) solves the problem. So such customers have abandoned retries and just ignore 1 failed 5 minute poll and "retry" in 5 minutes.

CSPv4 issues include:

• Rockwell PLC and software tools do NOT support use of UDP/IP - my tests with UDP/IP have to be conducted with the Digi One IAP which happily bridges CSPv4 between TCP and UDP (as well as to or from Ethernet/IP and DF1).
• CSPv4 requires the exchange of a pair of 28-byte negotiation TCP packets when a new TCP socket is opened to inform the client (master) of a server (slave) assigned session handle. This nearly doubles the overhead of an open-poll-close socket paradigm.
  
• The 28 byte CSPv4 header really contains little useful information; such excess bytes cost nothing tangible under Ethernet but cost cash in the form of requiring larger cell plans over cellar.

In conclusion, CSPv4 will be a rather poor choice for raw, periodic polling of remote AB PLC. ODVA Ethernet/IP (as implemented by all vendors) is even worse.

Your only effective solution at present is to carefully craft a set of MSG blocks to push data from the field in a report-by-exception paradigm. Of course you also must include safe guards within your PLC to prevent rapid, repeated MSG block triggers during system failure that could cost you thousands of dollar ($$$) in a few days.

DF1 Open Source for Visual Basic 2005

Archie has started a SourceForge project for his AB DF1 code running under VB 2005. He has checked in a full first set of files (unlike many SourceForge projects which get created but NEVER have files :-\ )

http://sourceforge.net/projects/abdf1/

I haven't looked over his code yet, plus all I have is VB 2003 .NET.

Hopefully Microsoft has STOPPED the old VB issue that each new rev of VB is neither 100% forward nor backward compatible ... one always need to "tweak" a few lines to make the port work. I've used VB 1.0, 3.0, 4.0, 5.0, 6.0 and now VB 2003 and none of these have liked old code being pulled forward.

Cellular to Allen-Bradley SLC5/05 on TCP 2222

The old CSPv4 protocol.

The Rockwell/AB SLC5/05 and PLC5E natively speak an older "unpublished" protocol named CSPv4, although most third party vendors call it either AB/Ethernet or AB/TCP. It moves only on TCP port 2222 - ODVA Ethernet/IP I/O Messaging is only on UDP port 2222, so they don't conflict. The protocol consists (normally) of a 3-part packet:

• 28-byte header
• 4 or 15-byte LSAP or end-point addressing packet
• PCCC message which is basically what DF1 documents as an Application Packet

In general, the packets are fairly sparse and compressible (if you have the tools to do this). The characteristics of CSPv4 which impact cellular (and wide-area-network) support:

• Rockwell tools and PLC only support use of TCP/IP and port 2222; this greatly limits use of CSPv4 in NAT'd networks since the remote NAT router can only forward TCP port 2222 to a single remote PLC.
• CSPv4 includes a single TCP packet exchange to "register a session" or connect. If you are polling faster than the PLC will hang-up on you, then this is not important. However, if you poll slow enough that a new TCP/IP socket must be opened for each poll, then even ignoring the TCP socket open/close overhead this nearly doubles your traffic costs.
• In tests, a SLC5/05 seems effective at including the TCP ACK response to the host within the CSPv4 data response packet, so you only have to pay for one empty TCP ACK, which is the host's acknowledgment to the PLC for the response.
• TCP Keepalive could be an issue, since most hosts fail to issue it and the SLC5/05 I've tested against either doesn't issue TCP keepalives
  or does it very frequently.

Interested in Cellular? Setup a DMZ Lab

Summary: Last post I suggested people interested in cellular data start by learning how to use their home cable router. In this post I suggest the next step, of how to make your life easy at work once you're ready to start testing real cellular or satellite access by IP.

Your Second Step should be to set up a simple, isolated low-speed broadband link at work ... create your own DMZ lab.

Sigh - I waste so much time listening to customers complain about how difficult it is to get the IT department to give them custom firewall permissions. Since modern "Security" wisdom is to block everything until proven safe, I waste more time asking customers complaining that their Modbus/TCP or Rockwell access not working to first talk to their IT group to make sure they aren't blocking unknown binary traffic by default. I waste yet more time when customers struggle for days and finally have to formally get someone in IT to help study the corporate firewall logs to see if any traffic is getting through or not. An interesting epiphany occurs when I suggest they just look into paying roughly $50 per month for a private connection for this. It is surprisingly cheap to do this and makes a lot of people's jobs 200% easier.

So far the feedback from customers has been quite positive, with IT departments over-joyed at the idea (slight exaggeration :-] lessor-of-two-evils may be a better term). This really makes sense; IT is charged with keeping the corporate system working and secure, so when you ask for yet another odd, unknown firewall hole to be opened, you ask them to risk their jobs. Plus trying to keep custom firewall settings updated for 50 different projects is an ongoing headache and ongoing risk for mistakes. I know that Digi's IT group is very satisfied with their policy of not offering custom firewall rules on the corporate LAN but instead helping teams set up such private connections in a safe, isolated manner.

Simple DMZ Lab Design
The simplest lab design is little more than a copy of what you have at home: a computer or two, an 8 or 24-port Ethernet switch, and a simple NAT router to "share" the internet connection with a dozen devices. This allows you to freely set up a few OPC servers and Master PLC to test access to remote cellular and other wide-area-network based systems.

• Locate an empty office or lab room for your new network. Perhaps your IT people should pull out or disable the corporate Ethernet in this room. You are going to create a small "DMZ"; a small isolated network that has limited security consequences if you goof up and let a hacker inside. You'll want good security tool installed on your Windows and Linux computer used in here.
• Arrange for a low-speed business broadband link with one fixed IP address. 256Kbps is more than enough for general PLC/SCADA testing and should cost in the range of $35 to $50 per month. Yes, just $35-50 per month! I had one customer forced to pay his IT department $100 per month to open one TCP/IP hole in the corporate firewall!! Gee, he could install 2 DSL links for that. Now, be patent when you talk to your carrier, as they are geared to sell the expensive primary access lines used for all corporate traffic including servers. Keep stressing that you want a low-speed secondary line for use with some network testing and eventually you'll locate the low-cost plans you want.
  
• Set up a DNS name for your DMZ lab. Online dynamic DNS providers support user-selected DNS names for static IP addresses. I use dyndns.org for both my dynamic and static IP but there are many out there. You won't need any form of DDNS update client since your IP never changes and you must enter the name manually anyway.
• Unless you plan to implement large VPN systems, just buy a nice consumer-grade DSL/Cable Router ... the same kind you use at home is fine. If you plan to set up a serious VPN infra-structure, then you'll just need to bite-the-bullet (& suffer the learning curve) of buying a commercial IT-grade router with VPN server capability built in. Be warned that while many consumer-grade routers mention "VPN Support", they are in fact sub-optimized and documented only for home-office users who connect into a corporate Windows or Cisco VPN server. Normal human beings will find them nearly impossible to set up for anything else!
  
• Do you want more than one public IP address? You need to pay a monthly surcharge which varies greatly per carrier, but could be in the range of $25 per month for 8 IP addresses instead of just 1. Plus you will need a larger IT-grade router since the consumer-grade routers won't support more than 1 public IP address. Most users won't need more than 1 IP address. However, having more than one IP address is helpful if more than 1 team shares the lab; this prevents them from trying to setup conflicting router configurations. Also, a few extra IP are helpful if you want to place a PLC "online" for your customers or sales force to access during customer-site demonstrations in the field.
  
• If you need to access your corporate network from your DMZ lab, then you need to arrange some rules with your IT people. Perhaps the rule is using a notebook computer with 802.11 wireless to the corporate network is Ok as long as the notebook is NEVER connected to the lab's Ethernet. Remember, since your lab has it's own public IP address you can even use FTP or a VPN client to connect "legally" out your corporate network and back into your lab via the Internet.
  

Fancier DMZ Lab Design
Since Digi is basically a "communication company", the lab I get to use is much fancier. It has 32 public IP addresses and even limited secure access from the corporate LAN. Of course I have to share this lab with other teams, so I'm not owner of 32 IP. As an example, here is how my lab is setup:

• Digi's IT group maintains a Cisco PIX router (a mid-range $800 model) that manages the 32 public IP addresses. Actually, this is NOT a complication since this router does not by default firewall any traffic; it merely distributes raw traffic based on public IP to one of many to internal IP addresses in an organized manner. I was lucky enough to get in the lab early and to be assigned 2 of the 32 public IP addresses.
  
• My first IP address receives 100% raw internet traffic at an internal static IP I selected; so an external IP such as 70.x.x.140 forwards to my internal IP of 192.168.20.159. Since the goal of the lab is to avoid burdening IT with TCP/UDP port forwarding chores, one could place a consumer-grade DSL/Cable router at this IP. Placing 2 routes in series is NOT a problem - do a net-trace of how you access www.google.com and you'll see a dozen or more routers in series. Instead of a pure hardware box I have a Ubuntu Linux machine running firewall and router tools at this IP. This is where I forward Modbus/TCP to one PLC and Rockwell Ethernet/IP to another PLC. I prefer the Linux box to the $39 hardware box because it gives me a richer view of traffic in and out, plus I can run an Ethernet sniffer such as WireShark to see a complete trace of the 2-way conversation taking place.
  
• For my second IP address I had Digi IT setup the PIX router to just forward a simple, safe list of Modbus, Rockwell, Digi, and other industrial protocol TCP and UDP ports. I normally have a Digi One IAP (an industrial-protocol aware Ethernet-to-serial device) at this IP address, but I can safely swap in a Windows machine when a test requires use of Windows tools.
  

Just as your home router does, the main Digi IT-supplied PIX router does out-going NAT to the internet and internal DHCP address assignment. So our DMZ lab has its own internal subnet of 192.168.20.x addresses. Any device in the DMZ lab can access the Internet - with or without going through my Linux firewall/router. Of course the other teams in the lab don't go through my router, but direct to the PIX router.

Since i study cellular usage, I have have a Digi Connect WAN providing an Ethernet-based cellular router in the DMZ lab. So I really have 3 potential routers to use - while it takes a bit of IP experience to not get confused, this allows me to have devices configured to selectively treat any 1 of the 3 routers as "the default gateway/router".

• For example, I can have a Master/Client device connect out to a cellular-based Slave/Server using a route such as Master => out PIX+DSL => in Cellular => Slave.
• For example, I can have a Master/Client device connect via cellular to a DSL-based Slave/Server using a route such as Master => out Cellular => in DSL+PIX => in Linux-Box => Slave.
• In both of this situations I can see BOTH ends of the conversation, which is a huge help in testing, timing, or troubleshooting new applications.
  

The bonus for having Digi's IT team involved is the PIX router also allows controlled, secure access into the DMZ lab from our corporate LAN. Technically, the PIX runs a set of rules similar to those used by the main corporate firewall out to the Internet and it just treats the 192.168.20.x subnet as a miniature internet. So I can sit at my desk and safely check on equipment running in the DMZ lab. Of course this is limited to equipment treating the PIX as the default router - if you understand IP routing you'll understand why, but at least it lets me log into my Ubuntu Linux router from desk.