The latest stable IET release, 0.4.15, suffers yet another misfeature: it will likely break all initiators when the ietd process is restarted (ietd restart, machine restart etc.).

This is because on ietd shutdown, the user space daemon is just killed, but it doesn’t have the corresponding signal handler and the kernel space module doesn’t perform any cleanups for it.

From initiator’s perspective, several things may happen:

• the change will happen so fast that the initiator won’t notice anything
• initiator will break the connection, but after reconnection, you will see your iSCSI drives remounted read-only, weird hangs etc.
• initiator won’t be able to connect again

Of course, no one wants to have the filesystem remounted read only, or to have the hard drive ripped off from a working system (this is how it looks from the perspective of the kernel if we loose a iSCSI connection).

There is a simple solution to that, although some may say it’s an ugly hack.

Using iptables to block the traffic to and from the IET target just before it shuts down does the job – the change is trivial – in /etc/init.d/iscsi-target, add something similar to the lines marked in bold:

ietd_start()
{
echo -n "Starting iSCSI enterprise target service: "
configure_memsize
modprobe -q crc32c
modprobe iscsi_trgt
start-stop-daemon --start --exec $DAEMON --quiet
RETVAL=$?
sleep 3s
iptables -D OUTPUT -p tcp --dport 3260 -j DROP
iptables -D INPUT -p tcp --dport 3260 -j DROP

if [ $RETVAL == "0" ]; then
echo “succeeded.”
else
echo “failed.”
fi
}

ietd_stop()
{
iptables -A OUTPUT -p tcp –dport 3260 -j DROP
iptables -A INPUT -p tcp –dport 3260 -j DROP
echo -n “Removing iSCSI enterprise target devices: ”
# ugly, but ietadm does not allways provides correct exit values
RETURN=`ietadm –op delete 2>&1`

With it, and with changes described in part 1, your IET iSCSI target installation should be really bullet-proof.

Or, perhaps, instead of using these hacks over and over again, it’s time to switch to more mature iSCSI-SCST?

See Solving reliability and scalability problems with iSCSI, part 2 article.

IET, or iSCSI Enterprise Target, is an “open source iSCSI target with professional features, that works well in enterprise environment under real workload, and is scalable and versatile enough to meet the challenge of future storage needs and developments”¹.

By default, Open-iSCSI initiators² are able to deal only with very brief disconnections.

They determine connectivity by sending iSCSI NOP-outs as pings every 10 seconds. If no response is received in 120 seconds, the connection is considered failed, and an error is returned to the SCSI layer. As a result, the SCSI layer offlines the device. We can compare it to a regular server with its hard disk removed during normal operation. In this case the in-memory data is not written to the disk, filesystems are not cleanly unmounted, while the server keeps running in a failing state.

120 seconds is definitely too short. Replacing cabling, failing switch, upgrading a SAN or simply human errors that interrupts the connection for longer than 2 minutes should not bring the datacenter to an unpleasant halt.

After a testing period, I determined that the initiator, when set up properly, can handle disconnections lasting several minutes, hours, or even days. Where previously, failing hardware or operator mistake would cause corrupted filesystems, perhaps damaged data, and a need to restart several servers manually, now, the iSCSI initiator will handle such situations graciously. Processes will just be in an uninterruptible sleep state, waiting for I/O operations to complete. Once the connection is re-established, processes would continue to work correctly.

However, moving that knowledge into production was harder than expected, and extremely hard to debug.

Even a short, few-seconds disconnection was causing a lot of trouble – devices offlined immediately, corrupted filesystems, and painful server restarts. Worse – there was no 100% way to reproduce it reliably – tcpdump, running IET daemon process with changed options, on different architectures, or in a debugger did not give any obvious hints.

In the end, after a lot of testing, the diagnose was clear:

• it’s not the initiator’s problem
• it does not depend on architecture (x86, x86_64, ARM)
• there need to be hundreds of connected initiators to reproduce it
• all these hundreds of initiators need to be connected, disconnected (because of the SAN restart, IET daemon process restart, cabling etc.) and then, connected again

Why does this phenomenon occur?

1. We can call it a DDoS on the IET daemon:

   when hundreds of initiators are disconnected, they try to send a iSCSI NOP-out pings every 10 seconds to the target, and finally, determine that the connection is broken. When the connection is available again, all these hundreds of initiators try to re-establish the connection at the same time.

2. A mathematical approach:

   with ~200 initiators, and NOP-out pings sent every 10 seconds, on average, it makes ~20 of them try to reconnect every second. It is an unrealistic scenario that everything is distributed evenly. In fact, tcpdump tests show 50-100 initiators trying to reconnect to the target at almost the same time.

The following patch solved the problem:

— ietd.c.orig 2006-12-18 06:54:01.000000000 +0100

+++ ietd.c 2007-06-17 18:20:49.000000000 +0200

@@ -29,7 +29,7 @@

#include “ietadm.h”

#define LISTEN_MAX 8

-#define INCOMING_MAX 32

+#define INCOMING_MAX 256

enum {

POLL_LISTEN,

In ietd.c, the size of incoming[] array is decided by hardcoded, compile-time INCOMING_MAX. This is the maximum number of incoming connections ietd can process in a login session. When the iSCSI connection is fully established and reaches its full feature phase, it is passed to the kernel and this resource is available again. So it is only needed to increase this number for “connection storm” cases, when suddenly every initiator tries to connect.

Lesson learnt: always abuse your test system in every way you can imagine.

1 http://iscsitarget.sf.net
2 http://www.open-iscsi.org