DRBD doesn’t apply any form of compression on the data that is replicated; is it a good idea to enable compression in the VPN link if you replicate your data with DRBD over Internet? Here is a quick test.

I used OpenVPN with its lzo compression to test it, as it is easy to set up and to get the results.

tun0 interface is the “compressed” link; it shows the amount of data sent between two DRBD servers.

bond0 is the real network interface; as the underlying tun0 link sends data in compressed form, this one shows the real traffic.

As you can see by looking at “total” column, OpenVPN lzo compression saves us more than 50% of traffic. Not bad. I wonder how does IPsec’s IPcomp compare to OpenVPN’s lzo?

tun0
day ......... rx ... | .... tx ... | ... total
---------------------+-------------+--------------
04.06.      2292 MB  |   85.50 MB  |    2377 MB
05.06.      2085 MB  |   80.69 MB  |    2166 MB
06.06.      2585 MB  |   91.83 MB  |    2677 MB
07.06.      2047 MB  |   80.20 MB  |    2128 MB
08.06.      2298 MB  |   85.70 MB  |    2384 MB
09.06.      2440 MB  |   90.20 MB  |    2530 MB
10.06.      2461 MB  |   90.54 MB  |    2551 MB


bond0
04.06.      1035 MB  |  145.26 MB  |    1180 MB
05.06.    936.28 MB  |  130.98 MB  |    1067 MB
06.06.      1452 MB  |  157.91 MB  |    1610 MB
07.06.    885.48 MB  |  134.84 MB  |    1020 MB
08.06.    992.08 MB  |  147.36 MB  |    1139 MB
09.06.      1064 MB  |  151.25 MB  |    1215 MB
10.06.      1143 MB  |  154.26 MB  |    1297 MB


What was tested: a couple of virtual machines (Linux on Xen and Windows on VMware) with their block devices placed on LVM; LVM was placed on DRBD.

DRBD replication was disabled between 8 am and 8 pm, so normally, we would see more traffic.

Note that “tx” for bond0 is bigger than “tx” for tun0; it is because DRBD server was exchanging data with other machines in LAN as well (backups etc.).

Compressing the link between two DRBD machines make sense over a WAN interface only (if you have to pay for the amount of data transferred, the link is too slow, or both); it doesn’t make much sense locally if you have 1 Gbit network or better.

If yes, you may have problems downloading photos from it - depending on your kernel and udev settings, or more generally, on Linux distribution used.

The tips from below may help you.

Every USB device connected to your Linux system will make the kernel print some messages. You can print them with dmesg command, started in the console, i.e.:


# dmesg -c
scsi 4:0:0:0: Direct-Access MATSHITA DMC-TZ5 0100 PQ: 0 ANSI: 2
usb-storage: device scan complete
Driver 'sd' needs updating - please use bus_type methods
sd 4:0:0:0: [sda] 7954431 512-byte hardware sectors (4073 MB)
sd 4:0:0:0: [sda] Write Protect is off
sd 4:0:0:0: [sda] Mode Sense: 04 00 00 00
sd 4:0:0:0: [sda] Assuming drive cache: write through
sd 4:0:0:0: [sda] 7954431 512-byte hardware sectors (4073 MB)
sd 4:0:0:0: [sda] Write Protect is off
sd 4:0:0:0: [sda] Mode Sense: 04 00 00 00
sd 4:0:0:0: [sda] Assuming drive cache: write through

The above output meant that the connected camera was recognized as /dev/sda block device - good, your distro supports it out of the box. If you never see a block device like sda or sdb, but only output like below - it means you need to complain to your distribution vendor:


# dmesg -c
usb 1-2: new high speed USB device using ehci_hcd and address 3
usb 1-2: configuration #1 chosen from 1 choice
scsi3 : SCSI emulation for USB Mass Storage devices
usb 1-2: New USB device found, idVendor=04da, idProduct=2372
usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 1-2: Product: DMC-TZ5
usb 1-2: Manufacturer: Panasonic
usb 1-2: SerialNumber: 0000000000000000005F0208220775
usb-storage: device found at 3
usb-storage: waiting for device to settle before scanning
scsi 3:0:0:0: Direct-Access MATSHITA DMC-TZ5 0100 PQ: 0 ANSI: 2
usb-storage: device scan complete

There is a simple workaround to that. In /etc/udev, search for such a line:

SUBSYSTEM=="usb", ACTION=="add", ENV{DEVTYPE}=="usb_device", NAME="bus/usb/$env{BUSNUM}/$env{DEVNUM}", MODE="0644"

You can do it with grep command:

# grep -r "bus/usb" /etc/udev

When you localize it, comment it out, restart udevd, i.e. use killall command to stop it, and start it in verbose mode to see more information on what it does:

# killall udevd
# udevd --verbose

For me, I had problems getting photos from DMZ-FZ20 and DMC-TZ5 Lumix models on Mandriva and OpenSuse. On Ubuntu though, everything worked from scratch and I didn’t have to do any udev changes.

Probably several times, and usually, you had to log in using either VNC or RDP (Remote Desktop Protocol - Microsoft Terminal Services). If you’re used to SSH, you may wonder why the overhead of a complete desktop is needed just to start a few text commands.
Of course, there are free and commercial SSH servers for Windows, but one problem with them is that they have to be installed separately. Which may be too much work for a one-off task now and then. Are there any other methods of accessing a remote Window text console from your Linux station?

But hey, here comes winexe to the rescue! Below, accessing Windows CLI from Linux remotely (using a Windows domain account):

From the project’s page, “winexe remotely executes commands on WindowsNT/2000/XP/2003 systems from GNU/Linux (probably also other Unices capable to compile Samba4)“.

Some notes:

• note that winexe doesn’t offer encryption of any kind - you may want to use VPN to make sure your connection is secure,
• if your distribution doesn’t offer a packaged version of winexe (most distributions don’t), winexe homepage offers a static binary; on the other hand if you have a recent Linux distribution, the static binary will most probably not work for you - in that case, just build the tool from source,
• you can use winexe in your Linux scripts - this way, you may execute certain tasks on your Linux machines, other tasks on your Windows machines - all that as a part of one bigger task / script,
• winexe can be seen as a Linux equivalent of psexec (a similar tool available for Windows).

HP, as every reputable hardware vendor, provides support for the products it sells. Its Linux-based HP Compaq t5735 Thin Client is no exception here:

So what happens if we choose the only supported operating system, which happens to be Debian GNU/Linux 4.0?

Well, unless you’re used to the fact that your hardware vendor’s domain is to offer you RPM or DEB packages for Windows products, it shouldn’t surprise you that HP offers only Windows executables as a way to support its Linux products:

Heck, have fun figuring out what to do with these instructions on your Linux distro:

INSTALLATION INSTRUCTIONS:
1. Download the SoftPaq .EXE file to a directory on your hard drive.

2. Execute the downloaded file and follow the on-screen instructions.

These spam messages - which want the user to visit a crafted Blogspot website - contain a small, but simple JavaScript hack. In order to see it, we either have to disable JavaScript in a browser, or, download the page with a tool which does not interpret the HTML code, for example, wget.

If you already have this crafted Blogspot page’s source opened in your favourite text editor, search for “<script language=”javascript”>”. In this tag, you will find such a string:

document.write(unescape(”<some_long_and_rather_human_unreadable_encoded_text>“));

Now we know what the spammers did to trick us to visit a page which promises to grow your manhood and chest in no time:

• they encoded some text first to make it unreadable for human eye or simple spam filters,
• they make the browser decode it with JavaScript unescape function,
• finally, they write it to the original HTML page using document.write.

But still, we don’t know why does it redirect us to a totally different page? An easy way to find out is to use a HTML/JavaScript encoder/decoder - the first link in Google should be just right for us: http://www.google.com/search?q=javascript+decoder. Follow the first link and paste the obfuscated text into the right box titled “Escaped Text/HTML/JavaScript”. As we can see, it decodes to this JavaScript code:

<script language=”javascript”>location.replace(”http://spammers.website.example.com”);</script>

Which just replaces the current Blogspot website (the one which was found in a spam email) with the spammer’s website specified in location.replace.

Very simple, isn’t it?

Unfortunately, Blogspot doesn’t seem to care or just is unable to do anything about it. Which might be the same.

Unfortunately, simple tests revealed serious performance problems when reading from a drive encrypted with dm-crypt / LUKS - here is why (with a simple solution).

First, let’s use “ondemand” governor - reads will be only about ~8 MB/s.

# echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# cat /proc/cpuinfo | grep MHz
cpu MHz : 349.991
cpu MHz : 349.991

# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 8.64089 seconds, 7.6 MB/s

During the test, the CPU was using the lowest frequency possible.
However, starting “cat /dev/zero | bzip2 >/dev/null” for a second or two was causing the CPU to switch to the full speed and fast reading.

Now, let’s use performance governor (or, full CPU speed) - we get almost 60 MB/s:

# echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

# cat /proc/cpuinfo | grep MHz
cpu MHz : 2799.930
cpu MHz : 2799.930

# echo 3 > /proc/sys/vm/drop_caches

# dd if=/dev/san2_data/test of=/dev/null bs=64k count=1000
1000+0 records in
1000+0 records out
65536000 bytes (66 MB) copied, 1.14821 seconds, 57.1 MB/s

Why is it happening? Does a bell ring somewhere?

“ondemand” CPUfreq governor has a few tunables, described in Documentation/cpu-freq. One of them is up_threshold:

up_threshold: defines what the average CPU usaged between the samplings
of 'sampling_rate' needs to be for the kernel to make a decision on
whether it should increase the frequency. For example when it is set
to its default value of '80' it means that between the checking
intervals the CPU needs to be on average more than 80% in use to then
decide that the CPU frequency needs to be increased.

While reading from that dm-crypt device, the following CPU usage was shown on yet another machine (the
numbers were consistent across a few tests):

187 MHz -> ~75% system CPU time in top
375 MHz -> ~66% system CPU time in top
562 MHz -> ~94% system CPU time in top
750 MHz -> ~85% system CPU time in top
937 MHz -> ~85% system CPU time in top
1125 MHz -> ~83% system CPU time in top
1312 MHz -> ~70% system CPU time in top
1500 MHz -> ~70% system CPU time in top

dm-crypt won’t eat the whole CPU power.

With default 80 as up_threshold for “ondemand” governor, no wonder the
frequency didn’t scale up.

So a “fix” for this problem was a matter of lowering up_threshold.
One more thing as a final note. Some CPUs don’t support lowering their voltage automatically, and the only way to lower CPU frequency is by using throttling. However, throttling won’t save any energy on an idle machine. It’s designed only to cool down a busy CPU by forcing it to HALT from time to time.

• it can compress data transparently,
• it provides wear levelling,
• it works with block devices.

In other words, a perfect filesystem to use for all flash-based devices, like USB-sticks.

Any filesystem should aim for high reliability, no doubt about it. But how to test it? Linux kernel provides a nice feature called “fault injection”. If Documentation/fault-injection/fault-injection.txt isn’t clear for you after the first read, here’s a quick help.

You need fault injection and debugfs compiled in the kernel.

First, mount debugfs:

mount debugfs /debug -t debugfs
mount -t logfs /dev/sdb1 /mnt/2

Then, set up fault injection:

cd /debug/fail_make_request
echo 10 > interval # interval
echo 100 > probability # 100% probability
echo -1 > times # how many times: -1 means no limit

Mount the filesystem on the device, start copying files etc. and then, make the device fail:

echo 1 > /sys/block/sdb/sdb1/make-it-fail

Logfs was working for some time, but eventually, it failed - we can blame a proprietary nvidia module which tainted the kernel, of course

FAULT_INJECTION: forcing a failure
[<c0104d19>] show_trace_log_lvl+0×1a/0×2f
[<c01057b1>] show_trace+0×12/0×14
[<c0105837>] dump_stack+0×15/0×17
[<c01e87d8>] should_fail+0×113/0×144
[<c01d40f2>] generic_make_request+0×167/0×2c0
[<c01d6142>] submit_bio+0xd7/0xdf
[<c01837e6>] submit_bh+0xbc/0xd9
[<c0185e84>] block_read_full_page+0×26c/0×27c
[<c0187823>] blkdev_readpage+0xf/0×11
[<c014aecb>] read_cache_page_async+0×96/0×129
[<c014c903>] read_cache_page+0×12/0×42
[<c01c3aca>] bdev_read+0×5d/0xcd
[<c01bd9f8>] scan_segment+0×16e/0×331
[<c01bdbe8>] logfs_scan_pass+0×2d/0×71
[<c01bdce9>] logfs_gc_pass+0×44/0×3f3
[<c01c07a0>] logfs_get_wblocks+0×21/0×2f
[<c01c1751>] logfs_write_buf+0×22/0×30b
[<c01bd790>] logfs_commit_write+0xed/0×101
[<c014c11f>] generic_file_buffered_write+0×3a9/0×522
[<c014c6d3>] __generic_file_aio_write_nolock+0×43b/0×471
[<c014c76d>] generic_file_aio_write+0×64/0xc1
[<c01672f6>] do_sync_write+0xc4/0×101
[<c0167af5>] vfs_write+0xaf/0×138
[<c0167fe2>] sys_write+0×3d/0×61
[<c0103da2>] sysenter_past_esp+0×5f/0×99
=======================
Buffer I/O error on device sdb1, logical block 35497
————[ cut here ]————
kernel BUG at fs/logfs/gc.c:88!
invalid opcode: 0000 [#1]
PREEMPT
Modules linked in: iptable_nat nf_nat ipt_ULOG ipt_recent nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables capability usblp commoncap loop dm_mod video thermal sbs fan container dock battery ac floppy cpufreq_conservative cpufreq_powersave processor pcspkr kqemu af_packet usb_storage snd_pcm_oss snd_mixer_oss snd_via82xx snd_ac97_codec ac97_bus snd_pcm ehci_hcd snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore nvidia(P) uhci_hcd evdev i2c_viapro via_agp agpgart i2c_core usbcore 8139cp via_rhine 8139too sr_mod mii tsdev sg cdrom
CPU: 0
EIP: 0060:[<c01bd9fc>] Tainted: P VLI
EFLAGS: 00010282 (2.6.22.5-1 #2)
EIP is at scan_segment+0×172/0×331
eax: fffffffb ebx: c02cfa98 ecx: c1809d80 edx: 00000000
esi: 08aa0000 edi: c749fbdc ebp: c749fc1c esp: c749fb8c
ds: 007b es: 007b fs: 0000 gs: 0033 ss: 0068
Process rsync (pid: 3940, ti=c749e000 task=d79574e0 task.ti=c749e000)
Stack: 0000001c c749fbdc 0000031a 00000000 0000899c f695dff4 00000013 00000000
00000455 dd4c1c00 00020000 00000000 022cfa98 00009114 00009114 00000000
00000455 00000000 0000101c d7e7c800 5ccc4658 00040010 00000000 13000000
Call Trace:
[<c0104d19>] show_trace_log_lvl+0×1a/0×2f
[<c0104dc9>] show_stack_log_lvl+0×9b/0xa3
[<c0104fa8>] show_registers+0×1d7/0×30c
[<c01051db>] die+0xfe/0×1d6
[<c02bddd7>] do_trap+0×89/0xa2
[<c0105605>] do_invalid_op+0×88/0×92
[<c02bdbb2>] error_code+0×6a/0×70
[<c01bdbe8>] logfs_scan_pass+0×2d/0×71
[<c01bdce9>] logfs_gc_pass+0×44/0×3f3
[<c01c07a0>] logfs_get_wblocks+0×21/0×2f
[<c01c1751>] logfs_write_buf+0×22/0×30b
[<c01bd790>] logfs_commit_write+0xed/0×101
[<c014c11f>] generic_file_buffered_write+0×3a9/0×522
[<c014c6d3>] __generic_file_aio_write_nolock+0×43b/0×471
[<c014c76d>] generic_file_aio_write+0×64/0xc1
[<c01672f6>] do_sync_write+0xc4/0×101
[<c0167af5>] vfs_write+0xaf/0×138
[<c0167fe2>] sys_write+0×3d/0×61
[<c0103da2>] sysenter_past_esp+0×5f/0×99
=======================
Code: 00 00 00 0f a5 f7 d3 e6 f6 c1 20 0f 45 fe 0f 45 f0 8b 45 94 89 f9 89 f2 8d 7d c0 03 55 a8 13 4d ac 89 7c 24 04 ff 13 85 c0 74 04 <0f> 0b eb fe 8d 45 c0 b9 1c 00 00 00 ba ff 00 00 00 e8 82 55 00
EIP: [<c01bd9fc>] scan_segment+0×172/0×331 SS:ESP 0068:c749fb8c

Hopefully, the bug is not there anymore, as I notified the logfs’ author, Jörn Engel. He also fixed some other bugs I found.

I wonder if and how easy other filesystems can be abused with the fault injection?

fsid=num
This option forces the filesystem identification portion of the file handle and file attributes used on the wire to be num instead of a number derived from the major and minor number of the block device on which the filesystem is mounted. Any 32 bit number can be used, but it must be unique amongst all the exported filesystems.

This can be useful for NFS failover, to ensure that both servers of the failover pair use the same NFS file handles for the shared filesystem thus avoiding stale file handles after failover.

Some Linux filesystems are not mounted on a block device; exporting these via NFS requires the use of the fsid option (although that may still not be enough).

Obvious, and self-explanatory, isn’t it? If you happened to have this “Stale NFS file handle” problem and your exported NFS file systems were on a LVM volume, adding fsid for each exported filesystem should help:

/nfs4exports 192.168.18.129/26(ro,sync,insecure,no_root_squash,no_subtree_check,fsid=0)
/nfs4exports/vmware-data 192.168.18.129/26(rw,nohide,sync,insecure,no_root_squash,no_subtree_check,fsid=1)
/nfs4exports/xen-config 192.168.18.129/26(rw,nohide,sync,insecure,no_root_squash,no_subtree_check,fsid=2)

Note: if this error only happens when accessing certain files, this means that your (non-network) filesystem is broken:

# rm fCVS
rm: cannot remove `fCVS': Stale NFS file handle

In this case, running fsck is recommended.

The latest stable IET release, 0.4.15, suffers yet another misfeature: it will likely break all initiators when the ietd process is restarted (ietd restart, machine restart etc.).

This is because on ietd shutdown, the user space daemon is just killed, but it doesn’t have the corresponding signal handler and the kernel space module doesn’t perform any cleanups for it.

From initiator’s perspective, several things may happen:

• the change will happen so fast that the initiator won’t notice anything
• initiator will break the connection, but after reconnection, you will see your iSCSI drives remounted read-only, weird hangs etc.
• initiator won’t be able to connect again

Of course, no one wants to have the filesystem remounted read only, or to have the hard drive ripped off from a working system (this is how it looks from the perspective of the kernel if we loose a iSCSI connection).

There is a simple solution to that, although some may say it’s an ugly hack.

Using iptables to block the traffic to and from the IET target just before it shuts down does the job - the change is trivial - in /etc/init.d/iscsi-target, add something similar to the lines marked in bold:

ietd_start()
{
echo -n "Starting iSCSI enterprise target service: "
configure_memsize
modprobe -q crc32c
modprobe iscsi_trgt
start-stop-daemon --start --exec $DAEMON --quiet
RETVAL=$?
sleep 3s
iptables -D OUTPUT -p tcp –dport 3260 -j DROP
iptables -D INPUT -p tcp –dport 3260 -j DROP

if [ $RETVAL == "0" ]; then
echo “succeeded.”
else
echo “failed.”
fi
}

ietd_stop()
{
iptables -A OUTPUT -p tcp –dport 3260 -j DROP
iptables -A INPUT -p tcp –dport 3260 -j DROP
echo -n “Removing iSCSI enterprise target devices: ”
# ugly, but ietadm does not allways provides correct exit values
RETURN=`ietadm –op delete 2>&1`

With it, and with changes described in part 1, your IET iSCSI target installation should be really bullet-proof.

Or, perhaps, instead of using these hacks over and over again, it’s time to switch to more mature iSCSI-SCST?

# diff -u full/2007-09-25-full.sql incr/incr.sql
diff: memory exhausted

After some googling and looking through various blogs and articles, it appeared to me that GNU diff just works like that - wants to load both files in memory. So, perhaps adding some more space would help?

# dd if=/dev/zero of=/swapfile bs=1024 count=5242800
# mkswap /swapfile
# swapon /swapfile

Indeed, it helped - I was able to make a diff of these two ~800 MB files. However, as soon as I tried to make a diff of ~1 GB files, diff again exited with “memory exhausted” message.

I tried a number of tools, but none of them was giving the output I wanted (similar to that of diff -u).

As I needed that diff for backup only, in the end, I use xdelta tool:

# xdelta delta -9 full/2007-09-25-full.sql incr/incr.sql 2007-09-25-sql.delta

It gives a binary delta file, but is able to produce it using much less memory than diff.

Ooh, and if your files are so big that xdelta is failing for you…

# xdelta delta -9 full/2008-06-01-full.sql incr/incr.sql 2008-06-02-incr.delta.temp
xdelta: mmap failed: Cannot allocate memory

…you may consider using xdelta3. The deltas it produces are a big bigger, but it works much better (read: does not fail) with large files. Syntax:

# xdelta3 -9 -f -e -s full/2008-06-01-full.sql incr/incr.sql 2008-06-02-incr.delta.temp

Still, it would be great to have a tool which could produce diff -u -style output for large files.