ERROR: invalid page header in block 13760 of relation base/16995/67484


or that, when doing your database dump:


pg_dump: SQL command failed
pg_dump: Error message from server: ERROR: invalid page header in block 11947 of relation base/16995/17062
pg_dump: The command was: COPY public.samaged_table (column1, column2, column3, ...) TO stdout;


it is likely your PostgreSQL server has hardware issues or has crashed (or, the VM host, as in my case). If it’s the case, pg_dump / pg_dumpall will error and exit when seeing a corrupted block.

Unfortunately, it mean’s you’ll loose some data, so depending on what you have in your database, you may resort to some other methods.

Here is a quick way to fix it:


database=# SET zero_damaged_pages = on;
SET
database=# VACUUM FULL damaged_table;
WARNING: invalid page header in block 13748 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13749 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13757 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13758 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13759 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13760 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13782 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697831 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
WARNING: invalid page header in block 13762 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13816 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13817 of relation base/16995/67484; zeroing out page
WARNING: invalid page header in block 13818 of relation base/16995/67484; zeroing out page
(...)
WARNING: index "damaged_table_site_id" contains 14697498 row versions, but table contains 14709258 row versions
HINT: Rebuild the index with REINDEX.
VACUUM

database=# REINDEX TABLE damaged_table;


Do a database dump after that to make sure it’s no longer corrupted (pg_dump / pg_dumpall should no longer complain).

First, you need to create a /var/cpanel/mainips/ directory, if it doesn’t exist:

mkdir /var/cpanel/mainips/

Then, create a /var/cpanel/mainips/root file, with the following content:

1.2.3.4
3.4.5.6

Basically, each line is an additional shared IP in WHM. That’s it!

# hpacucli ctrl all show

Error: No controllers detected.

Here is how to solve it.

If your RAID controller is similar to the one below (I’ve had it in ProLiant DL180 G6):

06:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)

 Subsystem: Hewlett-Packard Company Smart Array P410

 Flags: bus master, fast devsel, latency 0, IRQ 24

 Memory at fbc00000 (64-bit, non-prefetchable) [size=2M]

 Memory at fbbff000 (64-bit, non-prefetchable) [size=4K]

 I/O ports at d800 [size=256]

 Expansion ROM at fbb00000 [disabled] [size=512K]

 Capabilities: [40] Power Management version 3

 Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+

 Capabilities: [70] Express Endpoint, MSI 00

 Capabilities: [ac] MSI-X: Enable+ Count=16 Masked-

 Capabilities: [100] Advanced Error Reporting

 Kernel driver in use: hpsa

 Kernel modules: hpsa

you may want to use these instructions:

• download and compile uname26 tool: http://mirror.linux.org.au/linux/kernel/people/ak/uname26/
• load sg module and try to run hpacucli again as below:

# modprobe sg
# ./uname26 hpacucli ctrl all show config

[root@hpg6.starhub.sg ~]# ./uname26 hpacucli ctrl all show config 

Smart Array P410 in Slot 1                (sn: PACCRID11251VYT)

 array A (SAS, Unused Space: 0 MB)

 logicaldrive 1 (136.7 GB, RAID 1, OK)

 physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
 physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

 array B (SAS, Unused Space: 0 MB)

 logicaldrive 2 (1.6 TB, RAID 5, OK)

 physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 450 GB, OK)
 physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SAS, 450 GB, OK)
 physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SAS, 450 GB, OK)
 physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SAS, 450 GB, OK)
 physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SAS, 450 GB, OK)

 Expander 250 (WWID: 50014380079C1660, Port: 1I, Box: 1)

 Enclosure SEP (Vendor ID HP, Model DL18xG6BP) 248 (WWID: 50014380079C1673, Port: 1I, Box: 1)

 SEP (Vendor ID PMCSIERA, Model  SRC 8x6G) 249 (WWID: 5001438016827A8F)

To create the floppy image containing bnx2 firmware:


wget http://http.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.34_all.deb
dd bs=1024 count=1440 if=/dev/zero of=imagefile.img
mkfs.msdos imagefile.img
mkdir /mnt/floppy
mount -o loop imagefile.img /mnt/floppy
cp firmware-bnx2_0.34_all.deb /mnt/floppy/
umount /mnt/floppy


And, that’s it! Add the floppy via virtual media in DRAC, boot the server, it should install fine this time.

Note that when you connect a floppy image via DRAC, it will briefly disconnect your virtual CD-ROM, and the installation will fail – so make sure you do it before the system boots (or after all data was read from the CD, i.e. when it asks for the image, it is fine – but not before).

IPv6 addrconf: prefix with wrong length 56
IPv6 addrconf: prefix with wrong length 56
IPv6 addrconf: prefix with wrong length 56

Here is what you can do to get rid of it.

Add the following to /etc/sysctl.conf and run sysctl -p:

# gets rid of "IPv6 addrconf: prefix with wrong length 56" in dmesg
net.ipv6.conf.eth0.autoconf=0
net.ipv6.conf.eth1.autoconf=0
net.ipv6.conf.all.autoconf=0
net.ipv6.conf.default.autoconf=0

Dear Mr. Chmielewski,

thank you for your support request.

The Acer Aspire One 751 is only avaliable with a Microsoft Windows licence.

As a matter of principle the return of your licence is possible. The following criteria have to be met:

1.) The licence agreements, which appeared when you first started the system, MUSTN'T be accepted.

2.) The item has to be sent in to our Service Center including all accessoires and equipement. If you already created recovery discs it is utmost important that these will also be sent in with the item. No copy of the former deliverred operating system is allowed to remain in your property / hands.

3.) The ACER service is then going to remove the Windows lincence plate and will erase the whole harddisc.

4.) You then will receive a refund (by bank transfer) for the preinstalled OEM licence (about 30 Euro) .

5.) You have to pay the shipping cost to the acer service center and the shipping costs back to your address.

If you wish to return the licence we need the serial number of your notebook, your contact details including the shipping address and your bank details.

We will then provide you with a reference number with which the item can be dispatched to our Service Center.

Please follow the instructions below before dispatching the item:

- Please note the reference number clearly visible on the outer package / cardboard box.
- If possible pack the item in the original packaging or use another package that is safe for transport.
- Please note that ACER will not accept items that arrive being dispatched "freight collect".

ACER Computer GmbH
Repair Center
Kornkamp 4
22926 Ahrenburg

If you have any further questions regarding this topic please do not delete the previous correspondence.

Best regards
C. Riecken

Your ACER support team

• running memtest86+ means server downtime,
• it may not be possible to run memtest86+ on a remote server without KVM-IP, iLO or similar access.

What to do in such situations? User-space memtester to the rescue!

The obvious disadvantage is that you will not be able to test all RAM in the server, but sometimes, it’s not needed to determine if RAM is good or not.

The below command will try to allocate ~15GB RAM and make one test pass – if you see errors like below, your RAM is hosed!

Note that with the exact command line shown below you should have at least 15 GB of free, unallocated memory, otherwise, you’re likely to hang your server, or at least cause a serious downtime or OOM-killer.

# memtester 15000 1
memtester version 4.0.8 (64-bit)
Copyright (C) 2007 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 15000MB (15728640000 bytes)
got  15000MB (15728640000 bytes), trying mlock ...locked.
Loop 1/1:
  Stuck Address       : testing   0FAILURE: possible bad address line at offset 0x5d922ec8.
Skipping to next test...
  Random Value        : ok
FAILURE: 0x7f40f12dd82264da != 0x7f40f52dd82264da at offset 0x22fa2d27.
FAILURE: 0xe800dc254863429d != 0xe8005c254863429d at offset 0x22fa2de7.
FAILURE: 0x95005ca5bc351fd0 != 0x95007ca5bc351fd0 at offset 0x22fa2e47.
FAILURE: 0x16cc8129822daa5 != 0x16c4a129822daa5 at offset 0x22fa2e5f.
FAILURE: 0xce8945572424d79a != 0xce89c5572424d79a at offset 0x22fa2f1f.
FAILURE: 0x14054d669820e595 != 0x1405cd669820e595 at offset 0x22fa306f.
FAILURE: 0x8d283c130c69dc25 != 0x8d28bc130c69dc25 at offset 0x22fa309f.
FAILURE: 0x19285abae23f881 != 0x19287abae23f881 at offset 0x22fa30b7.
FAILURE: 0x55302b281ab13a51 != 0x55302f281ab13a51 at offset 0x22fa31bf.
FAILURE: 0x48b8ae1715e5084b != 0x48b8aa1715e5084b at offset 0x22fa31d7.
FAILURE: 0x28a2b64b9039074a != 0x28a2164b9039074a at offset 0x22fa5db7.
FAILURE: 0xc6080cfa18200698 != 0xc6080efa18200698 at offset 0x22fa5dcf.
FAILURE: 0x2b30fefcd69a347 != 0x2b38fefcd69a347 at offset 0x22fa5de7.
FAILURE: 0xbc27b13358294271 != 0xbc27b33358294271 at offset 0x22fa5e2f.
FAILURE: 0xc17e0d24dc21fe7e != 0xc17e0f24dc21fe7e at offset 0x22fa5e47.
FAILURE: 0x29ac181590911ae3 != 0x29ac1a1590911ae3 at offset 0x22fa5f07.
FAILURE: 0x233001ba38a53ccd != 0x233021ba38a53ccd at offset 0x22fa5f37.
FAILURE: 0x7542cd4b38a3ea9f != 0x7542c94b38a3ea9f at offset 0x22fa5f7f.
FAILURE: 0x86c89d49992151c8 != 0x86c8bd49992151c8 at offset 0x22fa5ff7.
FAILURE: 0x16208f9b584969b3 != 0x16200f9b584969b3 at offset 0x22fa609f.
FAILURE: 0x6a08171eb0a18751 != 0x6a08151eb0a18751 at offset 0x22fa60b7.
FAILURE: 0x8214f12e9a254014 != 0x8214f72e9a254014 at offset 0x22fa60cf.
FAILURE: 0x8005ab68021ad9f != 0x800dab68021ad9f at offset 0x22fa612f.
FAILURE: 0x884e76469ba12421 != 0x884e72469ba12421 at offset 0x22fa61a7.
FAILURE: 0x30509c65d890299e != 0x30509e65d890299e at offset 0x22fa62af.
  Compare XOR         : FAILURE: 0x3f81821efe3ced18 != 0x3f81861efe3ced18 at offset 0x22fa2d27.
FAILURE: 0xa8416d166e7dcadb != 0xa840ed166e7dcadb at offset 0x22fa2de7.
FAILURE: 0x5540ed96e24fa80e != 0x55410d96e24fa80e at offset 0x22fa2e47.
FAILURE: 0xc1ad5903be3d62e3 != 0xc1acdb03be3d62e3 at offset 0x22fa2e5f.
FAILURE: 0x8ec9d6484a3f5fd8 != 0x8eca56484a3f5fd8 at offset 0x22fa2f1f.
FAILURE: 0xd445de57be3b6dd3 != 0xd4465e57be3b6dd3 at offset 0x22fa306f.
FAILURE: 0x4d68cd0432846463 != 0x4d694d0432846463 at offset 0x22fa309f.
FAILURE: 0xc1d3169cd43e80bf != 0xc1d3189cd43e80bf at offset 0x22fa30b7.
FAILURE: 0x1570bc1940cbc28f != 0x1570c01940cbc28f at offset 0x22fa31bf.
FAILURE: 0x8f93f083bff9089 != 0x8f93b083bff9089 at offset 0x22fa31d7.
FAILURE: 0xe8e3473cb6538f88 != 0xe8e2a73cb6538f88 at offset 0x22fa5db7.
FAILURE: 0x86489deb3e3a8ed6 != 0x86489feb3e3a8ed6 at offset 0x22fa5dcf.
FAILURE: 0xc2f3a0e0f3842b85 != 0xc2f420e0f3842b85 at offset 0x22fa5de7.
FAILURE: 0x7c6842247e43caaf != 0x7c6844247e43caaf at offset 0x22fa5e2f.
FAILURE: 0x81be9e16023c86bc != 0x81bea016023c86bc at offset 0x22fa5e47.
FAILURE: 0xe9eca906b6aba321 != 0xe9ecab06b6aba321 at offset 0x22fa5f07.
FAILURE: 0xe37092ab5ebfc50b != 0xe370b2ab5ebfc50b at offset 0x22fa5f37.
FAILURE: 0x35835e3c5ebe72dd != 0x35835a3c5ebe72dd at offset 0x22fa5f7f.
FAILURE: 0x47092e3abf3bda06 != 0x47094e3abf3bda06 at offset 0x22fa5ff7.
FAILURE: 0xd661208c7e63f1f1 != 0xd660a08c7e63f1f1 at offset 0x22fa609f.
FAILURE: 0x2a48a80fd6bc0f8f != 0x2a48a60fd6bc0f8f at offset 0x22fa60b7.
FAILURE: 0x4255821fc03fc852 != 0x4255881fc03fc852 at offset 0x22fa60cf.
FAILURE: 0xc840eba7a63c35dd != 0xc8416ba7a63c35dd at offset 0x22fa612f.
FAILURE: 0x488f0737c1bbac5f != 0x488f0337c1bbac5f at offset 0x22fa61a7.
FAILURE: 0xf0912d56feaab1dc != 0xf0912f56feaab1dc at offset 0x22fa62af.
  Compare SUB         : FAILURE: 0x770f6d7d59b31bc0 != 0x64680d7d59b31bc0 at offset 0x22fa2d27.
FAILURE: 0x17cc5d24cac64438 != 0x6cb85d24cac64438 at offset 0x22fa2de7.
FAILURE: 0x8bda06d04720f630 != 0xf69f06d04720f630 at offset 0x22fa2e47.
FAILURE: 0x191ec89353a0b578 != 0xe4b7189353a0b578 at offset 0x22fa2e5f.
FAILURE: 0xb1bc42b59ee189c0 != 0x5cd042b59ee189c0 at offset 0x22fa2f1f.
FAILURE: 0x86575304584f0af8 != 0x316b5304584f0af8 at offset 0x22fa306f.
FAILURE: 0x4d053942f1b9f178 != 0xf8193942f1b9f178 at offset 0x22fa309f.
FAILURE: 0x206d80c97f8947d8 != 0x9719d0c97f8947d8 at offset 0x22fa30b7.
FAILURE: 0x6cf332350ca77058 != 0x5a4bd2350ca77058 at offset 0x22fa31bf.
FAILURE: 0xcae418d26b8f9b68 != 0xdd8b78d26b8f9b68 at offset 0x22fa31d7.
FAILURE: 0xdc58887f8f9e1d40 != 0xc67f887f8f9e1d40 at offset 0x22fa5db7.
FAILURE: 0x66470a0f89743570 != 0xdcf35a0f89743570 at offset 0x22fa5dcf.
FAILURE: 0x678cf2f8066c7ac8 != 0x12a0f2f8066c7ac8 at offset 0x22fa5de7.
FAILURE: 0x7e705d8189837558 != 0xf51cad8189837558 at offset 0x22fa5e2f.
FAILURE: 0x62530ceb060c3560 != 0xd8ff5ceb060c3560 at offset 0x22fa5e47.
FAILURE: 0xe1f0dc5c71b99328 != 0x589d2c5c71b99328 at offset 0x22fa5f07.
FAILURE: 0xa1a2ad6a30317bb8 != 0xc67ad6a30317bb8 at offset 0x22fa5f37.
FAILURE: 0x1d86ded3b0c73088 != 0x302e3ed3b0c73088 at offset 0x22fa5f7f.
FAILURE: 0xf1b9a3e0fcfa14f0 != 0x5c7ea3e0fcfa14f0 at offset 0x22fa5ff7.
FAILURE: 0x806d7d4232efc3a8 != 0xd5597d4232efc3a8 at offset 0x22fa609f.
FAILURE: 0x98a6f4d473117858 != 0x21faa4d473117858 at offset 0x22fa60b7.
FAILURE: 0x617c86f84528d8d0 != 0xc58176f84528d8d0 at offset 0x22fa60cf.
FAILURE: 0xbca540978cefa888 != 0x67b940978cefa888 at offset 0x22fa612f.
FAILURE: 0xe6ebd2a5a59fd8d8 != 0xf99332a5a59fd8d8 at offset 0x22fa61a7.
FAILURE: 0x1a30be8e9a1fb260 != 0x90dd0e8e9a1fb260 at offset 0x22fa62af.
  Compare MUL         : ^C^C^C^C^C
#

“script” command can be used to log user activity. Then, we can send the session as email to several recipients, and remove the log file.

Add this to user’s .bash_profile file, or simply construct something similar:

CURDATE=$(date +%F-%T)
RAND=$RANDOM
 
EMAILS="user@example.com otherrecipient@example.com"
 
script -f -q /tmp/session-$USER-$CURDATE-$RAND.log
for EMAIL in $EMAILS; do
    cat "/tmp/session-$USER-$CURDATE-$RAND.log" | mail -s "SSH session transcript for $USER at $CURDATE" $EMAIL
done
rm -f /tmp/session-$USER-$CURDATE-$RAND.log
exit

Notes:

• don’t assume it to be any security feature: the log file can be easily manipulated or removed by the user, a different shell can be used, etc.,
• sending an email may not be a good idea if you expect lots of output (i.e. cat /dev/urandom).

• First, download proftpd-dfsg_1.3.3a.orig.tar.gz and proftpd-dfsg_1.3.3a-3.diff.gz (or later) from http://ftp.us.debian.org/debian/pool/main/p/proftpd-dfsg/. Uncompress it, apply the diff file:

tar xpf proftpd-dfsg_1.3.3a.orig.tar.gz
patch -p0 < proftpd-dfsg_1.3.3a-3.diff

• For Debian Lenny, you will have to edit debian/control file (directory inside proftpd-dfsg-1.3.3a after you apply the diff) – at the beginning of this file, replace libssl-dev (>= 0.9.8l) with just libssl-dev, with no version remarks.
• If you still don’t have a build environment installed, do so now:

apt-get install build-essential

• You will still have to install some packages to make proftpd build possible:

apt-get install libmysqlclient15-dev libpam-dev debhelper zlib1g-dev libpq-dev libldap2-dev libssl-dev libwrap0-dev libcap-dev autotools-dev dpatch libacl1-dev libattr1-dev unixodbc-dev libsqlite3-dev

• Well, that should be it – run the below commands in proftpd-dfsg-1.3.3a directory, and it should result in built deb packages outside of this directory (cd ..):


chmod 755 debian/rules
dpkg-buildpackage

• If the build was complete with no errors, install proftpd-basic package, check if it contains mod_sftp.so file:

dpkg -i proftpd-basic_1.3.3a-3_amd64.deb
dpkg -L proftpd-basic | grep mod_sftp
/usr/lib/proftpd/mod_sftp.so
/usr/lib/proftpd/mod_sftp_pam.so

• To make the module work, add this line to /etc/proftpd/modules.conf:

LoadModule mod_sftp.c

• Add this one to /etc/proftpd/proftpd.conf:

Include /etc/proftpd/sftp.conf

• And finally, create /etc/proftpd/sftp.conf with the contents (this assumes proftpd will be listening on port 22 to accept incoming SFTP connections; if your OpenSSH server uses this port already, adjust either config appropriately):

<IfModule mod_sftp.c>
<VirtualHost 192.168.10.20 127.0.0.1>
SFTPEngine on
SFTPLog /var/log/proftpd/sftp.log
TransferLog /var/log/proftpd/xferlog-sftp.log
# Configure the server to listen on the normal SSH2 port, port 22
Port 22
# Configure both the RSA and DSA host keys, using the same host key
# files that OpenSSH uses.
SFTPHostKey /etc/ssh/ssh_host_rsa_key
SFTPHostKey /etc/ssh/ssh_host_dsa_key
# Configure the file used for comparing authorized public keys of users.
SFTPAuthorizedUserKeys file:~/.sftp/authorized_keys
# Enable compression
SFTPCompression delayed
# Allow the same number of authentication attempts as OpenSSH.
#
# It is recommended that you explicitly configure MaxLoginAttempts
# for your SSH2/SFTP instance to be higher than the normal
# MaxLoginAttempts value for FTP, as there are more ways to authenticate
# using SSH2.
MaxLoginAttempts 6
</VirtualHost>
</IfModule>


It changed with qemu 0.13.0 (still release candidate as of writing this blog post), where resize option was added to qemu-img.

To resize a qcow2 image, download the latest release candidate of 0.13.0, compile it, and resize the image:

srv1:/usr/src/qemu-0.13.0-rc1# ./qemu-img -h
qemu-img version 0.12.90, Copyright (c) 2004-2008 Fabrice Bellard
usage: qemu-img command [command options]
(...)
  resize filename [+ | -]size

srv1:/var/lib/vz/images/108# /usr/src/qemu-0.13.0-rc1/qemu-img resize vm-108-disk-1.qcow2 +5G
Image resized.