23 July, 2012

Working with disk images and dd

dd is an amazing tool. I use it to clone SDHC cards for small ARM-computers. In this post, I'll write down some tips/common scenarios.

Clone a disk
Couldn't be more simple. sudo dd if=/dev/sda of=/tmp/sda.img. That's it. This will create a bit-by-bit copy of the whole disk, including partition tables, partitions and file systems.

Restore a cloned disk
sudo dd if=/tmp/sda.img of=/dev/sda

It is also possible to clone individual partitions by specifying them instead (/dev/sda1).

Mount whole disk image as a "real" device
This is my favorite feature. The disk image is mounted as a loopback device, and can then be managed exactly as a real disk (using gparted, for example).

  1. First, we mount the image as device loop0.
    sudo losetup /dev/loop0 /tmp/sda.img
  2. If we now look at this partition in gparted (using sudo gparted /dev/loop0) there will be exclamation marks next to the partitions on the device, although it will find the partitions (the partition table is available). To make it find the partitions, we have to tell Linux to reread the partition table for the loop device.
    sudo partprobe /dev/loop0
  3. The disk (including partitions) can now be managed using sudo gparted /dev/loop0, and the individual partitions will also appear as /dev/loop0p1, /dev/loop0p2, etc.

Disconnect loop device
sudo losetup -d /dev/loop0

Mount individual partitions directly from hard drive image
To do this, we first need to know where the partition starts in the whole disk image. Find this using sudo fdisk -l /tmp/sda.img. The block count will be printed, look for the Start block for the partition you want to mount. There will also be a note of how many bytes each block is (in my case 512 bytes). Multiply that with the Start block for the partition. The resulting number is the byte offset where the partition begins in the disk image. To mount the partition, use sudo mount /tmp/sda.img /mnt/somefolder -o loop,offset=<bytenumberhere>.

There are several tools available for resizing partitions (resize2fs), creating filesystems (mkfs.ext4), checking filesystems (fsck.ext4) - usually I just end up using gparted, plain and simple.

That's it for now. Will add more if I find more useful stuff.

16 February, 2012

Unix file/directory permissions

A quick run-through/reference.

Each file or directory in Linux have permissions like below (ls -la lists these).

  • -rwxrwxrwx (files)
  • drwxrwxrwx (directories, specified by the prefix d)

Each pair of rwx specifies read, write and execute permission for (in order) the user (owner of the file/directory), the group and all others. Execute permission on a directory means permission to list the files and directories inside the directory. Some examples:

  • -rwx------ (the owner of the file can read, write and execute the file)
  • -rw-r-x--- (the owner of the file can read and write, the group can read and execute)
  • drwxr-x--- (the owner of the directory can read, write and list subitems in the directory, the group can read and list subitems)

The permission pair rwx can also be expressed as a single number.

  • the read permission is represented by 4
  • the write permission is represented by 2
  • the execute permission is represented by 1

The numbers are then added to get the final permission. For example, rwx is set by 4+2+1 = 7. 4+2 = 6 means rw-. 4+1 means r-x. This makes it possible to represent the three permission pairs (user, group and others) using three numbers, such as 777 for -rwxrwxrwx or 775 for -rwxrwxr-x.

Permissions on a Linux-based system is commonly altered using three commands.

  • chmod
  • chgrp
  • chown

Alters the permissions of a file or directory. Some examples:

  • chmod u+rwx file to add the rwx-permission to the user.
  • chmod g-w,o+x file to remove the w-permission from the group and add execute permission to others.
  • chmod -R 777 dir to recursively change the permissions on dir (all three permission pairs) to rwxrwxrwx.

Changes the group of a file/directory. For example, chgrp guest file sets the group to be affected by the group permissions to be guest. The -R option can be used here as well.

Changes the owner AND group of a file/directory. Works as chgrp, but instead of a single group name argument, chown is used as chown root:guest file. And the -R option can be used here as well. I kid you not.

Now to the interesting parts! What determines what permissions the files and folders we create get by default? The answer is: the default permissions together with something called an umask. The default permissions are

  • 777 (drwxrwxrwx) for directories, and
  • 666 (-rw-rw-rw-) for files.

But how come the files and directories you create won't have these permissions? Here is where the umask comes into play. The umask is set to 002 by default on any modern Linux system (probably on not-so-modern Linux systems as well...). This value is subtracted (digit per digit, minimum 0) from the default permissions for any new file or directory you create. So if you create a file, the default permission is 666 - 002 = 664. This means newly created files should get the permissions -rw-rw-r--. Go try. For directories, this will instead be 777 - 002 = 775, which corresponds to drwxrwxr-x. The execute permission is added by default to directories so all users are allowed to list contents in directories (as opposed to files, which all users certainly should not be allowed to execute).

The umask can be set by running the command - you guessed it - umask. For example, umask 022 sets the umask to be 022 for the current session. If you want it "forever", put it in ~/.bashrc or similar.

Another nice-to-know feature is the s option to the group permission pair. Usually, created files and directories are given the current user and the current user's primary group as user and group. This might not be what you want, for example if you have a team file share which all members of the team should be able to access, but you don't want to change all users' primary groups. This can be solved by setting the s-bit on the group permission pair, chmod g+s teamdir. Any files or directories created inside the teamdir directory will then get the group of the parent directory (recursively). This will however not affect directories already inside teamdir, so a command like find team -type d -exec chmod g+s {} \; might be useful here to set the s-bit on any existing directories.

The above is often mistakenly referred to as the "sticky bit". That is however not what setting the sticky bit means. The sticky bit can be set on directories, and, when set, only the file's owner, the directory's owner or a superuser can rename or delete created subfiles and subdirectories. The sticky bit is set using chmod +t dir and can be seen running ls -la as drwxr-xr-t (the t, or T if x is not set for permission pair o).

13 October, 2011

XBMC - gathered tips and tricks

Well, not yet, but I'll add more to this post once (if) I come upon them.

Scraper scrapes the wrong TV-show/does not find my TV-show at all
Add a file named tvshow.nfo into the root directory of the TV-show with a link to the correct show on thetvdb.com, for example http://thetvdb.com/index.php?tab=series&id=<id>.

Tearing seen on Ubuntu 11.10
This completely fixed my tearing issues on XBMC running on Ubuntu 11.10, nVidia ION (ZBOX). Somehow, Unity isn't letting go of the OpenGL something even though XBMC is running in fullscreen mode. To fix this, download the Compiz Config Settings Manager.

sudo apt-get install compizconfig-settings-manager

After the install it can be found in the launcher meny ("Start menu"). Check the option Unredirect Fullscreen Window. That's it! Start XBMC and your tearing issues should be gone!

03 October, 2011

Telia IPTV

So I just got myself television over Internet, commonly known as IPTV. From Telia. I should have acted on those warning signals... "from Telia". My idea was to simply install Tvheadend and add the IPTV channels, it should be simple and compliant as my HTPC is already connected to the TV "signal source": Internet. Turns out it is not.

I attached Telia's Motorola-STB and started watching a channel. I then plugged in my computer and started Wireshark to monitor the network traffic. I noticed whenever I switched channels, the STB would request a new "ICMP Join Multicast Group" for different IPs and ports. These are the IPs and ports I found to correspond to a channel (didn't take note which):


So I thought I could just use VLC Media Player (as a first test) to watch udp://@ Turns out I couldn't. Why? Because Telia encrypts every second/third frame! "Of course they do" you're saying, "how else would they stop unauthorized viewers?". Am I the only one who took for granted they restrict TO WHOM they send their IPTV multicast traffic instead of encrypting each channel?

I've Google'd around lots, but I can't seem to find anyone who successfully decrypted the channels (note that I do legally have the right to watch these channels, I'm not trying to hack access to channels I shouldn't have). I thought it should be simple, as I have the identification code, control code and PIN from Telia. If anyone ever reads this, and if that person happens to know anything about decryption of MPEG-streams (yeah right... what are the odds of that happening), leave a comment :) Actually, if you're interested in discussing this matter at all, leave a comment. I'm open for suggestions.

It would've been a very neat setup to have Tvheadend as backend and XBMC as the frontend (with functionality from the new PVR branch rapidly developing), thus removing the need for another box (less cables, less energy consumption).

More reading on the subject:

01 October, 2011

Laggy Flash playback in Ubuntu 11.04

I've had problems for years with choppy/laggy Flash playback in Ubuntu, and I've always been told its a problem with my graphic card drivers... until today! I found this simple and concise solution: http://akifcardak.blogspot.com/2011/01/choppy-flash-playback-in-full-screen.html.

I'll just repeat the solution here for my own reference:

sudo mkdir /etc/adobe
echo "OverrideGPUValidation=true" > ~/mms.cfg
&& sudo mv ~/mms.cfg /etc/adobe

After I ran these commands and restarted my browser (Google Chrome) my Flash videos started running smoothly even in fullscreen mode.

28 August, 2011

Monitoring VMware ESXi 4.1

There's a great blog about VMware related subjects everyone should know about: http://blog.peacon.co.uk/. The scripts I'm using in this post is grabbed from that blog, this post.

What we're doing here is monitoring our ESXi host using a virtual machine running Ubuntu. I have two scripts, one that runs on a weekly basis sending a status e-mail so that I know the server (and the monitoring) is working, and one that runs every 20 minutes to check for new alerts.

This is how I installed it. As no one is reading this blog anyway, I'll write a short version for my own reference. If anyone against all odds would like more information, let me know.

  1. Create a folder at /var/lib/esxihealth to hold our scripts.
    sudo mkdir /var/lib/esxihealth
  2. Get the files esx-health-custom.pl, esx-health.pl, minutely.sh and weekly.sh.
  3. Add a new user to ESXi named monitor, give him read access.
  4. Add a new user on the VM used for monitoring named monitor.
    sudo useradd monitor
  5. Install Perldoc, libssl and libxml.
    sudo apt-get install perl-doc libssl-dev libxml-libxml-perl
  6. Install VMware CLI and vSphere SDK for Perl 4.1 (5.0 didn't work for me, not sure what the problem was) on the VM.
  7. Make the scripts run using crontab.
    sudo crontab -u monitor -e
    Add the following lines:
    0,20,40 * * * * sh /var/lib/esxihealth/minutely.sh
    0 14 1,15 * * sh
    The first line tells cron to launch the script minutely.sh every 20 minutes. The second line tells cron to launch the script weekly.sh on the 1st and the 15th each month (not really weekly, I know..).

The content of the script minutely.sh:

perl esx-health.pl --server host.domain.com --username monitor --password somethingsmart --mailhost smtp.domain.com --maildomain domain.com --mailfrom esxi-status@domain.com --mailto user@domain.com,anotheruser@domain.com --cpuwarnpc 99 --memwarnpc 99

The content of the script weekly.sh is almost identical, except it launches the script esxi-health-custom.pl which sends an e-mail regardless of wether there has been an alert or not.

21 August, 2011

Handling SIGTERM in a Java application

Recently I had a Java application (more like a Java service/daemon) which I controlled using start-stop-daemon (more info here). When the Java daemon is shutdown using /etc/init.d/myscriptd stop, a SIGTERM is sent to the process. As my Java daemon was using sockets and a database, I would like to close those connections before terminating the application. How?

Turns out it is quite simple to do that in Java using the Runtime.addShutdownHook() method.

Runtime rt = Runtime.getRuntime();
rt.addShutdownHook(new Thread() {
public void run() { shutdown(); }; });

This registers a shutdown hook. The shutdown hook is a thread which is started when the Java application receives the SIGTERM signal, and its run method can be used to close any open resources (of course, there shouldn't be any time consuming calls made in this method as the OS might force-kill the process if it takes to long to exit). In the example above, the private method shutdown() is called (you just have to love those one-liners... this one creates an instance of a new class where the run method is overridden with custom functionality and passes it on to the addShutdownHook()-method - all in one row).