Speaker at JavaForum, Stockholm
This evening I have been a speaker at JavaForum in Stockholm. The topics was about pragmatic functional programming. How this ideom can be used on both the server-side (Groovy) and on the client-side (JavaScript/Prototype).
The offspring was a small application around GoogleMaps, using AJAX technology with the JavaScript library Prototype.
Here are my slides:
Convenient installation of Ubuntu @ KVM
This is my second post regarding virtualization using KVM on Ubuntu. You might want to read the previous tutorial first.
The last thing we did was to install Ubuntu using an ISO file and then after within the new VM, install Apache and show how one could access it from somewhere else at the LAN. This is all you need in order to start create your own VMs. However, the creation process can be improved in several ways. First, there are high-level tools like the virt-* tools, which I will address in some other post. Second there is a way to improve how a Ubuntu VM is defined.
JeOS
There is an ingenious shell script named ubuntu-vm-builder, that both creates the virtual disk and downloads all required packages of Ubuntu and creates a ready to use system, without the need to first download the ISO and then proceed through the (tediuos) installation process.
In addition, the flavour of Ubuntu is (normally) JeOS (Just enough OS), which is a shrink-wrapped Ubuntu Server OS optimized for running within a VM. It’s foot print is just 300Mb.
Installation
Start by installing the script
sudo apt-get install ubuntu-vm-builder
Usage
The minimal command line you can use to create a Ubuntu (8.04) system on KVM is
sudo ubuntu-vm-builder kvm hardy
This will create disk file in a generated sub-directory, download and install the JeOS packages and leave you with a fresh VM which you can launch (as usual) and logon to using ubuntu/ubuntu as usr/pwd. There are of course plenty of command line switches you in practice want to use. See ‘man ubuntu-vm-builder’ and/or ‘ubuntu-vm-builder –help’ för more information.
Virtualization @ Ubuntu
f you I have just started to explore virtualization for an internal project. The objective is to provide an “unbounded” number of ad-hoc servers for testing and customer show-case. So, I’m going to assemble a physical server, with a 4-core 64bit CPU, 8GB RAM and 1TB of mirrored disk space (RAID1). In the mean time I’m investigating how to set up and use KVM running on Ubuntu, on one unused server at home.
Virtualization
Virtualization is the art of emulating computer with a computer. It has been around for a very long time, becuase it was one of the corner stones of main-frames. However, it is only recently it has been easly available for “mere mortals”. The term is sometimes blurred by the two different concepts of application virtualization and system virtualization respectively. The former means virtual machines like the JVM for Java, CLR for .NET, JAM for Erlang, Parrot for Perl6 and more.
However, it is the latter (system) we normally refer to as virtualization. That means running one or more emulated computers on top of a real computer. User-space emulators has been around for several year, like VMware Workstation, VirtualBox and QEMU, they do not require hardware support nor changes of the OS.
When hardware manufactors noticed the increased interest for virtualization, the started to support it natively. Intel-VT and AMD-V are two such technologies. As soon as the these CPUs hit the market new tools came out relying on the built-in support. KVM is one such open source tool, which is directly supported in Ubuntu.
So the rest of this (quite long post) will discuss how to
- Install KVM on Ubuntu (Server)
- Create virtual machines and install an OS on them
- Configure bridged networking, so one can reach a VM
Before you proceed reading, just a small disclaimer: The text is the result of my own recent investigation of the topic and serves as a presentation of my findings. Googling around gives many different view points of how to setup KVM and most of them are not complete from a newbies point of view. Therefore, I’m trying to fill that missing spot.
Prerequisites
The first step is to understand and check the requirements: Does my CPU support virtualization?
If you are considering buying a CPU for assembling you own server, look for support of Intel-VT or AMD-V, depending on your preference of CPU manufactor.
If you already have a system you can check for harware support with this command:
jens@goofy:~$ egrep '(vmx|svm)' /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr lahf_lm flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr lahf_lm
If you don’t see any output, it means your CPU do no support hardware virtualization. You can still use user-space virtualization tools, but not KVM. The flag ‘vmx’ is for an Intel CPU and ’svm’ for AMD.
You can use virtualization on both 32-bit and 64-bit systems. However, for the former you are limited to 2GB address space, therefore it is common to go for a 64-bit system. For a 64-bit system, you need both a 64-bit CPU and an 64-bit OS installed. (N.B. all commands below are for Ubuntu Linux)
Use this command to check if your CPU is 64-bit
jens@goofy:~$ grep ' lm ' /proc/cpuinfo flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr lahf_lm flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr lahf_lm
If nothing is printed it means the ‘lm’ long mode flag is not present. To see if your OS is 64-bit you can use the following command
jens@goofy:~$ uname -m x86_64
If you see anything else, like 386, i486, i586 or i686, it means your are running a 32-bit OS.
Installation
The first step is to install the basic package of software.
sudo apt-get install kvm qemu uml-utilities bridge-utils
This will install
- kvm - provides kernel based virtualization
- qemu - provides virtualization tools (kvm and qemu goes together)
- uml-utilities - user mode Linux, contains a set of tools and we are going to use tunctl
- bridge-utils - another set of tools for creating network bridges
Next you need to add yourself to the kvm group and add the kvm kernel module (kvm-intel for Intel and kvm-amd for AMD)
sudo adduser `id -un` kvm sudo modprobe kvm-intel
Finally, logout and login again, so the group change will take place. Remark; you can postpone the logout till after the network configuration, because you will need to reboot anyway. However, if you want to create your first VM without the bridge, do the logout/login now.
Virtual Machines
Compared to using products like VMware and VirtualBox, it is relievingly easy to create a VM in KVM. First you create the virtual disk, then you launch KVM with an OS ISO to run the installer. Therefore, grab an ISO file (or CD) for an OS. I have chosen to go for Ubuntu Server 8.04 64-bit (the link goes to a Swedish mirror, you might want to use another mirror). You can find all Ubuntu variants here. Let’s assume you now have the file ubuntu-8.04.1-server-amd64.iso, in a sub-directory called iso/.
Create a virtual disk
We are using the qemu tool to create the virtual disk.
jens@goofy:~/vm$ qemu-img create ubuntu-server-8_04-disk.img -f qcow2 6G Formatting 'ubuntu-server-8_04-disk.img', fmt=qcow2, size=6291456 kB jens@goofy:~/vm$ ls -lFh total 40K drwxr-xr-x 2 jens jens 4.0K 2008-09-23 13:06 iso/ -rw-r--r-- 1 jens jens 36K 2008-09-23 13:06 ubuntu-server-8_04-disk.img
This will create a file named ubuntu-server-8_04-disk.img in the current directory, of maximum size 6GB (expands on demand) and with the QEMU file format ‘qcow2′. You can for example create a virtual disk that is compatible with VMware. See the manual for more information ‘man qemu-img’. That’s it. Now time to install the OS.
Launch the installer
When you launch KVM it will pop-up a console, which of course requires you to be running on a desktop system. If you are running on a server, without a desktop, you can instead launch KVM with a VNC server providing the console. That means you can connect remotely to using a VNC viewer, such as TightVNC for Windows or Vinagre for a GNOME desktop.
Desktop console
kvm -m 512 -k sv -boot d -cdrom iso/ubuntu-8.04.1-server-amd64.iso -hda ubuntu-server-8_04-disk.img
This will boot a VM from a CDROM (’-boot d’), where the CD is pointing to an ISO file (-cdrom iso/…). The VM will be running with 512 MB of RAM (’-m 512′) and use a Swedish keyboard layout (’-k sv’). If you are not Swedish you probably want to change ’sv’ to something else or skip this option. Finally, it will use the virtual disk we created before as the IDE hard disk 0 (’-hda ubuntu-…’). You can find more info with the command ‘kvm –help’.
VNC remote console
If you want to work remotely, just append ‘-vnc :1′ to the command. This will launch KVM with a VNC server running on port 5901 (-vnc :n => 5900 + n).
kvm -m 512 -k sv -boot d -cdrom iso/ubuntu-8.04.1-server-amd64.iso -hda ubuntu-server-8_04-disk.img -vnc :1
Installing OS
If you went for the desktop console you will see a console pops up and the machine booting, finally the Ubuntu server installer screen. I will proceed with the VNC console. My host machine (goofy) has (an internal) IP of 192.168.189 and I will connect using VNC. (Of course, you should substitute your own IP)
When I’ve logged on to the console, the VM has already booted and the first screen of the installer is shown (click on the image for a full-scale picture). Choose your language and press ENTER, it will then show you the main menu.
I will not digress into installing Ubuntu Server, just point out a few topics. The installer will ask you for the system language to use, your location, machine name (e.g. vm1) etc. When it comes to disk partitioning, let the installer do the full job.
Installing the OS will take serveral minutes. When it prompts you for setting up a user name and password, write it down as well, becuase you will need it shortly. Towards the end it will ask you to install various packages. Just choose SSH to start with, you can install the server tools later using apt-get.
At last, the installer will ask you to reboot. The reboot will fail, becuase you need to unmount the CD and change the KVM launch parameters. For now, just terminate KVM (CTRL-C in the command window you launched it from) and return to the command prompt, You will of course loose your VNC connection when the VM goes down.
After the installation we can see the disk has grown a bit.
jens@goofy:~/vm$ ls -lFh total 704M drwxr-xr-x 2 jens jens 4.0K 2008-09-23 13:06 iso/ -rw-r--r-- 1 jens jens 703M 2008-09-23 13:55 ubuntu-server-8_04-disk.img
Running VM
Now is the time to launch the new VM. Basically, you will use the same command as above, except without the boot and cdrom flags. Here I’m launching the VM with a VNC server console.
kvm -m 512 -k sv -hda ubuntu-server-8_04-disk.img -vnc :1
I’m now logged on to the VM as you can see. The next step is to check the network connectivity. Do we have an IP number? As you can see below, we have one (10.0.2.15).
Next, let’s see if we can reach out from the VM.
wget -o /dev/null -O - http://www.ubuntu.com/ | head
This will print the first lines of the welcome HTML page of the Ubuntu web. Choose you favorite site. You you can now reach out, but not reach in, i.e., without a bridge you cannot logon using SSH, for example.
jens@goofy:~$ ssh jens@10.0.2.15
^C
jens@goofy:~$ ping 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
^C
--- 10.0.2.15 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6008ms
jens@goofy:~$ traceroute6 10.0.2.15
traceroute: unknown host 10.0.2.15
jens@goofy:~$ tracepath6 10.0.2.15
getaddrinfo: Resolver Error 0 (no error)
jens@goofy:~$ tracepath 10.0.2.15
1: goofy.local (192.168.0.189) 0.112ms pmtu 1500
1: kerberos (192.168.0.1) 0.282ms
1: kerberos (192.168.0.1) 0.228ms
2: kerberos (192.168.0.1) 0.249ms reached
Resume: pmtu 1500 hops 2 back 64
jens@goofy:~$
Shutdown the VM and procced to the next section. Enter this command inside the VM’s console
sudo poweroff
Network Configuration
After you have created a VM and installed an OS, it has access to the outer world using NAT, but is hidden inside a private VLAN. You have only access to it using its console, i.e., you cannot login using SSH nor set up Apache and view it over HTTP. Therefore, you almost always want to create a network bridge, which means the VM will operate as if it was a ‘normal’ machine on your network. Setting up a bridge and launch KVM requires several steps: some system re-configuration, creating some scripts and a reboot. The steps below are taken from the Ubuntu community KVM doc.
In short, these are the steps:
- Add a bridge interface to /etc/network/interfaces
- Add permissions to a TUN device in /etc/udev/rules.d/40-permissions.rules
- Sudo:fy an kvm interface startup hook in /etc/kvm/kvm-ifup
- Add sudo rights for the kvm group
- Create a kvm launcher script
- Reboot
I suggest you perform all changes directely on the host machine’s console, not via SSH, because some of the changes will temporary cutoff your network connectivity.
You can use your favorite editor to edit the files, just remember to launch the editor using sudo. Ubuntu comes preinstalled with both VIM and NANO. Personally I prefer Emacs (and hate VI), but you have to install it first. Nano is a good very small editor and intuitive to use in a console window.
/etc/network/interfaces
The first file to edit requires you to temporary cutoff your network. Shut it down with
sudo invoke-rc.d networking stop
The open /etc/network/interfaces in nano
sudo nano /etc/network/interfaces
Change the content to something similar to
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
auto br0
iface br0 inet dhcp
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
The important points are the physical device eth0 is set to manual mode, before it was probably set to dhcp. A new device (br0) is defined, which bridges to eth0 and gets its IP from DHCP. You can also use a static IP, for details, see the Ubuntu docs mentioned above. Save and exit with ^O ENTER ^X. Finally, restart the network again.
sudo invoke-rc.d networking start
/etc/udev/rules.d/40-permissions.rules
Next, edit the file /etc/udev/rules.d/40-permissions.rules and append the following line to the end of the file
KERNEL=="tun", GROUP="kvm", MODE="0660"
/etc/kvm/kvm-ifup
Next, edit /etc/kvm/kvm-ifup and add ”’sudo”’ (tripple single quote around sudo) in front of ifconfig and brctl. Here is how it should look.
#!/bin/sh
switch=$(ip route ls | awk '/^default / { for(i=0;i<NF;i++) { if ($(i) == "dev") print $(i+1) }}')
'''sudo''' /sbin/ifconfig $1 0.0.0.0 up
'''sudo''' /usr/sbin/brctl addif ${switch} $1
exit 0
/etc/sudoers
Next, add sudo rights for the kvm group. In this case, you must use the VI tool called ‘visudo’, which updates the file /etc/sudoers. The tool visudo checks the syntax and guards you from accidental misstakes.
sudo visudo
Append the following line to the end of the file
%kvm ALL=(ALL) NOPASSWD: /sbin/ifconfig, /usr/sbin/brctl, /usr/sbin/tunctl
This will permit members of the kvm group to use ifconfig, brctl and tunctl. If you never have used VI before, please be very, very careful. VI operates in modes, that means insert, append and navigation mode. If you screw up, leave without saving with <ESC>: q!
Use the arrows keys to scroll to the very last character of the file, type <ESC>a to go into append mode, press <ENTER> for a new line and paste the line above. Then press <ESC>: wq to write and quit. Verify by looking at the file.
sudo cat /etc/sudoers
kvm-run
At last, you will now create your own launcher script. You can put it in the same directory as your VMs or in ~/bin (if you have one) or somewhere else. Just remember where. Create the file (kvm-run) and fill it with the text below
#!/bin/bash USERID=`whoami` MAC=$(echo DE:AD:BE:EF:`echo $RANDOM | cut -c -2`:`echo $RANDOM | cut -c -2`) NIC_MODEL=rtl8139 NIC_IF=`sudo tunctl -b -u $USERID` kvm -net nic,vlan=0,macaddr=$MAC,model=$NIC_MODEL -net tap,vlan=0,ifname=$NIC_IF $@ sudo tunctl -d $NIC_IF &> /dev/null
The script will be used as your alternate kvm launcher. It first gets your user name, then generates a hardware id (MAC) for the network interface (NIC), assigns a model name to the nic and creates a tunnel device owned by you and with brief output. Finally, it launches kvm with the network settings and the command line parameters appended at the end. You need to make the launcher script executable
chmod +x kvm-run
Reboot
Now is the time to reboot your host machine.
sudo reboot
Running with bridged network
Back again, it’s time to take your VM out for a ride. Start it using the launcher script and add parametes for memory, disk, keyboard and vnc.
./kvm-run -m 512 -k sv -hda ubuntu-server-8_04-disk.img -vnc :1
Attach to the VNC console and check the IP. This time we have an IP at our local net (192.168.0.186).
Now, let’s check if we can logon using SSH (PuTTY).
We are in, great! From now on, you barely need the VNC console, use PuTTY or your favorite SSH tool.
Before I end this (very long) post, let’s install Apache and verify it works as expected. Within your VM (now use your ssh access), type the command below and confirm with ‘Y’
sudo apt-get install apache2
After the installation you should have two servers running, the SSH server from before and the new Apache web server.
jens@vm2:~$ netstat -l -t inet Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:www *:* LISTEN tcp6 0 0 [::]:ssh [::]:* LISTEN
Check with your browser, from somewhere in your local net and you will see the default Ubuntu Apache start page.
Final words
This concludes my tutorial of how to setup KVM on Ubuntu and make the VMs visible on your local net. There are many more topics to describe, but that will be in some other post. For example (no promise):
- High-level tools for creating and managing VMs
- Convenient installation of Ubuntu direct from the network, using a shrink-wrapped Ubuntu OS (JeOS)
- Installation of some other Linux OS
- Installation of some Windows flavor OS
Good luck and please comment.
Speaker at JavaForum
I will give a (swedish) speach at JavaForum in Stockholm, 30 September 2008. The title is Rena kartor - GoogleMaps med AJAX and I will talk about how functional programming now is commodity on both the server and the client side. The basis of my talk is a small AJAX application using Google Maps. More info can be found here.
Unique portlet ID
Sometimes one need to retreive a unique ID for a particular portlet instance. In my case I needed it to implement configuration storage for each portlet instance independent on the user (see another post for comments on this topic).
The catch is, there is no such thing in the portlet specification. The closest you can find is the getNamespace(). The spec says “to ensure they are unique in the context of the portal page”. However, not neccessarily unique through the whole portal. At least, you don’t know until you have investigated it, for example asking the vendor or reading the code. Let’s say the mthod returns a unique ID. Now, here is the catch; you need that ID in processAction() as well!
In most cases, a portlet goes through render mode (doView) and then after the user clicks a button, which invokes processAction(). That means, in render() you can grab the ID and for example inject it into the session.
public void render(RenderRequest req, RenderResponse res) throws ... {
String ID = res.getNamespace();
//... do something with ID ...
req.getPortletSession().setAttribute("portlet.id", ID);
super.render(req, res);
}
And then pull it out again in processAction()
public void processAction(ActionRequest req, ActionResponse res) throws ... {
String ID = req.getPortletSession().getAttribute("portlet.id");
//. . .
}
So far so good, as they say. Now, consider a lazy user, which stares at the portlet page until the session times out and thenafter clicks that darn button - pow! NullPointerException.
The only solution is to figure out how the portal handles portlet IDs internally. For example, the URL might be annotated with a portlet ID after an action request. For the portal I have been working with (SiteVision) I found out that its ID can be found in the session using the following code snippet.
public String getID() {
String id = (String) getSessAttr("portlet.id");
if (id == null) {
String portletID = "" + getReqAttr("sitevision.portlet");
id = "svid" + portletID.replace('.', '_');
setSessAttr("portlet.id", id);
}
return id;
}
I have to emphasize this again: this is not portal independet code.
-
Jens Riboe
