Trevor

Migrating Email from Outlook to Evolution

 Email, Evolution, Gentoo  No Responses »
Feb 122009
 

Introduction

There are just a few applications that keep me tied to Microsoft Windows.  For work, there is primarily one, and it’s a bigee!  Microsoft Office, especially Outlook.  We have looked at a few other solutions previously that mitigate this problem by running Office on top of either Wine (a Windows emulation layer) or KVM (a Virtual Machine with a Windows installation).  However, I wanted something more complete.  I wanted a native client that could access my corporate email and calendar on a Microsoft Exchange server.  As far as I know, at this point in time, there is only one client that would claim such functionality – Gnome’s Evolution.

Installing Evolution with the Microsoft Exchange Connector plug-in is not so bad.  Under the hood, it’s a bit of kludge, accessing your email through Exchange’s HTTP interface, but hey, it works!  🙂  The big question is, “How do I import all my Outlook data, archived in local PST files?”

Unfortunately, PST is a proprietary Microsoft format (I know, you are shocked), and the only way to access it is through a proprietary Microsoft library (mapi32.dll or mapi.dll) that only exists on a Microsoft Windows installation.  (I know, again, you are flabbergasted.)

There are 3 types of solutions that I found to migrate and liberate your email.  None of them are trivial.  Yuck!  😥  All of them are potentially time-consuming, especially if you have lots of data, but here are a few pointers.  🙂

Solution #1:  Migrate Through Mozilla Thunderbird on Windows

Assuming you still have access to your Windows machine that is running Outlook, you can install Mozilla Thunderbird on the same machine.  The advantage being that Thunderbird can use the proprietary library to import your Outlook email!

The importing process if fairly simple.  Roughly, it is:

  1. Install Thunderbird, and uses its import wizard to suck in your Microsoft Outlook email and contacts.
  2. Close Thunderibrid.  Find where it stored the email, usually in a place like:
    C:\Documents And Settings\<USERNAME OR DEFAULT>\Application Data\Mozilla\Profiles\default\XXXX\Mail\imported.mail
    where, XXXX is some random collection of letters and numbers.
  3. In Evolution, import each email folder (named after the email folder, saved as an extensionless file by Thunderbird in the MBOX format) — one at a time!  Yikes!

As you can see, there are a few problems here:

  1. You have to process each folder one at a time!  Very time consuming, tedious, and boring…
  2. You cannot have nested folders, especially with email in a folder with additional folders.  Thunderbird’s storage format (MBOX) is standard, but old.  In fact, it has been “hacked” to support nested folders.  That was not part of the original idea.  Anyway, things can get messy if you have lots of folders, especially nested folders with messages and sub-folders in the same folders.
  3. Furthermore, this does not really address your calendar.
  4. Some people have reported that Thunderbird drops some of their email!!!!

I gave up on this method somewhere between problem #1 and #2.  … This could still work for you, and it does for many people, provided you either don’t have many folders or you don’t mind lots of pointy-clicky.  😉  If you go this route, make sure you count the number of messages in each folder inside Outlook and verify those counts once you import your email into Evolution.  You don’t want to lose any email.

Solution #2:  Migrate Through Outport on Windows

This solution is very similar to the above solution, except it uses an open source tool, Outport, to migrate your email on a Windows machine.  Consequently, I believe it has the same limitations.  It may be able to migrate calendar, but I still don’t like the idea of having to import my email into Evolution one MBOX file at a time.

Plus, Outport seems to have not been updated since 2003.  That’s worrisome, but maybe it’s no big deal…  Anyway, I wanted something easier.

Solution #3:  Migrate via an IMAP server

You may be thinking, “What?  This is easier?”   Well, it was to me.  🙂

An IMAP email server not only supports providing messages, but it also supports uploading them.  In most email clients, you can add an IMAP based email account and then simply “drag-and-drop” your email or folders from your local computer to the IMAP server, or vice-versa.

The basic idea of this method is roughly this:

  1. Setup or procure access to an IMAP server. (You have friends, right?  Gmail provides IMAP support too.  ;-))  You will need to know the machine name, possibly the port number, account id, password, and authentication method (plain text, SSL, TLS, etc.).
  2. Add this server as an IMAP account to Outlook.
  3. Drag and drop all your email to the IMAP account.  (You may not be able to put folders inside the “Inbox”, but you can add them to the “root” of the account.)  I know this is still “pointy-clicky”, but it’s much less than the other methods.
  4. Close Outlook.
  5. Open Evolution.
  6. Add the same IMAP server as an account in Evolution.
  7. Drag and drop all your email from the IMAP account to your local store on Evolution.  Done!  Tada!!!  🙂

If you are worried about setting up an IMAP server, I have some basic instructions listed below.

Migrating Contacts

You can export your contacts in Outlook to CSV files or other formats, which you can read into Evolution.  Thunderbird can also read them and export them into other file formats, if you wish.  No biggee here.

Migrating Calendar Entries

This can be more problematic.  If you are like me, you may not have many entries, so I simply moved my calendar entries back up the Exchange server, using it as a temporary staging area.

Others have suggested exporting the Calendar entries up to Google’s Calendar and then back down to Evolution in iCal format.  Sounds good, but I can’t offer any experience on this method.

Setting Up A Temporary IMAP Server

I did not want to transfer all my email via Gmail’s IMAP server for lots of reasons (bandwidth, privacy, possible sub-folder complications, etc.).  So, I decided to set up my own local IMAP server on my Gentoo Linux box.  Here’s a few quick pointers, if you decide to do the same.

Install dovecot’s IMAP server:

emerge dovecot

Verify the configuration is set up to use the “maildir” format, not MBOX.  The “maildir” format was created by the qmail folks, and it has several advantages over the older MBOX format, including native sub-folder support.  Here are the important configuration changes that I had to make in /etc/dovecot/dovecot.conf. You may want to verify or research these further:

# Config file = /etc/dovecot/dovecot.conf
# This is required to enable network access - otherwise, it may be restricted to localhost only
listen = *, [::]
# This enables plain text passwords.
# This is not good on a public network, but works fine for private LAN's.
# And, it simplifies the login process in Outlook and Evolution.
disable_plaintext_auth = no
# This sets the mailbox format to 'maildir', and it stores the mail
# in your user's homespace on the IMAP server, avoiding
# various file permission problems on the remote box.
mail_location = maildir:~/.maildir

Then you fire up the dovecot server:

# /etc/init.d/dovecot start

And, away you go!

Conclusion

It’s not a pretty process, but if you bang your head on it long enough, you’ll get through it!  🙂  Just remember, freedom awaits all those who persist!  😀  Good luck!

Share

Copying Evolution’s Mail and Configuration Files

 Evolution  No Responses »
Feb 122009
 

Introduction

I recently switched to using Gnome’s Evolution groupware client for email and calendar management.  Although I really like Mozilla’s Thunderbird interface better for email, Evolution offers connectivity to Microsoft’s Exchange server, while Thunderbird does not.   Plus, Evolution provides built-in calendar support, which is still in the works as a plug-in (SunBird) for Thunderbird, and again, no Exchange connectivity support.  😥

Well, I am learning to like Evolution.  It has some nice features, like various spam filters plug-ins (Bogofilter, SpamAssassin).  And, it generally works.  But, now the question arises, how do I migrate my Evolution settings and email data from one machine to another?  Unfortunately, it’s not a simple matter of copying a single directory with its contents.  But, the solution is not too bad…

Solution

Evolution’s data and settings live in 4 places:

  • $HOME/.evolution – email data (Inbox, Sent, etc.)
  • $HOME/.gconf/apps/evolution – your account settings
  • $HOME/.gnome2_private/Evolution – your passwords
  • $HOME/.camel_certs – SSL Certificates, if any

However, you should not just close Evolution and copy these directories.  Evolution uses a calendar server and a Gnome settings server, which may keep some of these files open.  Therefore, you must shut-down Evolution and the appropriate servers, and then archive the data, like so:

$ gconftool-2 --shutdown
$ evolution --force-shutdown
$ cd
$ tar -zcvf evolution-backup.tar.gz .evolution .gconf/apps/evolution .gnome2_private/Evolution .camel_certs

This creates a compressed archive file, which can be copied to another Linux box and unarchived, like so:

$ tar -zxvf evolution-backup.tar.gz -C ~/

Others have recommended shutting down the Evolution and Gnome settings servers on the new machine, before installing the new files.  Afterward, the servers can be restarted, like so:

$ gconftool-2 --shutdown
$ evolution --force-shutdown
$ tar -zxvf evolution-backup.tar.gz -C ~/
$ gconftool-2 --unload evolution_setting.xml
$ gconftool-2 --load evolution_setting.xml

I did not have to do this, but it may be useful.

After doing this, simply fire up Evolution, and you should be good to go!

Share

Missing Icons in Evolution on Gentoo

 Evolution, Gentoo  1 Response »
Feb 122009
 

Problem

Recently, I switched to using Gnome’s Evolution for my email and calendar groupware client on my Gentoo Linux workstation.  I encountered multiple problems that had to be worked.  One of the more noticeable and annoying problems was the messed up icons.  All of the button icons were pictures of blank paper with a red X in the middle, similar to a “file not found” icon.

Gnome Evolution with broken icons.

Apparently, this is a fairly common problem on new installs where people primarily use KDE and not Gnome.  Here is a collection of solutions that may help.

Solutions

Do you have any gnome icons installed?  Make sure you have at least gnome-icon-theme and hicolor-icon-theme installed.  For Gentoo, this is performed, like so:

# emerge gnome-icon-theme hicolor-icon-theme

Do you have read access to the installed icons?  For whatever reason, the installed icons occasionally lose their read permission.  As root, or using sudo, you need to ensure the appropriate permissions, like so:

# chmod -Rf a+rX /usr/share/icons

Have you configured all your Gnome-based tools, including Evolution, to use an installed icon theme?  As your standard, desktop user (not root), check for the existence of this file:

$ cat ~/.gtkrc-2.0

If it does not exist, or if it does not specify an icon theme, you need to add one, like so:

$ echo gtk-icon-theme-name=\"gnome\" >> ~/.gtkrc-2.0

Restart your Evolution email client and enjoy your pretty icons!  Red X’s be gone!!!  🙂

References

Gnome Evolution with proper icons

Share

Updating Album Dates in Gallery2

 SQL, web-tools  No Responses »
Jan 232009
 

Problem Introduction

I use an open-source solution for hosting my family’s photos, called Gallery. Recently, my gallery2 install crashed, and I lost my online database. “No big deal”, I thought. “I have all the pictures saved on my local computer.” WRONG-O!!! I hit two major snags: One, gallery can stumble very easily while doing a bulk upload from a local server. Two, all of the album dates were stamped with the time of my recent upload, which destroyed by chronological sorting of the albums.

I started manually “editing” each album through Gallery2’s web interface, but that gets old real fast, so I decided to try a little MySQL wizardry.

Solution

Gallery2 stores most of its data, except for the photos and movies, in a MySQL database on the web-server. Of course, you can use other back-ends, but MySQL is very popular. The database can be manipulated using your favorite MySQL monitor, whether that occurs through a command shell or a web-interface, like phpMyAdmin.

My solution was to use MySQL to search through each album, find the oldest picture, and update the album’s origination date to match the oldest picture.

Using my favorite MySQL interface, I crafted the following SQL statement to examine the problem:

SELECT g_id AS albumId, g_title, g_originationTimestamp, (
SELECT MIN(g_originationTimestamp)
FROM g2_Item
INNER JOIN g2_ItemAttributesMap
 
ON g2_Item.g_id = g2_ItemAttributesMap.g_itemId
WHERE g_parentSequence LIKE CONCAT( '%/', albumId, '/' )
) AS oldestPicture
FROM g2_Item
INNER JOIN g2_ItemAttributesMap ON g2_Item.g_id = g2_ItemAttributesMap.g_itemId
WHERE g_canContainChildren &gt; 0

This produced results like:

albumId albumTitle currentTime oldestPicture
7 Gallery 1232042846 NULL
58896 Caleb’s Turn 1232650930 1175871585
59400 Originals 1232650977 1175871585
59416 Landscapes and Nature 1232651065 1232651128
59417 Blossoms and Sunsets 1232651128 1081207812
59600 icebergs 1232651167 1232651167
59633 More Sunsets and Bathtime 1232651168 1174399484
59634 Originals 1232651168 1174399529
59943 Trees and Geese 1232651264 1081295988
60013 Family Photos 1232653712 978487347

As you can see, many of the albums had pictures much older than the timestamp on the folder. The theory appears sound. Let’s experiment!

Attempt #1

Since we are experimenting on your gallery database, obviously you want to back it up first. If I have to tell you that, you are in over your head. 😉

Using a modified version of the above query, I tried to update the origination timestamp to the value of the oldest picture, like so:

UPDATE g2_Item AS albumId SET g_originationTimestamp =
(
SELECT MIN(g_originationTimestamp)
FROM g2_Item INNER JOIN g2_ItemAttributesMap
ON g2_Item.g_id=g2_ItemAttributesMap.g_itemId
WHERE g_parentSequence LIKE CONCAT('%/', albumId.g_id, '/')
) WHERE g_canContainChildren &gt; 0

Unfortunately, this produces the following error:

ERROR #1093 - You can't specify target table 'albumId' for update in FROM clause

Apparently, UPDATE will not allow you to modify a table that is part of the query. That seems reasonable. However, that is exactly what I needed to do. This forced me to create a temporary table to hold the intermediate results, and then use those results to update the desired table.

Attempt #2

Here was my final solution, which required 4 separate statements:

DROP TABLE IF EXISTS newAlbumTimes;

This first statement is only necessary, if you iterate and experiment with this approach. It deletes the temporary table, if it exists, which may happen after you tweak something and try again, depending on your connection method.

CREATE TEMPORARY TABLE newAlbumTimes (albumId INT(11), albumTitle VARCHAR(128), albumTimeStamp INT(11), oldestPicture INT(11))
SELECT g_id AS albumId, g_title AS albumTitle, g_originationTimestamp AS albumTimeStamp, (
SELECT MIN(g_originationTimestamp )
FROM g2_Item
INNER JOIN g2_ItemAttributesMap
 
ON g2_Item.g_id = g2_ItemAttributesMap.g_itemId
WHERE g_parentSequence LIKE CONCAT( '%/', albumId, '/' )
) AS oldestPicture
FROM g2_Item
INNER JOIN g2_ItemAttributesMap ON g2_Item.g_id = g2_ItemAttributesMap.g_itemId
WHERE g_canContainChildren &gt;0
ORDER BY oldestPicture;

Now that is the “brains” of the operation. First, notice the select statement, very similar to the original query. However, these results are being fed into a “CREATE TEMPORARY TABLE” statement, which catches the results. Notice, we had to tell MySQL the structure of this temporary table, which should generally match the structure of the output columns. This temporary table will be destroyed when the connection closes. However, we will use this table in the meantime, to update the timestamps based on the oldest picture’s timestamp.

UPDATE g2_Item INNER JOIN newAlbumTimes
ON g2_Item.g_id = newAlbumTimes.albumId
SET g_originationTimestamp=oldestPicture
WHERE oldestPicture AND oldestPicture &lt; albumTimeStamp;

Using the temporary table, the origination time is updated for all albums in the item table, but only if the oldest picture column is non-NULL and if the picture is stamped older than the album. Otherwise, we assume the time stamp for the album is better than what we have calculated, so we leave it alone.

UPDATE g2_Entity INNER JOIN newAlbumTimes
ON g2_Entity.g_id = newAlbumTimes.albumId
SET g_creationTimestamp=oldestPicture, g_modificationTimestamp=oldestPicture
WHERE oldestPicture AND oldestPicture &lt; g_creationTimeStamp;

We also have to update the creation time stamps, which is the date actually displayed and permitted to be edited for the album. This is the statement that does what we want!

Conclusion

Running the above 4 statements in order produces the desired affect.  Preferably, these should be executed as a single entry to avoid the temporary table disappearing. However, this procedure only “bubbles up” the oldest time stamp by one level.  Most albums will be updated correctly after this.  However, if you have nested albums (albums inside albums), you will have to run this procedure at least once for each level of album (or folder) hierarchy.  This will cause your top level album to show a creation date equal to or older than the oldest picture anywhere in its hierarchy.

Notes

If you are running an older version (Gallery1) or newer version (Gallery3, currently in early development), you will obviously have to modify the above statements to match the your database structure. Also, these statements were executed using MySQL 5.0.70. If you use an older version, like MySQL 4.0, you may find the nested queries do not work. This will require you to create an additional temporary table to cache those intermediate results – much more complicated.  It may be better to upgrade, if you can.

You can include additional constraints on the WHERE clause of the final two statements to only modify certain albums.  This can be useful if you are only want to fix a certain subset of albums in your gallery.

Share

Using EchoAudio Mia on Gentoo with ALSA

 Audio, Gentoo  1 Response »
Dec 052008
 

Introduction

Every computer you buy or assemble comes with some decent integrated sound card.  A while back (~2000), I bought a “prosumer” (entry-level professional) sound card, the Mia by EchoAudio.  It was a pain at first, because the Windows drivers for XP were just not mature.  However, after about 6 months of updating drivers every other week, I finally got a rock-solid sound card that blew the doors off any integrated sound card I have used, even up to this day (12-2008).  All the integrated cards sound so “tinny” and “weak”, compared to the rich, full sound of this card.

For some time, I have avoided moving my main workstation to Linux, in some measure, due to the lack of drivers for this sound card.  However, a few weeks ago, I noticed an entry for the EchoAudio Mia in the kernel config!  Here is how I managed to get it working on Gentoo Linux.

Default ALSA Installation

The most modern sound system on Linux at this time is ALSA.  The Gentoo wiki page for installing it is here:

http://www.gentoo.org/doc/en/alsa-guide.xml

One important note:  ALSA can be compiled into the kernel or compiled separately as loadable modules.  Currently, Gentoo has mostly abandoned the in-kernel approach (which uses the alsa-drivers package), and it now uses the loadable module approach.  There’s no reason to buck the system here, so we are going to use the loadable module approach.

When you activate the appropriate ALSA kernel options, make sure you include the driver for the EchoAudio Mia, or whatever Echo Audio product you may be using.  Otherwise, you can follow the above guide up to the point where you are ready to run alsaconf.

Make sure you:  Recompile the kernel.  Copy it into place.  Update grub.conf.  Reboot.  You know the drill.  🙂

Modified ALSA Installation for Mia

The Mia driver depends on alsa-utils, but it also needs other ALSA packages, which are not necessary for other integrated sound cards, like hda-intel.  Furthermore, the default make.conf flags do not include the Mia components.  To include these, and to use the latest version of ALSA ;), let us first add a few keywords and compile flags to the ALSA build configuration:

Now, we are ready to install ALSA, again, in addition to the other necessary packages:

# Use latest version of everything ALSA
$ echo -e "media-sound/alsa-tools ~amd64\nmedia-sound/alsa-utils ~amd64\nmedia-sound/alsa-firmware ~amd64\nmedia-sound/alsa-headers ~amd64\nmedia-libs/alsa-lib ~amd64" >> /etc/portage/package.keywords

# Include Mia during ebuilds
$ echo 'ALSA_CARDS="mia"' >> /etc/make.conf

# Check for weirdness:
$ emerge -pvt alsa-utils alsa-tools alsa-firmware alsa-lib alsa-headers

# Build!
$ emerge alsa-utils alsa-tools alsa-firmware alsa-lib alsa-headers

Now update the ALSA configuration using alsa-conf, and you should be good to go!

$ alsaconf

Other Tips

The best mixer to use is the echomixer, which is made for the EchoAudio products, like the Mia.

In the default configuration, most everything is muted, so you will have to slide up the appropriate sliders.  Just be careful not to overdrive the card.  Stop at “0 dB” or less.  Do not slide it up to “+6 dB”; otherwise, you get a fair amount of distortion.

Other pages that mention the “alsa-drivers” package are based upon the “IN-KERNEL” approach.  Those instructions are not compatible with these.  Be careful if you decide to “mix and match”.

References

  1. ALSA’s EchoAudio development status – http://www.webalice.it/g_pochini/ead
Share

Use Windows XP on Gentoo Linux with KVM

 Gentoo, Networking, virtualization  2 Responses »
Dec 052008
 

Introduction

Regrettably, there are some software applications that just run better on Windows, specifically, Windows XP.  Of course, Windows runs better on Linux, so I guess we can still hold to our axiom that “All things run better on Linux”. 😯 Ok, not really.  😀

In my case, I would like to run the full Office 2003 suite on Linux.  Using Wine is an option, but it can be a bit buggy.  CrossOver is a better option, but it costs money, and I am cheapskate.  Plus, it’s like “double-taxation”.  I have to pay somebody a tax, so I can pay my Microsoft tax.  That ain’t right!  Well, I am not really a Linux purist – I am just a practical cheapskate.  And, I would like to learn about OS virtualization, and apparently, so do you!  Otherwise, you would not be reading this.  😉

Installing KVM on Gentoo

KVM is just one of many possible virtualization methods, which is a way to run one OS inside of another OS.  (Imagine a “window” that is running Windows XP inside, and it “thinks” it is the entire computer.  It does not realize that it is running inside of another “computer”.)

Note:  These instruction are for:

Host:  Gentoo 2008.0
RAM:  >= 1.5GB
Kernel:  2.6.27
KVM:  v79
Guest:  Windows XP Pro

Of course, there these instructions may have to be varied slightly to accommodate your exact application.

Here are my modified instructions, based on the Gentoo Wiki:

1. Update the kernel with IN-KERNEL KVM (no need for module mayhem):

$ cd /usr/src/linux
$ make menuconfig

[*] Virtualization --->
        --- Virtualization
        <*> Kernel-based Virtual Machine (KVM) support
        <*>   KVM for Intel processors support
        < >   KVM for AMD processors support
        <*>   PCI driver for virtio devices (EXPERIMENTAL)
        <*>   Virtio balloon driver (EXPERIMENTAL)

If you want to be able to do networking, you should also enable VLAN bridging and tapping, while you are here:

Device Drivers --->
    [*] Network device support --->
            <M> Universal TUN/TAP device driver support

Networking --->
    Networking options --->
        <*> 802.1d Ethernet Bridging
        <*> 802.1Q VLAN Support

Copy new kernel into place.  Update grub.conf.  Reboot using new kernel. … You know the drill. 🙂

2. Ensure the latest version of KVM:

$ echo 'app-emulation/kvm ~amd64' >> /etc/portage/package.keywords

3. Activate useful USE flags:

$ echo 'app-emulation/kvm gnutls sdl' >> /etc/portage/package.use

4. Check emerge for weirdness and install:

$ emerge -pvt kvm usbutils bridge-utils usermode-utilities

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild   R   ] sys-apps/usermode-utilities-20040406-r1  0 kB
[ebuild   R   ] net-misc/bridge-utils-1.4  0 kB
[ebuild   R   ] sys-apps/usbutils-0.73  USE="zlib -network-cron" 0 kB
[ebuild   R   ] app-emulation/kvm-79  USE="alsa esd gnutls modules ncurses sdl -havekernel -pulseaudio -test -vde" 0 kB

Looks ok to me.  Does it look ok to you?  Let’s go:

emerge -pvt kvm usbutils bridge-utils usermode-utilities

5. Setup access for non-root users:

For each non-root user, add them to the KVM group:

gpasswd -a <non-root-userid> kvm

Launching Guest for First Time and Installing Windows:

First, you need to create an “image” file, which will contain the entire Windows XP guest OS (think C:\ drive).  Here’s the default way:

kvm-img create winxp_raw.img 30G

This will create a RAW image format that is 30 GB in size.  This is the simplest and most portable image format.  However, it is not the coolest!

kvm-img create -f qcow2 winxp.img 30G

This does the same thing, but it uses the latest QEMU format, which enables additional features, like image overlays.

Second, if you use ALSA for your host’s sound, then you can enable it like so:

export QEMU_AUDIO_DRV=alsa

Third, install Windows XP into image.  Here’s the simplest method:

kvm -hda winxp.img -cdrom /dev/cdrom1 -boot d

This will do the same thing but it will use a local image of the install ISO (faster?), use 1GB of RAM (default is much less), use host’s local clock (helps Windows see the right time), emulate better VGA card (more colors and resolution), and allow access to 2 processors:

kvm -hda winxp.img -cdrom /winxp/ISO/WINXPSP2.ISO -m 1024 -localtime -vga std -smp 2 -boot d

Using The Virtualized Guest Windows XP

The emulated “box” will reboot once as part of the Windows XP installation process.  After it comes back up, you should be good to go!  You can now download programs, install programs, update the install, etc., just like you would with a regular Windows XP installation.  Of course, there will be some limitations, because the emulated hardware is not exactly feature-rich.

At some point, you will “shut down” the emulated Windows XP machine.  To restart it, use the a similar command – with the exception of not booting from the install disk (or ISO):

kvm -hda winxp.img -cdrom /dev/cdrom1 -m 1024 -localtime -vga std -smp 2

Accessing Host Drives

To access a local partition, first ensure that samba is installed – not running – just installed.

Then, simply add the path to the mounted partition, like so:

kvm -hda winxp.img -cdrom /dev/cdrom1 -m 1024 -localtime -vga std -smb /path/to/dir

Otherwise, you can add the share name, if you have samba already running and properly configured, like so:

kvm -hda winxp.img -cdrom /dev/cdrom1 -m 1024 -localtime -vga std -smb <share_name>

Inside the Windows guest OS, the mounted share is available at:

\\10.0.2.4\qemu

Also, from inside the guest OS, you can SSH, SCP, SFTP, FTP, or telnet to the host, depending on running host services, using this IP:

10.0.2.2

Other options are listed on the Arch Linux Wiki.

Using Overlays

I have found that this process is not entirely stable.  Some combinations of host hardware, host OS, emulated hardware, and guest OS, work better than others.  If I tried to emulate too much hardware, the Windows XP installation would crash, so I typically had to install using the most modest, simplest emulation.

Also, I found that this process could be slow during “boot-up” and “installation”.  Maybe disabling ACPI emulation would help?

Anyway, you can quickly make a wrong turn and wreck your “virtual machine”, basically ruining your created image, in which you spent so much time setting up and installing.  Fortunately, there are 2 techniques to help mitigate this annoyance.

One, with the emulator shut-down, simply copy the image file to another location or file name to back it up, like so:

cp -fp winxp.img winxp_orig_install.img

Then you can always copy the good install back over a broken install, like so:

cp -fp winxp_orig_install.img winxp.img

Ta-Da!  Of course, the downside of this approach is rampant disk-usage.  You need double the disk space, possibly more, depending on how many backups you make.

Another technique is using “overlays”.  You can create an “overlay” of a good image like so:

kvm-img create -b winxp.img -f qcow2 winxp_20081225.ovl

Then you can boot from the overlay, just like you would any other image, like so:

kvm -hda winxp.ovl -cdrom /dev/cdrom1 -m 1024 -localtime -vga std -smp 2

The overlay contains a “diff” of the new state and the original image, so it is much smaller, since it only contains what changed. If the overlay gets corrupted, you can simply delete the overlay, create another, and go again!

You can also stack overlays, but I think this can waste diskspace too, and it requires that you keep the whole “stack” in place.  Pull out one overlay in the stack, or just move it, and the whole thing tumbles down!  🙁

Based on a tip from Bryan Murdock’s blog for resizing image files, you can combine an overlay stack into a new, single, independent image file, like so:

# create a new image file, which will be the consolidated image
kvm-img create -f qcow2 winxp_new.img 30G

Download the latest clone-zilla LIve-CD (or DVD) ISO.

http://www.clonezilla.org

In a KVM session, boot from the downloaded ISO, and include your original overlay as HDA and your new image as HDB, like so:

kvm -cdrom clonezilla-live-1.2.1-17.iso -hda winxp.ovl -hdb winxp_new.img -m 1024 -vga std -boot d

Generally, you should accept the defaults, unless you know what you are doing, and of course, you do. 😉  The key is to choose the option for a “device-device disk/partition to disk/partition” clone, or something to that effect.  (I don’t remember the exact wording.)  Make sure you copy the complete contents, including the MBR.  Your source is HDA, and your target is HDB.  … The cloning takes a while.  After it finishes, be sure to halt, and then start up a new KVM session, using the new image file:

kvm -hda winxp_new.img -cdrom /dev/cdrom1 -m 1024 -localtime -vga std

Try hiding the original image and overlay files to see if it works.  It should!

Other Things

The default network setup is good for surfing the web, downloading stuff, and checking email.  However, if you want other devices on your LAN to “see” the guest OS as another machine, you will have to create a bridge and tap.  This gets a little more complicated.  See the references below for more details.

If the installation or something crashes, try restarting the machine – but, don’t boot from the installation disk.  Many times the install process completed “good enough” before crashing.  😮  Yeah, I know.  It smells funny to me too, but it works.  🙄  Just be sure to keep lots of backup copies of your images or overlays.

References

  1. http://en.gentoo-wiki.com/wiki/KVM
  2. http://kvm.qumranet.com/kvmwiki/HOWTO1
  3. https://help.ubuntu.com/community/WindowsXPUnderQemuHowTo
  4. https://help.ubuntu.com/community/KVM
  5. http://bryan-murdock.blogspot.com/2007/12/resize-qemukvm-windows-disk-image.html
  6. http://www.linuxjournal.com/video/run-your-windows-partition-without-rebooting
Share

Resetting TikiWiki’s admin Password and Email with MySQL

 web-tools  No Responses »
Nov 202008
 

Introduction

In TikiWiki’s setup, an admin account is created.  The default password is “admin”.  And, you are forced to change that password as soon as you login.  Unfortunately, you are not required to provide an email address for the “admin” account during setup.

This sets the stage for two moments of sheer panic:

  1. You log out of admin, forget the password, and try to reset the password:  Resetting passwords typically require an email address; therefore, you cannot reset the password.  Yikes!  8-|
  2. While in admin you enable, “Challenge-Response Authorization”.  Later, you log out of admin and try to log back in.  Unfortunately, “Challenge-Response Authorization”, although more secure and therefore desirable, depends on the user additionally entering his email address.  But, the admin account has no email address, and so you cannot log back in as admin, even if you know the password.  Double Yikes!!! 8-|

If you get bitten, here are a couple of anti-venom therapies.

Solution

If you have database access (MySQL, in this case), either via a shell, MySQL client, or phpMyAdmin, you can directly update the database.  Instructions here are for shell access:

To simply reset the admin password to “admin”:

$ mysql -u my_db_user_id i -p my_db_name
Enter password:

mysql> UPDATE `users_users` SET `password`='admin', `hash`= md5('adminadmin') WHERE
    -> `login`='admin';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> quit;
Bye

Once you log in, you should obviously change the password to a strong, non-default password.

If you are unable to login, and you cannot reset the password, because you forgot to assign an email address to the “admin” account, do the following:

$ mysql -u my_db_user_id i -p my_db_name
Enter password:

mysql> UPDATE `users_users` SET `email`='myemail@server.com' WHERE `login`='admin';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> quit;
Bye

By setting the email address for the admin account, you can now reset the password, assuming you did not disable that feature too, before you logged out. >8-|

If you are still stuck, be sure to check out phpMyAdmin, which provides you a graphical tool to explore and edit the underlying MySQL database without knowing the command-line syntax.

References

http://doc.tikiwiki.org/tiki-index.php?page=lost+admin+password&bl=y

http://doc.tikiwiki.org/tiki-view_faq.php?faqId=7

Share

Configure NetVanta 3120 for HP C7280 beyond VPN LAN

 Networking  No Responses »
Nov 122008
 

Problem Introduction

I frequently use a workstation that sits behind an Adtran NetVanta 3120.  The NV3120 is powerful little box.  It provides secure VPN access back to corporate headquarters, but it also provides a 4-port switch, a highly configurable firewall, and generally more bells and whistles than you could ever want.

Recently, I added a Hewlett-Packard Photosmart C7280 to the network.  However, it sits beyond the NV3120’s LAN, so other workstations on the greater LAN can use it, like my Gentoo laptop.

The default printer configuration went great!  I was printing in no time from my workstation behind the NV3120.  However, scanning was another issue.

Apparently, when used in scan mode, the HP C7280 originates traffic on a non-established port, so it becomes blocked or is otherwise lost.  I knew everything else was working fine, because I could bypass the NV3120 and scanning would work great!  But, that was not going to be acceptable for frequent use.

Network Topology

Here is an ASCII representation of the relevant network subsection:

                                            Incoming Line
                                                  |
                                         [ Wireless Router ]
                                            192.168.1.1
                      /                           |                        \
               192.168.1.100                192.168.1.101              192.168.1.102
            [ NetVanta 3120 ]    [ HP C7280 Printer-Scanner-Fax ]    [ Workstation #3 ]
                10.10.0.110                                               Laptop
             /             \
    10.10.0.99           10.10.0.100
[ Workstation #1 ]   [ Workstation #2 ]
  Windows XP Pro           Linux

The critical path is highlighted in red.

Solution

Eventually, I called the Adtran tech support.  I was pleasantly surprised to receive a call back from a support engineer in short order.  He understood my problem very quickly, and he knew immediately what to do!  What follows are my scribbled notes for the steps he proscribed:  (Of course, your policy names and IP numbers may vary.)

  1. Backup NV3120 configuration, in case something goes wrong.  😉
  2. Configure NV3120 to grab static IP, not DHCP-based IP from wireless router:

    Click on:  System -> Public Interface -> IP SettingsComplete as follows:IP:  192.168.1.100
    MASK:  255.255.255.0
    DEF GW:  192.168.1.1

  3. Add UDP relay for NetBios broadcast by HP C7280 printer (192.168.1.101) to be encapsulated and relayed through NV3120 (192.168.1.100) to its LAN (10.10.0.X) and vice-versa:Click on:  Data -> UDP Relay -> IP Helper AddressAdd following addresses:

    10.10.0.99 – Public (eth0)
    10.10.0.100 – Public (eth0)
    192.168.1.101 – vlan1
    UDP Forward Protocol:  netbios (port 137)  [Press “Add”]

  4. Allow traffic between 10.10.0.X subdomain and and 192.168.1.X subdomain:Click on:  Data -> Firewall -> Security Zones -> Edit Security Zones -> Public
    Add Policy to Zone “Public”
    Type:  Allow
    Description:  Allow 192.168.1.X to 10.10.0.X
    Stateless Processing:  OFF
    Destination Security Zone:  <Any Security Zone>
    Source – Specified:  192.168.1.0 / 255.255.255.0
    Destination – Specified:  10.10.0.96 / 255.255.255.240
    Protocol:  any

    Use “arrows” to move new policy right below “VPN Selector” and before everything else.

  5. Allow traffic between 192.168.1.X subdomain and 10.10.0.X subdomain:Click on:  Data -> Firewall -> Security Zones -> Edit Security Zones -> Private

    Add Policy to Zone “Private”
    Type:  Allow
    Description:  Allow 10.10.0.X to 192.168.1.X
    Stateless Processing:  OFF
    Destination Security Zone:  <Any Security Zone>
    Source – Specified:  10.10.0.96 / 255.255.255.240
    Destination – Specified:  192.168.1.0 / 255.255.255.0
    Protocol:  any

    Use “arrows” to move new policy right above “NAT list wizard-ics” and below everything else.

  6. Create policy for UDP Relay:Click on:  Data -> Firewall -> Security Zones -> Edit Security Zones -> Public
    Add Policy to Zone “Public”
    Type:  Advanced
    Description:  Relay netbios
    Policy Action:  Allow
    Destination Security Zone:  <Self Bound>
    Stateless Processing:  OFF

    – Add New Traffic Selector –
    Type:  Permit
    Protocol:  UDP
    Source:  Any, Any
    Destination:  Any host, Port:  “Well Known” : 137 – netbios-ns

    Use “arrows” to move second from top, below “VPN selector”, but above recent “Allow 192.168.1.X to 10.10.0.X” policy.

  7. Reassign VPN Crypto Map – It occasionally gets lost during the above changes:Click on:  Data -> VPN -> VPN Peers -> Advanced VPN Policies -> Assign Crypto Maps to Interfaces:

    Public    VPN
    vlan1     none

  8. Save configuration changes and reboot NV3120 unit.  Backup configuration again, in case something goes wrong in the future.  😉
  9. On wireless router, add a “static route”, so traffic intended for the VPN subdomain (10.10.0.X) that leaves the printer (192.168.1.X) can find its way back to VPN subdomain and not onto global internet:

    On wireless router’s configuration page (not NV3120), click on:  Advanced -> Static Routes -> Add (Or, similar depending on brand and model):

    Name:  NV3120-VPN
    Private:  Off
    Active:  On
    Destination IP:  10.10.0.96
    Gateway IP:  192.168.1.100
    Metric:  2Beyond the destination and gateway IP’s, the exact settings and menu navigation path will vary depending on router’s brand and model.

Explanation

Admittedly, the solution is a bit complex, but the problem is a bit complex too.  Part of the complication comes from the fact that the printer broadcast various netbios-ns UDP packets to find computers on its domain.  However, the computer used in this case does not exist on that domain.  It exists on a private, VPN domain.  So, we have to not only configure the firewall to allow traffic, but we must also relay UDP broadcasts between the two domains.

Many thanks to the Adtran support engineer, who guided me through the above steps, including configuring the 3rd party router!

Share

Replacing Newline Characters (^M) in Emacs

 Editors  No Responses »
Nov 102008
 

The Problem

Occasionally, I find a text file that was written on a Windows box that contains additional garbage text.  Most often the text displayed, looks like this:

/*^M
 * @(#)MyApplication.java  2.0  01 April 2005^M
 *^M
 * Copyright (c) 2003-2005 Werner Randelshofer^M
 * Staldenmattweg 2, Immensee, CH-6405, Switzerland.^M
 * This software is in the public domain.^M
 */^M^M

Or, even worse, as a single line, like this:

/*^M * @(#)MyApplication.java  2.0  01 April 2005^M *^M * Copyright (c) 2003-2005 Werner Randelshofer^M * Staldenmattweg 2, Immensee, CH-6405, Switzerland.^M * This software is in the public domain.^M */^M^M

Either way, this is annoying, if not unusable.

Brief Explanation

The primary cause of the problem is a difference of encoding ‘newline’ between the Unix and DOS (Windows) conventions.  The difference is long-standing, dating back to the days when printers were the primary ‘display’.

The Windows’ convention uses two ASCII characters, which signal ‘line-feed’ (which meant to roll the printer paper up one line) and ‘carriage-return’ (which meant to send the printer head back to the beginning of the line).  Unix selected one of those characters (‘carriage-return’) to do the same thing.

These symbols usually appear as:

^M^J

Or,

^M

Depending on the encoding, platform, and application.

The Solution Using Emacs

On most Unix platforms, commands such as unix2dos and dos2unix can be used to convert a text file from Windows to UNIX format or vice-versa.  However, sometimes a file can get so garbled that even these tools do not work.  Regardless, it is nice to know-how to fix this in Emacs.

The easiest way to fix the second case in Emacs is:

  1. Place the cursor on the first part of the strange character, the caret (^).
  2. Press C-‘ ‘ (Control + Space) to begin marking.
  3. Move to the right one character.  (You’ll notice that it jumps an extra character.  That is because ^M is really one ASCII character.)
  4. Press C-W to remove the text.
  5. Immediately, press C-Y to yank the text back.
  6. Jump to the top of the document (Esc-< or M-<).
  7. Replace all occurrences:
    1. M-x replace-string
    2. Press C-Y to paste in the text to be replaced.
    3. Press C-Q, C-J to replace with a ‘quoted’ ^J, which is the Unix newline (or, C-Q, C-M, C-Q, C-J for Windows).
    4. Press ‘Enter’ to replace all occurrences.

A little experimentation will be necessary to adapt to other cases.  You can read more here:

http://lists.freebsd.org/pipermail/freebsd-questions/2006-October/134422.html

Share

Connect a Linux computer to an HP PhotoSmart C7280 Printer

 Gentoo  No Responses »
Oct 302008
 

HOWTO Connect a Linux computer to an HP PhotoSmart C7280 Printer

The HP PhotoSmart C7280 All-In-One printer contains a photo printer, scanner, and fax machine.  It can be setup as a wired Ethernet print server, wireless 802.11g print server, or a local USB printer.  It is very nice, and if you watch the NewEgg specials, you can often find one for a very good price every so often.  I have enjoyed using it from my Windows workstation; however, since I have the C7280 connected to my network through its Ethernet port (a wired print server), I would like to be able to use my Linux laptop to also print to it.

Fortunately, most HP printers are well supported in Linux.  So, I had high hopes!

As mentioned in other posts, my current favorite distribution of Linux is Gentoo, so my directions will be for Gentoo; however, you can probably adapt them to your favorite distro.

CUPS

CUPS is the modern Unix/Linux printing interface.  It provides both a server and client for the common printing tasks (lpr, lpq, lpstat, etc.).  Therefore, CUPS must be installed before you can do anything else.

I added a few extra USE flags to my CUPS install, although I don’t think these are necessary in general:

$ echo 'net-print/cups dbus ppds' >> /etc/portage/package.use

Beyond that, installation is simple:

$ emerge cups

Since we are connecting to the C7280 via the network, no configuration changes are required for CUPS.  However, you will have to fire up the CUPS daemon and add it to your start-up services:

$ /etc/init.d/cupsd start
$ rc-update add cupsd default

You can find more info on configuring CUPS to work on Gentoo with other setups here:

http://www.gentoo.org/doc/en/printing-howto.xml

HPLIP

The HP printer drivers are based on a standard HPLIP package, which is used with all modern HP printers, and a PPD file, which is specific to your printer model.  The latest HPLIP package can be installed in Gentoo, like so:

# For AMD64, Intel Core2, and newer x86 64-bit archs
$ echo 'net-print/hplip ~amd64' >>/etc/portage/package.keywords
# Install HPLIP
$ emerge hplip

The latest PPD file for the C7280 should be downloaded from the Linux Printing repository.  Currently, the C7200 model covers the C7280, and it’s PPD can be downloaded from here:

http://www.openprinting.org/show_printer.cgi?recnum=HP-PhotoSmart_C7200

On a Gentoo box, the PPD file should be saved in a certain location, and only root should have access to it:

mv <path_to_download>/HP-PhotoSmart_C7200-hpijs.ppd /usr/share/ppd/HP/
chown root:root /usr/share/ppd/HP/HP-PhotoSmart_C7200-hpijs.ppd

With that put in place, you are now ready to configure the HPLIP program, like so:

$ hp-setup

The wizard should make everything self-explanatory, except you may have to manually search for the PPD file, if the wizard cannot find it for you.  When I used the wizard, it was able to find the printer automatically and very quickly.  However, I had to locate the PPD file for it.

If everything goes smoothly, you will be done.  All that remains is to restart cups, like so:

$ /etc/init.d/cupsd restart

If things don’t go smoothly, you may have to add the printer manually through the CUPS interface or to the printers.conf file, as I had to do.

Manually Adding the C7280 to CUPS

Unfortunately, the HPLIP setup wizard was not working correctly, and I had to manually add the printer to CUPS.  I used the web interface to CUPS, which can be accessed using a web-browser on the Linux box at:

http://localhost:631

From here, I clicked on “Add Printer”, and manually entered the necessary information.  (You should know the IP address of the C7280 printer on your network.)  Most of it was obvious, except these two bits:  The device connection type was:

AppSocket/HP JetDirect

And, the “Device URI” was:

socket://192.168.0.11:9100

Of course, you will have to change the above IP address to match your needs.  … If you have already configured a Windows box to use the same printer, you can get some clues for the above info in the Windows’ printer’s properties.

The CUPS wizard may request a user id and password.  Any requested userid is referring to root and root’s login password.  These are needed near the end of the CUPS wizard, so it can edit the CUPS configuration files for you.

After entering the necessary info, pointing to the downloaded PPD file, and completing the web install, I was printing my first test page in no time!

If you prefer to work on the command line, and you are comfortable with CUPS, here are the modifications to my CUPS’ files:

/etc/cups/printers.conf

# Printer configuration file for CUPS v1.3.8
# Written by cupsd on 2008-10-30 17:37
<DefaultPrinter HP-PhotoSmart-C7280>
Info HP PhotoSmart C7280
Location My Office
DeviceURI socket://192.168.0.11:9100
State Idle
StateTime 1225405864
Accepting Yes
Shared Yes
JobSheets none none
QuotaPeriod 0
PageLimit 0
KLimit 0
OpPolicy default
ErrorPolicy stop-printer
</Printer>

That’s it!

And, for good measure, you should always restart CUPS after monkeying around with its files:

$ /etc/init.d/cupsd restart

Conclusion

Well, it took a little longer than I first hoped, but it was not so bad.  Now, I can print, scan, and fax from HP PhotoSmart C7280 using my Gentoo Linux laptop. 🙂

Share
 Older Entries  Newer Entries