My personal notes on all things technical and factual....

Fri, 20 Aug 2010

/Hosting/Amazon/EBS: Use python-boto to Snapshot Amazon EBS Volume

The Amazon EC2 toolkit[1] works great, but it is a Java app which makes it an incredibly bloated way to gain access to a couple of command-line tools on a server. python-boto[2] takes up all of about 1M, and its minimal Python dependencies were already installed on my Amazon instance. The necessary python script is very simple:

The only gotcha was that the stale version of python-boto in Debian stable did not seem to have a "create_snapshot" function, so I just grabbed a newer version from Debian testing.

And of course, just call this script from cron daily to make backups automatic.

[1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88
[2] http://code.google.com/p/boto/

posted at: 13:27 | path: /Hosting/Amazon/EBS | permanent link to this entry

Fri, 25 Jun 2010

/Hosting/Amazon/EC2: How to Backup an Amazon EBS-Boot Server

This is a script I wrote to backup several Amazon AWS EBS-boot servers, using the Python boto library[1]. Basically it takes a snapshot of the server's root EBS volume (defined in volumeIDs list) and then builds a new bootable Amazon Machine Image (AMI) for the server based upon that snapshot.

#!/usr/bin/env python

##########################################################
# Installation:
# This script requires boto ('pip install boto') to communicate with Amazon AWS API,
# and at least version 1.9a to handle an EBS boot instance
# Script inspired by http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html
#
# As of early June, we need svn version of boto:
#   pip install -e svn+http://boto.googlecode.com/svn/trunk/@1428#egg=boto
# until next release, which should be soon per this forum post:
#   http://groups.google.com/group/boto-users/browse_thread/thread/21dc3482ed7e49da
##########################################################

##########################################################
# This script is for the backup of Amazon AWS EBS Boot servers, and will perform one backup
# of one server, per script invocation. Note that there is nothing machine environment-specific
# in this script at the moment, so it can be run on any machine with the correct environment,
# to backup any other machine.
# Usage:
# * adjust the "Constants" section to reflect you current account & server environment.
# * Choose which server you would like to backup in the "which server" section.
# * Install the ElasticFox plugin in Firefox to observer the results of the backup process.
# * Run the script to perform a backup of the chosen server.
##########################################################

##########################################################
# Constants
##########################################################
instances = {"master":"i-xxxxxxxx",   "staging":"i-xxxxxxxx",   "production":"i-xxxxxxxx"}
volumeIDs = {"master":"vol-xxxxxxxx", "staging":"vol-xxxxxxxx", "production":"vol-xxxxxxxx"}
ownerID = 'xxxxxxxxxxxx' # I got this from ElasticFox, should be AWS account specific
waitSnapshot = 10 # wait increment (in seconds) while waiting for snapshot to complete

##########################################################
# Which server is this script backing up?
##########################################################
# thisServer = "master"
thisServer = "staging"
# thisServer = "production"

thisInstance = instances[thisServer]
thisVolume = volumeIDs[thisServer]

##########################################################
print ''
print '##########################################################'
print 'Backup of ' + thisServer + ' server:'
print 'Logging into Amazon AWS....'
from boto.ec2.connection import EC2Connection
conn = EC2Connection('xxxxxxxxxxxxxxxxxxxx', 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx')

##########################################################
from datetime import datetime
timeStamp = str(datetime.today())

print ''
print timeStamp
print 'Snapshotting the ' + thisServer + ' servers boot volume....'
comment = thisServer + ' ' + timeStamp
name = comment.replace(":",".") # AWS does not like ":" in AMI names
name = name.replace(" ","_") # make life easier when deleting stale images
snapshot = conn.create_snapshot(thisVolume, comment)
newSnapshot = snapshot.id
print 'Created new volume snapshot:', newSnapshot

##########################################################
import time
waitSnapshotTotal = waitSnapshot
snapshot = conn.get_all_snapshots(str(newSnapshot))
print ''
while snapshot[0].status != 'completed':
    print 'Snapshot status is ' + snapshot[0].status + ', ' \
          'wait ', waitSnapshotTotal, ' secs for the snapshot to complete before building the AMI.'
    time.sleep(waitSnapshot)
    waitSnapshotTotal = waitSnapshotTotal + waitSnapshot
    snapshot = conn.get_all_snapshots(str(newSnapshot))

##########################################################
print ''
print 'Building a bootable AMI based up this snapshot....'

# setup for building an EBS boot snapshot"
from boto.ec2.blockdevicemapping import EBSBlockDeviceType, BlockDeviceMapping
ebs = EBSBlockDeviceType()
ebs.snapshot_id = newSnapshot
block_map = BlockDeviceMapping()
block_map['/dev/sda1'] = ebs

# use the same kernel & ramdisk from running server in the new AMI:
attribute = conn.get_instance_attribute(thisInstance, 'kernel')
kernelID = attribute['kernel']
print 'kernel ID = ', kernelID
attribute = conn.get_instance_attribute(thisInstance, 'ramdisk')
ramdiskID = attribute['ramdisk']
print 'ramdisk ID = ', ramdiskID

# create the new AMI:
result = conn.register_image(name=name,
    description=timeStamp,
    architecture='i386',
    kernel_id=kernelID,
    ramdisk_id=ramdiskID,
    root_device_name='/dev/sda1',
    block_device_map=block_map)
print 'The new AMI ID = ', result

[1] http://code.google.com/p/boto/

posted at: 08:12 | path: /Hosting/Amazon/EC2 | permanent link to this entry

Mon, 15 Feb 2010

/Hosting/Amazon/EC2: Amazon AWS: Information You Need to Give Your System Administrator

Amazon AWS is designed to be able to give someone else the necessary privileges to control one's Amazon servers, without giving up the password of your Amazon AWS account. Here are a couple of very thorough treatments on the subject of Amazon AWS credentials: [5][6].

In order to broadly manage your account and its servers, there are two sets of keys your System Administrator is probably going to need to access and control your servers and data stores:

AWS Access Key / Secret Access Key
X.509 Certificate and Private Key

These two methods of authentication are also explained in the "Authentication" section of [1], and both sets of keys can be obtained from "Your Account" --> "Access identifiers" in your Amazon AWS account.

The "Access Key / Secret Access Key" is comprised of two long strings, much longer then what one typically thinks of as a "password". This is what a System Administrator needs most of the time for most Amazon AWS management tasks. The ElasticFox Firefox Extension[4], for instance, uses these for authentication. Following are examples of what these keys look like:

Access key: AKIAJQXQL474IJIOJATA
Secret Access Key: XQbln80m5ms8a4xUSxPd7xmyF/7IM9hM24bv9aez

The "X.509 certificate" is a pair of encryption keys (each of them much longer then either elements of the "Access Key / Secret Access Key") primarily used by the Java-based Amazon EC2 API Tools[2], as explained here[3].

The certificate looks like this:

And the private key looks like this:

[1] http://clouddb.info/2009/05/17/using-and-managing-aws-part-3-aws-security/
[2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351
[3] http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=100&externalID=1791&printable=true
[4] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609
[5] http://alestic.com/2009/11/ec2-credentials
[6] http://www.elastician.com/2009/06/managing-your-aws-credentials-part-1.html

posted at: 11:31 | path: /Hosting/Amazon/EC2 | permanent link to this entry

Wed, 11 Nov 2009

/Hosting/Amazon/EC2: Transferring an Amazon EC2 Image to Another Account

This[1] would be the document of record from Amazon....

First, configure ElasticFox so that there is one window connected to each of the two accounts (sending and receiving). Also, setup two terminals, each one configured with the necessary environment variables to use the standard EC2 command line tools[2] for one of the accounts.

Second, a logistical observation.... As I am not the owner of the Amazon accounts I am using, but rather a consultant using and operating the accounts on behalf of the owners, I do not have access to the "AWS Management Console". I "only" (this is actually quite a lot) have the privileges conferred by possessing the account keys. Using these privileges I can transfer a server AMI, but there appears to be no way to transfer an EBS store between accounts. So the first step is to login to the sending account, mount any EBS stores that are desired, copy there contents into the volatile area of the server, and create a new AMI that will contain both the server and the data.

One can get the list of AMIs belonging to the sending account thusly:

ec2-describe-images -o self

Now (briefly) make the just-created AMI containing the data public. (One can make the AMI visible only to one account, but I am under time pressure and I do not at the moment know the AWS account number of the receiving account):

ec2-modify-image-attribute ami-1573907c -launch-permission --add all

Now in the receiving account start the AMI (point and click with ElasticFox), then back at the command line open SSH in the firewall (necessary if this is a new account):

ec2-authorize default -p 22

In a terminal login to your running server in the receiving account, using the keypair you used in ElasticFox when the server was started, and the "public DNS" for the server being reported by ElasticFox:

ssh -i ~/ec2/id-keypair root@ec2-xx-xx-xx-xxx.compute-1.amazonaws.com

Go back to the sending account and reset the transferred AMI to the private state:

ec2-reset-image-attribute ami-1573907c -l

Double-check the permissions on an AMI with:

ec2-describe-image-attribute ami-1573907c -l

(It would appear to return nothing when there are no non-private permissions set....)

Back to the receiving account to: use ElasticFox to create a new EBS store, format it, then move the data from the server's volatile storage into the EBS. Also create and associate an ElasticIP to the new server. (This means that the server's "public DNS" will change, and the terminal login must be redone.) Open the http port (80).

On the new server, edit the Apache configuration. Delete unneeded MySQL databases. Start Apache and MySQL. Test.

Change the keys in the EBS snapshot script and test.

Change the keys in the server snapshot script and make a bundle of this just-started AMI for the new account.

[1] http://developer.amazonwebservices.com/connect/entry.jspa?entryID=530
[2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88

posted at: 13:05 | path: /Hosting/Amazon/EC2 | permanent link to this entry

Thu, 15 Oct 2009

/Hosting/Godaddy: Goodbye Godaddy!

For a while now I have been somewhat unhappy with Godaddy. Living in China, their website is bloated and very slow. There user interface is bloated and complicated. And on top of that, in the slow process of tranferring my domains out over a period of months, I have found their domain transfer-out process to be extremely complicated, and seemingly designed to baffle and confuse casual users (read: discourage domain transfers....)

May I recommend nearlyfreespeech.net[1], with a very fast, clean website, comparable domain name prices, and much cheaper website hosting prices.

[1] http://blog.langex.net/index.cgi/Hosting/NearlyFreeSpeech/

posted at: 04:04 | path: /Hosting/Godaddy | permanent link to this entry

Mon, 14 Sep 2009

/Hosting/vpslink: One Year of vpslink service

As of September 2009 I have been running a VPS Xen server on vpslink for just over a year now, and am sufficiently satisfied with their service to start giving them six months of money at a time and increase my server to 128M so I can also run Apache. So I am now running a lightly loaded e-mail server and a web server on my vpslink VPS.

There have been some issues with uptime in the past year, but their tech support has always been prompt in addressing outages, and generally I believe them to be a good company to work with.

Just for the record, I bumped into another ISP[1] that challenges their low price-point with a 512M OpenVZ VPS for only $6 per month. (I believe that OpenVZ means no swap.....) If this is not a temporary teaser price, they might be worth checking out at some point.

[1] http://w4networks.com/vps_servers.html

posted at: 13:43 | path: /Hosting/vpslink | permanent link to this entry

/Hosting/vpslink: Review of Virtual Private Server (VPS) at vpslink.com:

As of summer 2008, I have been using a vpslink.com[1] VPS for a about a month now, and so far, all is well, especially the price.

If your needs are simple, their 64 MB RAM / $7.95 plan is a good deal. If you choose XEN hosting, you also get some swap. I currently have postfix / dovecot / MySQL and a couple other minor daemons running in this skinny little RAM no problem, no swapping (and as yet, very little load! I am still setting up....)

There is a wide choice of Linux distros to choose from for your VPS, I chose Debian.

This is my first VPS, so I can't really compare with other service offerings. I can say that getting installed and setup was trivial, and thereafter everything works just like the same old Debian I am used to.

The only snafu so far is some unnecessary password confusion. When you start up with the service, the same password works everywhere. However, if you start changing passwords, you will soon find that your newly changed password does not necessarily apply everywhere you think it should. And most confusing of all, at least the first time, if you reboot your server from the Control Panel on their website, the root password gets reset to the original password. Lesson: keep that original password safe, you might need it for a long time.

I will keep this post up-to-date with my on-going experience.

If you register for an account, be sure to use my referral code: F5H33Z. You will get a 10% discount, and I will get a few pennies as well.

[1] http://vpslink.com/?ref=F5H33Z

posted at: 13:35 | path: /Hosting/vpslink | permanent link to this entry

Fri, 14 Aug 2009

/Hosting/Amazon/EC2: How to Snapshot an Amazon EC2 Server to S3

This is actually a script that will do the job automatically, if called periodically from cron:

Note that /mnt/pk-xxx.pem & /mnt/cert-xxx.pem are your Amazon account keys, which need to be uploaded to the server. Other keys are embedded in this script. Make your file permissions restrictive. In the end, this is a Python script only because of the

response = conn.register_image(manifest)

line. I am using a Debian AMI published by alestic.com[1] and it comes equipped with a small set of Ruby tools capable of bundling a server and uploading it to S3. And it is those tools that my script is calling up to the point of the above line. However, these Ruby tools do not have the capability of registering the uploaded bundle as an AMI, which is necessary to have it show up ready for activation in the MyAMIs list of ElasticFox, for instance.

My script does not as yet weed out old backup images. I will probably continue to do that manually, as I only plan on snapshotting the server once or twice a month (volatile data is all stored in an EBS volume which is backed-up daily, as it should be). There is a wee bit of a gotcha in the process of deleting an old AMI: one cannot delete an S3 bucket that is not empty. and the s3cmd delete command used to delete bucket contents does not seem to accept wild cards. The solution is not well documented:

s3cmd del --recursive --force s3://bucket-name/
s3cmd rb s3://bucket-name/

and the AMI named "bucket-name" is gone forever.

[1] http://alestic.com/

posted at: 09:09 | path: /Hosting/Amazon/EC2 | permanent link to this entry

Tue, 28 Jul 2009

/Hosting/Amazon/EC2: My Migration Process: An Old (Crashed) Amazon EC2 Server to a Newer Version

(Note: these are abbreviated notes with some detail deliberately left out.)

When one updates the software on an Amazon EC2 server, not everything gets upgraded (the kernel, for one very important example). When one starts an EC2 server for the first time, the original bundle usually comes from someone or some organization that is actively maintaining it, and periodically issuing updates. Therefore it is good practice to occasionally move one's server to a more current bundle ("AMI", in Amazon parlance).

Amazon's EC2 servers have some other unique characteristics, one of them being that data and the OS are kept in separate "partitions", and backed up by different processes. The server backup is quite a bit more onerous in terms of time and steps, if it has not yet been scripted (and I have not....) so it tends to not happen as often and the current server can become significantly different from the last backup. And when an EC2 server stops or crashes for whatever reason, the delta from the last backup is irrevocably lost.

All this to say, sometimes when there is a server problem, it is more expedient to create a new server then to restart an old backup:

Setup a new server:

Go to Alestic[1] and find out what the latest version of Debian Stable is.
Start up a new instance.
Create an EBS volume from the latest snapshot of the volume still attached to the hosed server.
Attach the new volume to the new server.
On the new server, mount the volume on /vol.

On the new server, /vol now contains my MySQL databases and /var/www. Sym link these into the standard locations in the new server's file structure.

/vol also contains a /vol/etc copy of /etc (no more then 24 hours old). 'cp -a' the backed up versions of the apache2, php5, and mysql directories into /etc.

'apt-get install apache2 libapache2-mod-fastcgi mysql-server php5 php5-cli pyblosxom postfix'
(I prefer postfix over exim.)

Correct any ownership issues in /vol with the www and mysql directories (they were just mounted on another server).

Move Elastic IP from old server to new server.

Install ntpdate and add this to root crontab:
11 */6 * * * /usr/sbin/ntpdate us.pool.ntp.org

Install etckeeper:

cd /etc
etckeeper init
etckeeper commit
cd /vol
git clone /etc

Now add this to root crontab:
15 1 * * * cd /vol/etc && git pull

Create a user account:

Edit /etc/aliases to forward root mail to user.
run "newaliases"
/etc/init.d/postfix restart
Add a .forward file to user account to send mail out.

[1] http://alestic.com/

posted at: 14:15 | path: /Hosting/Amazon/EC2 | permanent link to this entry

Sun, 12 Jul 2009

/Hosting/NearlyFreeSpeech: MySQL For NearlyFreeSpeech.net Power Users

To get a MySQL command line:

mysql --user=your-user-name --host=your-database-name.db

And yes you can automatically back up you MySQL database, by yourself, for free. There are just a couple hoops to jump through.

First set up SSH passwordless authentication as described here[1], with the modification that you cannot install the public key yourself on the NearlyFreeSpeech.net end. You must send them a "Support Request" (click on the Support tab after logging in) with the public key, and they will register it within a few hours[2].

Their FAQ kind of tip toes around the issue without giving a clear answer: [3][4][5]. The member wiki has the real solution[6]. Create a /home/private/.my.cnf file in your nearlyfreespeech.net account:

[client]
host = put_your_DSN_here
user = put_your_username_here
password = put_your_password_here

Then on the machine where you want to store the database backup (your desktop?) execute the following:

ssh johndoe_thesite@ssh.phx.nearlyfreespeech.net 'mysqldump thedatabase | bzip2 -9' > ~/backups/thedatabase_backup-`date +%Y-%m-%d`.sql.bz2

Note that "mysqldump" is found in the "mysql-client" package, at least in Debian. And of course, to automate, stick the above in your crontab ("crontab -e").

(Note: replace "user" below with your nearlyfreespeech.net username.)
[1] http://blog.langex.net/index.cgi/Admin/SSH-SSL/passwordless-ssh-authentication.html
[2] https://members.nearlyfreespeech.net/user/support/faq?q=SSHKeys#SSHKeys
[3] https://members.nearlyfreespeech.net/user/support/faq?q=MySQLForward#MySQLForward
[4] https://members.nearlyfreespeech.net/user/support/faq?q=MySQLLocalhost#MySQLLocalhost
[5] https://members.nearlyfreespeech.net/user/support/faq?q=MySQLCommandLine#MySQLCommandLine
[6] https://members.nearlyfreespeech.net/wiki/HowTo/DatabaseBackup

posted at: 14:58 | path: /Hosting/NearlyFreeSpeech | permanent link to this entry

Sat, 13 Jun 2009

/Hosting/NearlyFreeSpeech: BackupPC and nearlyfreespeech.net

Just in case anyone else has the bright idea of trying to use their existing BackupPC backup server to backup websites on nearlyfreespeech.net, let me save you some time: it will not work.

nearlyfreespeech.net assigns different IP addresses to different web services. So you can ping your web host, for example, at xxxx.nfshost.com, but the SSH server you connect to in order to gain SSH access to your hosting directory is on a different IP at ssh.phx.nearlyfreespeech.net. If you try to ping ssh.phx.nearlyfreespeech.net you will get no response.

For better or worse, BackupPC insists on a backup client being pingable before it will attempt a backup.

If anyone has a bright idea, I am all ears....

In the meantime, rdiff-backup[1] works nicely.

[1] http://blog.langex.net/index.cgi/Admin/backups/rdiff-backup/

posted at: 09:30 | path: /Hosting/NearlyFreeSpeech | permanent link to this entry

Thu, 23 Apr 2009

/Hosting/NearlyFreeSpeech: Review of Web Hosting at Nearly Free Speech.net[1]:

I have been hosting several websites with nearlyfreespeech.net for over a year now, and believe I have found the perfect host for small sites. They may in fact be perfect for large sites as well, but I don't personally have one to try them on with.

Basically they provide a great service at an incredibly cheap price. The cheap price comes from the fact that you pay as you go for both disk storage (US$0.01 / MB / day) and bandwidth (sliding scale[2] starting at $1/G), ie. the bigger the site, or the busier the site, the larger the monthly bill will be. For a small static site without a lot of activity, you could easily pay as little as ten cents per month for hosting. I have several small modestly active sites, including this blog and:

http://www.doesharleysuck.com/
http://www.langex.net/

One of the sites uses MySQL, which I believe costs one cent per day. Combined, all of my sites have been costing me about one dollar per month to keep running.

Using Paypal, you can add as little as US$0.25 to your account at a time (they take a service charge, I think six cents). You may reduce the percentage of the service charge by increasing the size of the deposit, ie. there is a thirty cent service charge for a US$5 deposit.

Servers are FreeBSD, and you get full Unix shell account access with your account. Any time I have reached for a standard UNIX utility, it has been there: Midnight Commander, Unison, and nano come to mind. In addition to FTP, there is also SSH access to the account.

The only thing I have wanted and found them lacking was the Apache modpython module. That may be a FreeBSD limitation, I don't know.

The service and website have been so flawless that I literally have not once felt the need to try to contact support.

[1] https://www.nearlyfreespeech.net/
[2] https://www.nearlyfreespeech.net/services/hosting.php#pricing

posted at: 13:31 | path: /Hosting/NearlyFreeSpeech | permanent link to this entry

Mon, 30 Mar 2009

/Hosting/Amazon/EBS: Amazon Backups: Snapshotting EBS to S3

Amazon "Compute Cloud" services are setup such that storage for a running server is kept on an EBS ("Elastic Block Storage") volume mounted on the server. Long-term, offsite (geographic redundancy across multiple data centers) storage is kept in S3. There is a function to snapshot an EBS volume into S3.

List the volumes associated with your account:

ec2-describe-volumes

This will also show what volumes are mounted on which servers. Stop the services running on the server that might be writing to the EBS (databases in particular, if you wish to have a usable backup):

/etc/init.d/apache2 stop
/etc/init.d/mysql stop

Create a snapshot:

ec2-create-snapshot vol-3159bd58

Check status of snapshot:

ec2-describe-snapshots
SNAPSHOT snap-e309e38a vol-3159bd58 completed

Restart services:

/etc/init.d/mysql start
/etc/init.d/apache2 start

To restore file(s) from one of the backup snapshots, simply create a new volume from the chosen snapshot. Attach it to instance of your running server. Mount the volume temporarily on the server and grab your files. Then unmount the volume, detach it, and delete it. A process that takes only a couple minutes, especially if you use the extremely convenient Firefox ElasticFox plugin[1][2].

[1] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609
[2] http://s3.amazonaws.com/ec2-downloads/elasticfox.xpi

posted at: 06:33 | path: /Hosting/Amazon/EBS | permanent link to this entry

Thu, 26 Mar 2009

/Hosting/Godaddy: Godaddy's Domain Forwarding is Broken

(Godaddy Incident ID #5828327, should anyone care....)

I do not host any sites at Godaddy[2], but for quite some time now I have used them as a Domain registrar and sometimes DNS service.

"Sometimes" DNS because I have had more then one occasion to conclude their DNS does not work very well, particularly the domain forwarding. The most recent evidence is a sub-domain that is forwarded to the same server as the domain and another sub-domain. The domain and the other sub-domain work perfectly. For no apparent reason this latest sub-domain "sorta" works: depending on where in the world you are located, you might see the website, or you might get a timeout. When presented with traceroutes from multiple locations, even within the USA, that do timeout, Godaddy technical support drags its feet and repeats, and I paraphrase: "everything works on our network, the rest of the world is broken, go fix the rest of the world".

I conclude that Godaddy does not make much money from customers that only use their domain registration service, and they do not really care very much whether their DNS works for sites hosted elsewhere. Perhaps it is even deliberately broken? It is a wee bit suspicious and improbable that forwarding the main domain and the first sub-domain just works, and all other sub-domains thereafter seem to be subtly sabotaged.

Godaddy tech support do respond to their e-mails in a timely manner, I will give them that. But at this point I would emphatically *not* recommend them as a registrar, or much else, for that matter. There are too many other options out there, and some of them[1] are considerable cheaper, too.

[1] https://www.nearlyfreespeech.net/
[2] http://www.godaddy.com/

posted at: 12:25 | path: /Hosting/Godaddy | permanent link to this entry

Wed, 18 Feb 2009

/Hosting/Amazon/EBS: Keep Your Amazon Server's Data on EBS

Amazon's EC2 servers do not provide persistent storage. If the server is turned off or even perhaps halted, then all data since the server was initialized is lost. (I am pretty sure one can reboot an Amazon server without losing data, but have not tried....)

Therefore, at the least, all volatile data should be stored on an Amazon "Elastic Block Storage" (EBS) volume. As a further incentive, Amazon says[1] that EBS is faster then the storage associated directly with an Amazon server, and that it is replicated across multiple servers in the data center. The downside, of course, is that you pay for EBS, both by gigabyte per month and by I/O per month.

Here[2] is an excellent tutorial on how to use EBS with a MySQL database, and here[3] is a nice general introduction.

I am going to move my Amazon Server's /var/www/ onto EBS. First use ec2-describe-instances to find out which Amazon zone the server is located in[4]: us-east-1c. Now create a 5G EBS volume:

ec2-create-volume -z us-east-1c -s 5

From what I can see so far, changing the volume size probably involves snapshotting the volume, and then restoring the snapshot to a bigger volume. This small headache must be traded-off against the EBS cost of 10 cents per gigabyte per month.

ec2-describe-volumes will tell you when the new volume is available. Now attach the volume to the server instance at /dev/sdz:

ec2-attach-volume -d /dev/sdz -i i-c1a320a8 vol-3159bd58

Now login to the server and format the volume:

mkfs.ext3 /dev/sdz

(Note that xfs has apparently been a popular format choice in the past because of its explicit freezability for snapshots, but I have seen in more then one place now that Amazon EC2 has problems with xfs.)

Setup /etc/fstab by adding the following entry:

/dev/sdz /vol ext3 noatime 0 0

And mount the volume:

cd /
mkdir vol
mount vol

Now move the contents of /var/www to the volume:

/etc/init.d/apache2 stop
cd /var
mv www /vol/
ln -s /vol/www/ www
/etc/init.d/apache2 start

[1] http://aws.amazon.com/ebs/
[2] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1663
[3] http://blog.rightscale.com/2008/08/20/amazon-ebs-explained/
[4] http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1347

posted at: 07:35 | path: /Hosting/Amazon/EBS | permanent link to this entry

Home

Contact

Resume

Links

Categories:

Archives:

Subscribe XML RSS Feed