Clayton's Tech Bits

Home

Contact

Resumé / C.V.

Links

Search this site:
Custom Search

Categories:

/ (224)
  Admin/ (86)
    Apache/ (7)
      HTTPS-SSL/ (4)
    Cherokee/ (1)
    LAN/ (4)
    LVM/ (3)
    Monitoring/ (2)
      munin/ (2)
    OpenVPN/ (1)
    SSH-Proxy/ (3)
    SSH-SSL/ (6)
    backups/ (16)
      SpiderOak/ (1)
      backuppc/ (5)
      dirvish/ (1)
      misc/ (6)
      rdiff-backup/ (1)
      rsync/ (1)
      unison/ (1)
    commandLine/ (11)
    crontab/ (1)
    databases/ (8)
      MSSQL/ (2)
      MySQL/ (5)
      PostgreSQL/ (1)
    dynamicDNS/ (2)
    email/ (9)
      Dovecot/ (1)
      deliverability/ (1)
      misc/ (1)
      postfix/ (6)
    iptables/ (2)
    virtualization/ (8)
      VMware/ (1)
      virtualBox/ (7)
  Coding/ (11)
    bash/ (1)
    gdb/ (1)
    git/ (2)
    php/ (4)
    python/ (3)
      Django/ (1)
  Education/ (1)
  Hosting/ (23)
    Amazon/ (14)
      EBS/ (3)
      EC2/ (11)
    Godaddy/ (2)
    NearlyFreeSpeech/ (3)
    Rackspace/ (1)
    vpslink/ (3)
  Linux/ (20)
    Awesome/ (3)
    CPUfreq/ (1)
    Chinese/ (1)
    Debian/ (5)
      WPA/ (1)
    audio/ (1)
    encryption/ (2)
    fonts/ (1)
    misc/ (4)
    router-bridge/ (2)
  SW/ (39)
    browser/ (2)
      Chrome/ (1)
      Firefox/ (1)
    business/ (25)
      Drupal/ (8)
      KnowledgeTree/ (6)
      Redmine/ (2)
      SugarCRM/ (6)
      WebERP/ (2)
      eGroupware/ (1)
    email/ (1)
    fileSharing/ (1)
      mldonkey/ (1)
    graphics/ (2)
    research/ (2)
    website/ (6)
      blog/ (6)
        blosxom/ (3)
        rss2email/ (1)
        webgen/ (1)
  Security/ (12)
    IMchat/ (1)
    circumvention/ (2)
    e-mail/ (4)
    greatFirewall/ (1)
    hacking/ (1)
    password/ (1)
    privacy/ (1)
    skype/ (1)
  Services/ (1)
    fileSharing/ (1)
  TechWriting/ (1)
  xHW/ (13)
    Lenovo/ (1)
    Motorola_A1200/ (2)
    Thinkpad_600e/ (1)
    Thinkpad_a21m/ (3)
    Thinkpad_i1300/ (1)
    Thinkpad_x24/ (1)
    USB_audio/ (1)
    scanner/ (1)
    wirelessCards/ (2)
  xLife/ (17)
    China/ (9)
      Beijing/ (5)
        OpenSource/ (3)
    Expatriation/ (1)
    Vietnam/ (7)

Archives:

  • 2012/03
  • 2012/01
  • 2011/12
  • 2011/11
  • 2011/10
  • 2011/09
  • 2011/08
  • 2011/07
  • 2011/06
  • 2011/05
  • 2011/04
  • 2011/02
  • 2010/12
  • 2010/11
  • 2010/10
  • 2010/09
  • 2010/08
  • 2010/07
  • 2010/06
  • 2010/05
  • 2010/04
  • 2010/03
  • 2010/02
  • 2010/01
  • 2009/12
  • 2009/11
  • 2009/10
  • 2009/09
  • 2009/08
  • 2009/07
  • 2009/06
  • 2009/05
  • 2009/04
  • 2009/03
  • 2009/02
  • 2009/01
  • 2008/12
  • 2008/11
  • 2008/10
  • 2008/09

    • Subscribe XML RSS Feed

    Fri, 14 Aug 2009

    /Hosting/Amazon/EC2: How to Snapshot an Amazon EC2 Server to S3

    This is actually a script that will do the job automatically, if called periodically from cron:

    #!/usr/bin/env python print "Snapshot Amazon server:" import time bucketName = "server-" + time.strftime("%Y%m%d") print "First create the bundle:" import os os.system("ec2-bundle-vol -k /mnt/pk-xxx.pem -c /mnt/cert-xxx.pem -u {Amazon_user_id} -d /mnt --arch i386") print "Create an S3 bucket named " + bucketName command = "s3cmd mb s3://" + bucketName os.system(command) print "Now upload the server image to the bucket" command = "ec2-upload-bundle -b " + bucketName + " -m /mnt/image.manifest.xml -a {aws-access-key-id} -s {aws-secret-access-key-id}" os.system(command) # boto required here because "ec2-register" does not exist on the server print "Now register the image as an official (private) AMI" from boto.ec2.connection import EC2Connection conn = EC2Connection('<aws_access_key_id>', '<aws_secret_access_key>') manifest = bucketName + "/image.manifest.xml" response = conn.register_image(manifest) print response print "cleanup /mnt (otherwise next server snapshot will break)" os.system("rm -r /mnt/im*")

    Note that /mnt/pk-xxx.pem & /mnt/cert-xxx.pem are your Amazon account keys, which need to be uploaded to the server. Other keys are embedded in this script. Make your file permissions restrictive. In the end, this is a Python script only because of the

    response = conn.register_image(manifest)

    line. I am using a Debian AMI published by alestic.com[1] and it comes equipped with a small set of Ruby tools capable of bundling a server and uploading it to S3. And it is those tools that my script is calling up to the point of the above line. However, these Ruby tools do not have the capability of registering the uploaded bundle as an AMI, which is necessary to have it show up ready for activation in the MyAMIs list of ElasticFox, for instance.

    My script does not as yet weed out old backup images. I will probably continue to do that manually, as I only plan on snapshotting the server once or twice a month (volatile data is all stored in an EBS volume which is backed-up daily, as it should be). There is a wee bit of a gotcha in the process of deleting an old AMI: one cannot delete an S3 bucket that is not empty. and the s3cmd delete command used to delete bucket contents does not seem to accept wild cards. The solution is not well documented:

    s3cmd del --recursive --force s3://bucket-name/
    s3cmd rb s3://bucket-name/

    and the AMI named "bucket-name" is gone forever.

    [1] http://alestic.com/

    posted at: 09:09 | path: /Hosting/Amazon/EC2 | permanent link to this entry