Archive for the ‘AWS’ Category

AWS: Gaining SSH access to an EC2 instance you lost access to.

Posted on April 9th, 2013 in AWS | 4 Comments »

I had a situation where an employee created a new EC2 instance with his keypair and was out the next day. I needed access to it immediately, so this posed a problem. Here is how I gained SSH access via AWS web console: I detached the EBS drive, mounted it on another EC2 instance I did have access to, added my ssh pub key to ~ec2-user/.ssh/authorized_keys, then reattached it back to the old instance. Amazing the ideas that strike you in the event of an emergency.

As long as you have full AWS console access and some light unix chops, it should be fairly straightforward:

  1. Go to the Amazon EC2 control panel, and click “Volumes” (under Elastic Block Store). Look at the attachment information for the old EBS volume to find the EBS attached to the old EC2 instance.
  2. Detach it and attach it to an EC2 instance you have SSH access to via this web console. Keep note the path, probably /dev/sda1. You will have to reconnect it to this path later and AWS doesn’t always guess correctly. When you attach it to your other EC2 it will probably attach as /dev/sdf or something since /sda is taken by the root drive. You can see this in the EBS Volumes table under “Attachment information”. This will be something like (<instance name>):/dev/sdg
  3. Use SSH and connect to your good instance. Type
    mkdir /mnt/oldvolume and then sudo mount /dev/sdf /mnt/oldvolume (or whatever the path given in the attachment information panel was). Your files should now be available under /mnt/oldvolume.

  4. Add your pub ssh key to /home/ec2-user/authorized_keys.
  5. Unmount the volume with umount -d /dev/sdf and follow the above steps to reattach it back. You should be able to login to ec2-user now on your old box.

Resize the root EBS disk on an EC2 instance

Posted on January 16th, 2013 in AWS | No Comments »

Running out of space on your EC2 instance? Here’s a great writeup on resizing the root EBS disk to something beefier. It requires having installing Amazon command line tools through another EC2 instance you have running. Works great.

Securing your SSH on Centos, especially on EC2.

Posted on December 6th, 2012 in AWS, security, ssh | No Comments »

The popular port.

Example brute force attempt on an EC2 server.

By default SSH runs on port 22 on your server. Though I’m not a proponent of security by obscurity, moving your SSHd to a high-number port (like 3293) will reduce the number of brute force attempts on your SSH by nearly 100% (Stats given by reviewing /var/log/secure on servers I have or have had access to for years.) This used to be pretty important but even more so with cloud services like Amazon EC2 where hackers and script kiddies can ping sweep whole subnets of Amazon’s elastic IP ranges. (On a related note, Stackoverflow blocked all EC2 instances from accessing their site using the same approach.)


If for some reason though, you cannot move SSH ports—for example, you’re working in a legacy system where dozens or hundreds of people are already connecting to, the next best thing is to install DenyHosts, a free tool written in Python that reviews and bans IP addresses that make too many failed attempts.

Installing DenyHosts

Centos doesn’t have direct access to installing it. But you can install it by running the following:

sudo rpm -Uvh
sudo yum install denyhosts

After that, add your IP address to /etc/hosts.allow in the format of sshd: so that it will never blacklist your IP, tweak /etc/denyhosts.conf to your liking, and restart denyhosts with sudo service denyhost restart. It started analyzing your /var/log/secure and banning ips pretty instantly as you can see from this screenshot. After a few weeks it can optionally purge them so the ban list doesn’t get super big.

DenyHosts auto blocking IP addresses.

The folks at Digital Ocean have a great entry that can walk you through the rest.

Note: The one downside of switching SSH ports is that you can no longer use git in it’s common syntax since git uses SSH as it’s secure transport. See my article on using git on non-standart ports to solve that easily.

Hardening your /etc/ssh/sshd_config

There are a few adjustments you should make to your SSHd config so that it’s harder for the wrong person to gain access. This includes disabling password-based logins (everyone should be using SSH keys), disabling root user login, and throwing in a welcome message that this SSH session is being logged:

# Disable password based login. Only SSH keys are allowed.
PasswordAuthentication no
# Add a message banner notifying this server is logged.
Banner /etc/ssh.go.txt
# Do not allow root to login
PermitRootLogin no
# Deny root user login
DenyUsers root

Following the above steps should make your SSH more secure. Happy SSHing!

Amazon Elastic Load Balancer vs HAProxy

Posted on November 22nd, 2012 in AWS, unix | No Comments »

When it comes time to horizontally scale up your application with multiple server instances on AWS you generally have three choices: DNS round-robin (simply assigning multiple A names to a domain), using Amazon Elastic Load Balancer, or using one or more HAProxy‘s on EC2. The Robin-robin approach, while easiest, is not good since DNS will always forward round-robin to an IP even to a dead or overloaded EC2 instance, hence, a real HTTP load balancer is needed.

Amazon ELB is pretty awesome as it’s inherently fault tolerant and highly available. You never have to worry about scaling it up like regular EC2 instances. It just works, and it’s relatively cheap. But it also has a few drawbacks including lacking complex routing algorithms, no logging outside of Cloudwatch, and can’t gracefully handle great spikes in traffic like a Slashdot or a consumer flash sale without “prewarming” the load balancer (an act that requires a premium support agreement and contact with Amazon ahead of time).

HAProxy, the premiere open-source load-balancer contains a swiss army knife of functionality, including tons of routing algorithms, that addresses many of the limitation of Amazon ELB at the cost of a lot more manual setup, complexity, and cost (since it’s hosted directly on EC2 instance(s). This article is a great discussion on the merits of both and worth a read if you’re like to know which to go with.

But wait there’s more! What if you could have the best of both worlds? The following article discusses pairing both ELC and HAproxy together.

Lastly, if you don’t want to get into the complexity that is HAProxy, you can always try load balancing through Nginx. There’s less fine-grained controls but it’s totally useable as an efficient load balancer.