AWSCLI tips

Snippets I’ve used or collected.


Configure awscli

Enter keys, region and output defaults.

$ aws configure
AWS Access Key ID [****************FEBA]:
AWS Secret Access Key [****************I4b/]:
Default region name [ap-southeast-2]:
Default output format [json]:

Add bash completion via Tab key

echo "complete -C aws_completer aws" >> ~/.bash_profile
source ~/.bash_profile

Other shell completion options

Supecharged AWSCLI SAWS or AWS-shell

Both offer improvements over the standard bash completions above.


S3 Accelerate Transfer
if you need to sync a large number of small files to S3, the increasing the following values added to your ~/.aws/config config file will speed up the sync process. (also need to enable accelerate transfer on bucket – see next command)

Modify .aws/config

[profile default]
...
s3 =
  max_concurrent_requests = 100
  max_queue_size = 10000
  use_accelerate_endpoint = true

Enable S3 bucket for Accelerate Transfer

The following example sets Status=Enabled to enable Transfer Acceleration on a bucket. You use Status=Suspended to suspend Transfer Acceleration.  This incurs an additional cost, so should be disable about the bulk upload is completed.

 $ aws s3api put-bucket-accelerate-configuration --bucket bucketname --accelerate-configuration Status=Enabled


Set default output type (choose 1)

[default]
output=text|json|table

Wait

#!/bin/bash
echo "Waiting for EBS snapshot"
aws ec2 wait snapshot-completed --snapshot-ids snap-aabbccdd
echo "EBS snapshot completed"

Get the public IPs or DNS of EC2 instances

Uses jq for filtering output.

apt-get install jq
aws ec2 ls --ec2-tag-value mystack-inst* | jq --raw-output ".Reservations [].Instances [].PublicIpAddress

AWS Tags – jq improvements

With jq, AWSCLI json output can be hard to query can but you can map it into a normal object like this:

jq '<path to Tags> | map({"key": .Key, "value": .Value}) | from_entries'

 
Find untag EC2 instance. 

This cmd finds instance NOT tagged ‘owner’.

aws ec2 describe-instances   –output text    –query ‘Reservations[].Instances[?!not_null(Tags[?Key == `owner`].Value)] | [].[InstanceId]’
Create a tag for EC2 instance

 NB. Add a for loop to tag all instances

aws ec2 create-tags –resources $i –tags Key=owner,Value=me

Advertisements

cloud9 IDE http://c9.io

If you need a Lnux Virtual Desktop for developers, then check out http://c9.io

Within 5 minutes, I had registered, logged in, view the sample code, deployed it and an Apache web server and run the code. Super Easy!!

Some of the constraints of other solutions may be;

* constraints
VMWare View 6 (v1.0 linux desktops)
Hardware purchases
Building and Managing infrastructure
Short cycle project work

Easy way to rotate AWS access keys

We all know we should change passwords often, well same goes for access keys.

This handy INTERACTIVE bash script walks you through to create a new AWS Access Key, save the .pem file in your .ssh directory.  And gives you the option to delete the old keys.

You can download from my gitlab – here

Hope this helps someone 🙂

#!/bin/bash
# @shallawell
# Program Name: aws-iam-access-keys.sh
# Purpose: Manage AWS access keys
# version 0.1

# new key content will be created in this file.
FILE=new_aws_key.txt
#remove the old file first
rm $FILE
### create a key
echo -n "Do you want to create a new Access/Secret key. (y/n) [ENTER]: "
#get user input
read response2
if [ "$response2" == "y" ]; then
echo "Ok.. Creating a new keys !!!"
aws iam create-access-key --output json | grep Access | tail -2 | tee -a $FILE
#Alternative create key command
#KEY=myIndiaAWSKeytest
#REGION=ap-south-1
#aws ec2 create-key-pair --key-name=$KEY --region $REGION --query="KeyMaterial" --output=text > ~/.ssh/$KEY.pem
#readonly the key
#chmod 400 ~/.ssh/$KEY.pem
echo "key created."
echo "REMEMBER: You should rotate keys at least once a year! Max of 2 keys per user."
echo "$FILE created for Access and Secret Keys"
echo "HINT: Run aws configure to update keys. (you just rotated your keys!)"
else [ "$response2" == "n" ]
echo "Not creating keys."
exit 0
fi

### list a key, save to DELKEY var
#this command LISTS the access keys for current user, sorts by CreateDate,
#gets the latest AccessKeyId result. awk grabs the Access key (excludes date field)
DELKEY=$(aws iam list-access-keys \
--query 'AccessKeyMetadata[].[AccessKeyId,CreateDate]' \
| sort -r -k 2 | tail -1 | awk {'print $1'})

echo "list-Access-key sorted to find OLDEST key."
echo -n "Key Found : $DELKEY. Do you want to delete this key. (y/n) [ENTER]: "
#get user input
read response
if [ "$response" == "y" ]; then
echo "you said yes. Deleteing Key in 3 secs!!!"
sleep 3
echo "delete-access-key disabled, NO REAL DELETE OCCURRED"
### delete a key, uncomment to activate the delete function.
#aws iam delete-access-key --access-key-id $DELKEY
echo "deleted $DELKEY"
else [ "$response" == "n" ]
echo "you said no. Not Deleting"
fi

echo "done."

GIT – setup a new private repo and pull,push code

This is suitable for GITLAB, but with a little modification would work with other GITs.

Install git on Ubuntu

apt-get install -y git

How to setup git

git config --global user.name "name"
git config --global user.email "email@example.com"
git remote add origin http://IP/path/to/repository

How to create new PRIVATE repo (for GITLAB)

This needs to be done via the GITLAB API.  Grab your Personal Token from GITLAB Account Settings and use this bash script.

Get it here https://gitlab.com/shallawell/git-init-remote-project.git

#!/bin/bash
# @shallawell
# Program Name: git-init-remote-project.sh
# Version = 0.0.2
# if the remote repo does not exist, use this script before you attempt to
# git push
# Can create a simple Gitlab repo
# start ===============================
#
#set vars
REPO_NAME=$1
TOKEN=YOURTOKENHERE
REMOTE_GIT=https://gitlab.com
#Public or Private repo, set to true for PUBLIC REPO
PUBLIC=true

# test var
if [ $TOKEN == YOURTOKENHERE ]; then
 echo "Error : update the git TOKEN variable."
 exit 0
fi

#usage
if [ ${#@} == 0 ]; then
 echo "Error : Repo name required."
 echo "Usage: $0 REPO_NAME"
 echo "You need to provider the repostiory/project name to create on the remote server."
 echo "Ensure $TOKEN has been set."
else
# main command
 curl -H "Content-Type:application/json" $REMOTE_GIT/api/v3/projects?private_token=$TOKEN -d \
 "{ \"name\": \"$REPO_NAME\", \"public\": $PUBLIC }"
fi
# end ===============================

How to Pull, Checkout and Push

git pull <repo-link>
git checkout -b master
<make your code changes>
<make changes to .gitignore to exclude files>
git add .
git commit -m "updated the code to/for ..."
git show
git push origin master

Other GIT handy commands

git fetch <repo-name>       # get changes, rather than pull whole code base
git init                    # initialize new local dir
git status                  # print status
git remote get-url origin   # check the origin

Nice list of Aliases for .bashrc

$cat .bashrc
# .bashrc

# Source global definitions
if [ -f /etc/bashrc ]; then
 . /etc/bashrc
fi

# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=

# User specific aliases and functions
alias cp='cp -iv' # Preferred 'cp' implementation
alias mv='mv -iv' # Preferred 'mv' implementation
alias mkdir='mkdir -pv' # Preferred 'mkdir' implementation
alias ll='ls -FGlAhp' # Preferred 'ls' implementation
alias la='ll -FGlAhpa' # Preferred 'ls -a' implementation
alias less='less -FSRXc' # Preferred 'less' implementation
cd() { builtin cd "$@"; ll; } # Always list directory contents upon 'cd'
alias cd..='cd ../' # Go back 1 directory level (for fast typers)
alias ..='cd ../' # Go back 1 directory level
alias ...='cd ../../' # Go back 2 directory levels
alias .3='cd ../../../' # Go back 3 directory levels
alias .4='cd ../../../../' # Go back 4 directory levels
alias .5='cd ../../../../../' # Go back 5 directory levels
alias .6='cd ../../../../../../' # Go back 6 directory levels
alias findall='find / -name' # find on the whole filesystem
alias sudo="sudo " # A trailing space in value causes the next word to be checked for alias substitution when the alias is expanded

# lr: Full Recursive Directory Listing
# ------------------------------------------
alias lr='ls -R | grep ":$" | sed -e '\''s/:$//'\'' -e '\''s/[^-][^\/]*\//--/g'\'' -e '\''s/^/ /'\'' -e '\''s/-/|/'\'' | less'

# showa: to remind yourself of an alias (given some part of it)
# ------------------------------------------------------------
 showa () { /usr/bin/grep --color=always -i -a1 $@ ~/Library/init/bash/aliases.bash | grep -v '^\s*$' | less -FSRXc ; }

alias mypubip='curl -s ipinfo.io/ip' # mypubip: Public facing IP Address

alias openPorts='ss -t -a -l' # openPorts: List Listening TCP ports

# ii: display useful host related informaton
# -------------------------------------------------------------------
 ii() {
 echo -e "\nYou are logged on ${RED}$HOST"
 echo -e "\nAdditionnal information:$NC " ; uname -a
 echo -e "\n${RED}Users logged on:$NC " ; w -h
 echo -e "\n${RED}Current date :$NC " ; date
 echo -e "\n${RED}Machine stats :$NC " ; uptime
 echo -e "\n${RED}Public facing IP Address :$NC " ;mypubip
 echo
 }

# httpDebug: Download a web page and show info on what took time
# -------------------------------------------------------------------
 httpDebug () { /usr/bin/curl $@ -o /dev/null -w "dns: %{time_namelookup} connect: %{time_connect} pretransfer: %{time_pretransfer} starttransfer: %{time_starttransfer} total: %{time_total}\n" ; }

Free SSL – Secure your website

letsencrypt.org

Get a free SSL Certificate for use with your sites.


These notes are pretty rough and really a reference for me.

I am using a Bitnami Joomla stack and hence I edit bitnami.conf (instead of httpd.conf)

###letsencypt install
## Main point is DNS MUST resolve to right IP
## Cannot put wiki.name.com and http://www.name.com on same certificate as they are on different IPs.
##
### Make sure nothing is listening on Port 80 as we start our own web-server.
sudo apt-get update
sudo apt-get -y install git
sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt
./letsencrypt-auto certonly -a standalone -w /opt/bitnami/apache2/htdocs/ -d yourdomain.com -d http://www.yourdomain.com

sudo crontab -e
#Add this to crontab for auto renewal
30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
####
#Update SSL Cert in Apache (httpd.conf/bitnami.conf)
# remove old and add these
SSLCertificateFile “/etc/letsencrypt/live/yourdomain.com/cert.pem”
SSLCertificateKeyFile “/etc/letsencrypt/live/yourdomain.com/privkey.pem”
SSLCertificateChainFile “/etc/letsencrypt/live/yourdomain.com/fullchain.pem”

Force SSL in Apache
From here > https://wiki.bitnami.com/Components/Apache#How_to_force_HTTPS_redirection_for_my_application.3f
# Add these to Apache (httpd.conf/bitnami.conf)
#<VirtualHost *:80>
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/(.*) https://%{SERVER_NAME}/$1 [R,L]

Python example : Test script / module using regex

This is a simple piece of code to test if sys.argv[1] matches a file type using a regex to match a specific file type, in this case *.flac (it doesn’t actual check to see if the file is an actual Flac file, just that it is named correctly).

Here is the code with plenty of comments.

Also included (commented out) is the logic to match a date string.

Happy coding.

#!/usr/bin/env python3
# Author : shallawell@gmail.com, @shallawell
# A test function to check a if argv contains *.flac
# Filesname: testFLAC.py
# Description:

# ensure PYTHONPATH is set properly
import sys,os,os.path
sys.path.append(os.path.expanduser('/usr/local/lib/python3.4/dist-packages'))

import re

def FLACfileTest():
    # Regex for *.flac
    pattern = r"([a-zA-Z]?\.flac)"
    # Regex for Date
    #pattern = r"([a-zA-Z]+) (\d+)"
    input = str(sys.argv[1])
    #input = "June 24"
#    input = "brian.mp3"  # FALSE condition
    #input = "brian.mp3.flac"  # TRUE condition
    if re.search(pattern, input):
        match = re.search(pattern, input)
        #match = re.search(regex, input)
    # This will print [0, 7), since it matches at the beginning and end of the
    # string
    #print "Match at index %s, %s" % (match.start(), match.end())
    # So this will print "June 24"
    #print "Full match: %s" % (match.group(0))
        print "File ends with FLAC.. runs ok %s" % (match.group(0))
        return 0 # pass
    # So this will print "June"
    #print "Month: %s" % (match.group(1))
    # So this will print "24"
    #print "Day: %s" % (match.group(2))
    else:
    # If re.search() does not match, then None is returned
        print("Error: Must be a FLAC file .. your file was " + sys.argv[1])
        return 1 # fail

# make the file usable as a script as well as an importable module
if __name__ == "__main__":
    import os, fnmatch
    # execute function
    FLACfileTest()

#version info
__version__ = '0.1'

To run

# python testFLAC.py brian.mp3  # fail
# python testFLAC.py brian.mp3.flac  # pass

Go to my Git repository to get the source source testFLAC.py source

Containers – Docker on Ubuntu 14.04

Update – February 11, 2017 – Added Cleaning up docker section.

Pre-reqs:

  • OS: Ubuntu 14.04
  • sudo access to root
  • internet access

Install Docker

Update package manager, install the docker.io package & create a symlink from docker.io to docker, fix bash auto complete to add ‘docker’

sudo apt-get update
sudo apt-get -y install docker.io
sudo ln -sf /usr/bin/docker.io /usr/local/bin/docker
sudo sed -i '$acomplete -F _docker docker' /etc/bash_completion.d/docker

Start Docker on system start-up

update-rc.d docker.io defaults

Download a Docker container

sudo docker pull ubuntu

Run Docker container

Setup a basic ubuntu container with a bash shell, we just run one command.  docker run will run a command in a new container, -i attaches stdin and stdout, -t allocates a tty, -h allocates a hostname, and we’re using the standard ubuntu container.

sudo docker run -i -t ubuntu /bin/bash

That’s it! You’re now using a bash shell inside of a ubuntu docker container.

To disconnect, or detach, from the shell/container without exiting/shutting down use the escape sequence Ctrl-p + Ctrl-q.

To shutdown the container, run exit, but remember that container storage is ephemeral, meaning data isn’t saved. You’ll need to use docker volumes.

To search for other community docker containers

sudo docker search aws
or
sudo docker search google

List containers

sudo docker ps
sudo docker ps -a
sudo docker ps -l
sudo docker ps -a --filter "status=running"

docker ps will list running container, -a list all containers, -l list latest created containers including non-running containers, a –filter “status=running” list all running containers

Attach to a container

sudo docker attach <container-id>
sudo docker attach d8894b58ecb6

Committing changes to your container

sudo docker commit <container-id> <new-container-name>
sudo docker commit d8894b58ecb6 new-docker-v1

To create a new image from changes to a container, it’s a simple as running just one command, docker commit.

View container logs

sudo docker logs <container-id>
sudo docker logs d8894b58ecb6

To view the logs for a container it’s as simple as running just one command, docker logs. If you don’t have any logs listed then no changes have been made to the container.

Restart a running container 

sudo docker stop <container-id>
sudo docker stop d8894b58ecb6
sudo docker stop d8894b58ecb6

To restart a running container, it’s a simple as running just one command, docker restart.

Stop a running container (or multiple containers)

sudo docker stop <container-id>
sudo docker stop d8894b58ecb6
sudo docker stop d8894b58ecb6 bafb4b2fc70f

To stop a running container, it’s a simple as running just one command, docker stop.

List all images/containers on this server

sudo docker images

 

To view docker usage stats – CPU, MEM & Network

sudo docker stats <container-id>
sudo docker stats d8894b58ecb6

Keeping persistent data – docker storage volumes

There’s no way to directly create a “data volume” in Docker, so instead we create a data volume container with a volume attached to it. For any other containers that you then want to connect to this data volume container, use the Docker’s --volumes-from option to grab the volume from this container and apply them to the current container.

sudo docker create -v /data1 --name datacontainer ubuntu

This created a container named datacontainer based off of the ubuntu image and in the directory /data1.

Now, if we run a new Ubuntu container with the --volumes-from flag anything we write to the /data1 directory will get saved

docker run -t -i --volumes-from datacontainer ubuntu /bin/bash
cd /data1
touch thisisapersistentfile.txt
exit

The file ‘thisisapersistentfile.txt’ we created will now be persistent on a container named datacontainer and in the directory /data1.

Cleaning up docker

After starting and stop all of those containers, stuuff get left behind (so you might inspect it later if you desire). But, if you are running out of disk space because of docker, give this script a try.

#!/bin/bash
# hosted at https://gitlab.com/shallawell/clean-up-docker/blob/master/cleanup-docker.sh
#from https://lebkowski.name/docker-volumes/

if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root"
   exit 1
fi

echo " current disk space"
df -h

# remove exited containers:
docker ps --filter status=dead --filter status=exited -aq | xargs -r docker rm -v

# remove unused images:
docker images --no-trunc | grep '<none>' | awk '{ print $3 }' | xargs -r docker rmi

# remove unused volumes:
find '/var/lib/docker/volumes/' -mindepth 1 -maxdepth 1 -type d | grep -vFf <(
  docker ps -aq | xargs docker inspect | jq -r '.[] | .Mounts | .[] | .Name | select(.)'
) | xargs -r rm -fr

echo "new disk space"
df -h

Docker help

sudo docker –help

https://docs.docker.com