zurück zum Anfang jw bargsten
Anfang
Projekte
Fetzen
Über mich
Kontakt

Fetzen aus Code

» Python

» Java & Vavr

Verschiedenes

» Colourful Bash Logging

Poor man’s logging implementation for bash scripts:

YELLOW=$'\e[33m'
BLUE=$'\e[34m'
RED=$'\e[31m'
NC=$'\e[0m'

isodate() {
  date "+%Y-%m-%d %H:%M:%S"
}

log_error() {
  echo "$RED$(isodate) [ERROR] $*$NC"
}

log_warn() {
  echo "$YELLOW$(isodate) [WARN] $*$NC"
}

log_info() {
  echo "$BLUE$(isodate) [INFO] $*$NC"
}

log_debug() {
  echo "$(isodate) [DEBUG] $*"
}

log_error "foo bar"
log_warn "foo bar"
log_info "foo bar"
log_debug "foo bar"

Bash logging output

» Bash Helper Functions

Frequently used in CI/CD pipelines:

to_lower() {
  tr '[:upper:]' '[:lower:]'
}

to_upper() {
  tr '[:lower:]' '[:upper:]'
}

to_alnum() {
  tr -dc '[:alnum:]'
}

snake_uc_to_camel() {
  to_lower | sed -r 's/_([a-z0-9])/\U\1/g'
}

camel_to_snake_uc() {
  sed 's/\([a-z0-9]\)\([A-Z]\)/\1_\2/g' | to_upper
}

confirm() {
  # https://stackoverflow.com/questions/1885525/how-do-i-prompt-a-user-for-confirmation-in-bash-script
  read -p "${1:-Are you sure?} (y/n) " -n 1 -r
  echo    # (optional) move to a new line
  if [[ $REPLY =~ ^[Yy]$ ]]; then
    return 0
  fi
  return 1
}

confirm_or_exit() {
  if ! confirm "$@"; then
    exit 0
  fi
}

check_env_var_presence() {
  : "${VARIABLE1?'not set'}"
  : "${VARIABLE2?'not set'}"

  if [[ ! "$ENV" =~ ^(dev|acc|prd)$ ]]; then
    echo "ENV needs to be one of dev, acc or prd, but not $ENV"
    exit 1
  fi
}

GIT_COMMIT_SHA="e4637392ae6a3fad3ec16d1e5a9b40657990d76b"
GIT_COMMIT_SHA_SHORT=${GIT_COMMIT_SHA:0:7}

» Generate passwords that need to fit character class and length requirements

In case you need to generate passwords for Azure:

#!/usr/bin/env bash

pw_length=${1:-18}
p=''
while [[ !("$p" =~ [A-Z] && "$p" =~ [a-z] && $p =~ [0-9] && "$p" =~ [[:punct:]]) ]]; do
  p=$(LC_ALL=C < /dev/urandom tr -dc "[:graph:]" | tr -d "'" | head -c${pw_length})
done

echo -n $p

» Export environment variables from dotenv (.env) files

To export everything in your .env file, you can run

export $(<.env grep -v "^#" | xargs)

Inside a bash script you could use:

load_dotenv() {
  set -o allexport
  # shellcheck disable=SC1091
  [[ -f .env ]] && source .env
  # default variable that should be exported if not set earlier
  : "${ENV:=dev}"
  set +o allexport
}

see also discussion on github

If you have a more complex setup, e.g. JSON blobs stored in environment variables

APP_CONFIG={"SOME":"important","APP":"configuration"}

you might have to fall back to a more sophisticated solution. Here is a small python script (fmt-env) I use to format the .env file:

#!/usr/bin/env python3
import sys
import shlex
import re


def quote(line):
    k, v = line.strip().split("=", 1)
    return "{}={}".format(k, shlex.quote(v))


stmts = [quote(x) for x in sys.stdin if not re.match(r"\s*(#|$)", x)]
print("\n".join(stmts))

Call the script via

export $(<.env fmt-env | xargs)

(Do not forget to chmod +x fmt-env before you use it)

In case you have to get your .env setup into JSON format (for e.g. cypress), you can use

node \
  -e "require('dotenv').config(); process.stdout.write(JSON.stringify(process.env, null, 2))" \
  >cypress.env.json

(this assumes that you have the dotenv-package installed.)

» How to run Lagom in development mode and in the background

To start a Lagom or Play project in development mode for testing, you start everything with sbt runAll. You can stop everything by hitting Enter (or any other key).

Now you might think: cool, Lagom is starting everything for me, so why not use the same approach in my CI pipeline to run integration tests in a clean environment. Well, good luck!

One way or another, no matter what, the server is stopped immediately after startup. This happens

  • in the container
  • as background process
  • with and without “</dev/null sbt runAll &”
  • …

If you trace down the issue in the Lagom source, you’ll end up in ConsoleHelper.scala, which uses System.in.read() to wait for “Enter”:

def blockUntilExit() = {
  // blocks until user presses enter
  System.in.read()
}

This piece of code also triggers on other input, even if you use /dev/null as standard input. In case of /dev/null, read() returns an end of file and will not block.

Luckily there is a solution for it. To figure out what to do, you have to return to the basics. The question is: “how to deal with System.in.read()” and not “how to run Lagom in the background”. For the first question StackOverflow knows the answer.

Put into action, you have to create a bash script, e.g. run_in_bg.sh with the following contents:

#!/usr/bin/env bash

(while true; do sleep 10000; done) | sbt runAll >lagom.log &

This will allow you to run Lagom (or Play) in the background.

» Pass tool fuzzy finder

Some of you might use pass - the standard unix password manager to store passwords. I use it exclusively, but at times it can be cumbersome to look up a password that you don’t remember exactly. As mentioned earlier, in such fuzzy situations, fzf can be of great help. Here is a snippet to fuzzy search the passwords of pass. Add this to your .bashrc or equivalent:

fpass() {
  local pwdir=$(cd ${PASSWORD_STORE_DIR:=~/.password-store/} && pwd)
	local paz=$(find $pwdir -name "*.gpg"  | \
		perl -pE "s#^(?:.*)\\Q$pwdir\\E/(.*)\.gpg#\\1#"  | \
		fzf +m);
  [[ ! -z "$paz" ]] && echo $paz && EDITOR=vim pass $* $paz
}

Now you can run fpass and start the fuzzy search of fzf.

» Fuzzy switching of git branches on cli, newest first.

Even though there are plenty of UIs out there, I cannot get my self detached from the git commandline interface (cli). I cannot escape! It keeps a tight grip on me. Probably the best way out is the way forward, so why not make the cli more like the UI? Luckily, there are plenty of possibilities. One of them is to use fzf to select branches. On thing is certain: fzf for changed my life on the cli. I can combine it with z.lua to jump to directories, I can use it in vim to select files, in the shell to wade through my history and to switch git branches. On the latter I want to focus here. This use case arose from working on a big project with many branches. Often I want to go back and forth between git branches, but I forgot the exact name, I only remember that I worked on it recently and probably a word contained in the branch name. (There are examples of using fzf with git branches, but I just could not find the right snippet that would satisfy my needs.)

asciicast

To get it running, you need to

  1. install fzf
  2. create git-fuzzy-branch in a directory that is part of your $PATH environment variable 
    #!/usr/bin/env bash

branches=$(git --no-pager branch --sort=-committerdate  -vv) &&
branch=$(echo "$branches" | fzf +m) && echo "$branch" &&
git checkout $(echo "$branch" | perl -pE 's/^\W?\s+(\S+)\s+.*$/\1/')
  3. make git-fuzzy-branch executable with chmod +x git-fuzzy-branch
  4. add an alias to your [alias] section in ~/.gitconfig 
    [alias]
  # use fzf to select branch
  fb = "!git-fuzzy-branch"

That’s it. Have fun!

» Skip Jest tests (js)

You can skip tests in Jest easily by prefixing them with a x.

  • For single tests, use xtest (or test.skip) instead of test.
  • For describe use xdescribe (or describe.skip) instead of describe.

(source)