Nginx + youtrack + systemd

Out task is to deploy YouTrack on CentOS 7.X behind Nginx.

What we need to be done for that task:

1. Download and install rpm with latest java from java.com
2. Download YouTrack jar file
3. Add new user for youtrack process and home dir for files
4. Create youtrack.service file for systemd for youtrack service management
5. Configure host in nginx 

1 and 2 need no additional comments. Only one thing, I placed youtrack jar file in /usr/local/sbin and created symlink to current version:

~ la -l /usr/local/sbin/
total 115508
-rwxr-xr-x 1 root root 118280108 Dec 9 13:42 youtrack-6.5.16953.jar
lrwxrwxrwx 1 root root 22 Dec 17 12:41 youtrack.jar -> youtrack-6.5.16953.jar

From management and security point of view it's better to create new user for new service:

useradd -m -d /opt/youtrack youtrack

Now we need .service file for youtrack in systemd:

~  cat /etc/systemd/system/youtrack.service
; /etc/systemd/system/youtrack.service
[Unit]
Description=JetBrains Youtrack
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=youtrack.target

[Service]
User=youtrack
Group=youtrack
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /var/run/youtrack
ExecStartPre=/usr/bin/chown -R youtrack:youtrack /var/run/youtrack/
PIDFile=/var/run/youtrack/main.pid
ExecStart=/usr/bin/java -Xmx1g -Djava.security.egd=/dev/zrandom -Djava.awt.headless=true -Duser.home=/opt/youtrack -Djetbrains.youtrack.disableBrowser=true -jar /usr/local/sbin/youtrack.jar 127.0.0.1:4080
ExecStop=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Two things are important here. Since /var/run is in tempfs in CentOS 7.X you need to create pid file dir before service starting up:  

PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /var/run/youtrack
ExecStartPre=/usr/bin/chown -R youtrack:youtrack /var/run/youtrack/

And second one is youtrack's start command in:

ExecStart=/usr/bin/java -Xmx1g -Djava.security.egd=/dev/zrandom -Djava.awt.headless=true -Duser.home=/opt/youtrack -Djetbrains.youtrack.disableBrowser=true -jar /usr/local/sbin/youtrack.jar 127.0.0.1:4080

Last part is nginx config:

server {
    server_name youtrack.domain.com;
    listen 80;
    charset utf8;

    rewrite ^ https://$host$request_uri? permanent;
}

server {
    server_name youtrack.domain.com;
    listen 443 ssl;
    charset utf8;

    root /www/youtrack.domain.com/htdocs; 

    ssl on;
    ssl_certificate             /etc/nginx/ssl/youtrack.domain.com.crt;
    ssl_certificate_key         /etc/nginx/ssl/youtrack.domain.com.key;
    ssl_session_timeout         5m;
    ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers                 ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass http://localhost:4080/;
        client_max_body_size 64M;
    }

    access_log /var/log/nginx/youtrack.domain.comu/access.log main;
    error_log /var/log/nginx/youtrack.domain.com/error.log warn;
}

Swap usage by pid

for i in `find /proc -maxdepth 1 -type d|xargs -i basename {}`;do echo -n "PID: $i SWAP: ";cat /proc/$i/smaps|fgrep Swap|awk '{s=s+$2}END{print s}';done|sort -k4n

Bitwise operators in Python

>>> x = 10
>>> y = 6
>>> print "x={x} ({x:08b}), y={y} ({y:08b})".format(x=x, y=y)
x=10 (00001010), y=6 (00000110)

>>> print "x XOR y = {r:d} ({r:08b})".format(r=x ^ y)
x XOR y = 12 (00001100)

>>> print "x OR y = {r:d} ({r:08b})".format(r=x | y)
x OR y = 14 (00001110)

>>> print "x AND y = {r:d} ({r:08b})".format(r=x & y)
x AND y = 2 (00000010)

Python dict.fromkeys() magic

I just got some magic from python and what to share with the world.
Imagine we have some python code:

keys = ("T1", "T2", "T3")
td = dict.fromkeys(keys, [])
print td

{'T2': [], 'T3': [], 'T1': []}

Ok. Now lets try to populate new dictionary with some data:

for i in range(0,5):
 for v in td.itervalues():
  v.append(1)
print td

I think you are expecting to get something like:
{'T2': [1, 1, 1, 1, 1], 'T3': [1, 1, 1, 1, 1], 'T1': [1, 1, 1, 1, 1]}

But Nope! You will see this one:
{'T2': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'T3': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'T1': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

The magic comes from fromkeys() function. It seems the function links all dictionary keys to one list object. Lets check it:

for k in td.itervalues():
 print id(k)

140567008377312
140567008377312
140567008377312

Yes, it's true! So be carefully with fromkeys() or use something like:

td = dict((k, []) for k in keys)

P.S.
Thanks to Alexey Gusev for this case.
And yes, when we have understood the problem I easily found this discussion on stackoverflow.

Simple moving average in RRD

It's quite easy to add simple moving average (SMA) to RRD with rrd PREV function.
PREV allows you to get data from previous time stamp. The magic is that you can run that function on sequences of variables to dig deeper in the past.

Here is SMA from 10 variables, current value and 9 from the past (php template for PNP).

$def[1] .= "DEF:idle=$rrdfile:$DS[3]:AVERAGE " ;

$def[1] .= "CDEF:idle1=PREV(idle), " ;
$def[1] .= "CDEF:idle2=PREV(idle1), " ;
$def[1] .= "CDEF:idle3=PREV(idle2), " ;
$def[1] .= "CDEF:idle4=PREV(idle3), " ;
$def[1] .= "CDEF:idle5=PREV(idle4), " ;
$def[1] .= "CDEF:idle6=PREV(idle5), " ;
$def[1] .= "CDEF:idle7=PREV(idle6), " ;
$def[1] .= "CDEF:idle8=PREV(idle7), " ;
$def[1] .= "CDEF:idle9=PREV(idle8), " ;
$def[1] .= "CDEF:idle_sma=idle,idle1,idle2,idle3,idle4,idle5,idle6,idle7,idle8,idle9,+,+,+,+,+,+,+,+,+,10,/ " ;

How to move lv volume from host to host

To move one LVM volume from host A to host B you should create on host B a volume with same size as moving volume from host A.

After creation you can copy data from one volume to another with dd, ssh commands and gzip for speeding it up:

dd bs=1M if=LVM_HOST_A | gzip --fast -c | ssh HOST_B "gzip -d -c | dd bs=1M of=LVM_HOST_B"

To speed it up little more you can turn off compression in ssh (we already have compressed data with gzip) and choose fast cipher by adding -c arcfour -o Compression=no to ssh:

dd bs=1M if=LVM_HOST_A | gzip --fast -c | ssh -c arcfour -o Compression=no HOST_B "gzip -d -c | dd bs=1M of=LVM_HOST_B"

You can install pv utility to see file transfer progress, rate and etc. Like this:

dd bs=1M if=LVM_HOST_A | pv -brtp | gzip --fast -c | ssh -c arcfour -o Compression=no HOST_B "gzip -d -c | dd bs=1M of=LVM_HOST_B"

And look at lvmsync - it allows you to sync only changes made since LVM snapshot was made.

PostgreSQL PL speed test

I tested four procedure languages for PostgreSQL 9.2: plpgsql, plperl, plpython2u, plperl and plv8 with Fibonacci numbers function. And V8 was the best!