Small python trick: Better way to init dictionary

From my php and perl background I remember that named arrays or hashes should be initialize correctly, like this:

if (haskey(x, somehash)) {
  somehash[x] += y;
} else {
  somehash[x] = y;
}

And it looks like a tumour inside a python code. But .get() method can cure it:

somedict[x] = somedict.get(x, 0) + y

Looks much better! 

XSendfile usage in Nginx

Suppose you have a corporate site or an internal portal or something else with access from web, but it's not a public site. And to gain access to any page or file your customer at least must be authenticated. It's not a problem for any web app, but it's a problem for static files (pdf, docx, xlsx and so on). With direct link anyone can download some sensitive information. And there are a lot of ways to share such links with the world unintentionally (mail, IM, web spiders, etc).

Old school way to protect a site from downloading direct links is to use http basic auth. It can solve a problem but It also creates some new problems. You need to manage one more database with users logins and passwords for basic auth and sync it with your app. Also http basic auth in web server is week against brute force attack. You can't use many tricks against brute force which are available for you in app. And don't forget about all your users pain because of double authentication process.

There is more elegant way to solve a problem with direct links. Nginx has XSendfile feature. You can use it for static files protection in such way:

1. Nginx works as a frontend for your App.
2. All links to static files send to App.
3. If App thinks that user is authenticated it sends response with X-Accel-Redirect header with real path to static file, otherwise it sends response with some http 4xx error.
4. In nginx vhost location for static files are marked with internal directive which protect files in it from direct download without X-Accel-Redirect header from upstream server.

Short example for Django App:

url(r'^app/static/(?P.*)$', StaticFile.as_view(), name="static_file")
 
class StaticFile(View):
    def get(self, request, path=None):
        if not request.user.is_authenticated():
            return HttpResponseForbidden("Access denied")
        if not path or not os.path.isfile(os.path.join(settings.STATIC_URL, path)):
            return HttpResponseNotFound("Not found")
        mimetypes.init()
        try:
            if "." in path:
                mimetype = mimetypes.types_map[".%s" % path.split(".")[-1]]
            else:
                raise KeyError
        except KeyError:
            mimetype = "text/plain"
        response = HttpResponse()
        response['X-Accel-Redirect'] = "/static/%s" % path
        response['Content-Type'] = mimetype
        return response

And location in nginx:

location /static/ {
    internal;
    root /www/app/;
}

Nginx + youtrack + systemd

Out task is to deploy YouTrack on CentOS 7.X behind Nginx.

What we need to be done for that task:

1. Download and install rpm with latest java from java.com
2. Download YouTrack jar file
3. Add new user for youtrack process and home dir for files
4. Create youtrack.service file for systemd for youtrack service management
5. Configure host in nginx 

1 and 2 need no additional comments. Only one thing, I placed youtrack jar file in /usr/local/sbin and created symlink to current version:

~ la -l /usr/local/sbin/
total 115508
-rwxr-xr-x 1 root root 118280108 Dec 9 13:42 youtrack-6.5.16953.jar
lrwxrwxrwx 1 root root 22 Dec 17 12:41 youtrack.jar -> youtrack-6.5.16953.jar

From management and security point of view it's better to create new user for new service:

useradd -m -d /opt/youtrack youtrack

Now we need .service file for youtrack in systemd:

~  cat /etc/systemd/system/youtrack.service
; /etc/systemd/system/youtrack.service
[Unit]
Description=JetBrains Youtrack
After=network.target
After=syslog.target

[Install]
WantedBy=multi-user.target
Alias=youtrack.target

[Service]
User=youtrack
Group=youtrack
PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /var/run/youtrack
ExecStartPre=/usr/bin/chown -R youtrack:youtrack /var/run/youtrack/
PIDFile=/var/run/youtrack/main.pid
ExecStart=/usr/bin/java -Xmx1g -Djava.security.egd=/dev/zrandom -Djava.awt.headless=true -Duser.home=/opt/youtrack -Djetbrains.youtrack.disableBrowser=true -jar /usr/local/sbin/youtrack.jar 127.0.0.1:4080
ExecStop=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Two things are important here. Since /var/run is in tempfs in CentOS 7.X you need to create pid file dir before service starting up:  

PermissionsStartOnly=true
ExecStartPre=/usr/bin/mkdir -p /var/run/youtrack
ExecStartPre=/usr/bin/chown -R youtrack:youtrack /var/run/youtrack/

And second one is youtrack's start command in:

ExecStart=/usr/bin/java -Xmx1g -Djava.security.egd=/dev/zrandom -Djava.awt.headless=true -Duser.home=/opt/youtrack -Djetbrains.youtrack.disableBrowser=true -jar /usr/local/sbin/youtrack.jar 127.0.0.1:4080

Last part is nginx config:

server {
    server_name youtrack.domain.com;
    listen 80;
    charset utf8;

    rewrite ^ https://$host$request_uri? permanent;
}

server {
    server_name youtrack.domain.com;
    listen 443 ssl;
    charset utf8;

    root /www/youtrack.domain.com/htdocs; 

    ssl on;
    ssl_certificate             /etc/nginx/ssl/youtrack.domain.com.crt;
    ssl_certificate_key         /etc/nginx/ssl/youtrack.domain.com.key;
    ssl_session_timeout         5m;
    ssl_protocols               TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers                 ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv3:+EXP;
    ssl_prefer_server_ciphers   on;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_pass http://localhost:4080/;
        client_max_body_size 64M;
    }

    access_log /var/log/nginx/youtrack.domain.comu/access.log main;
    error_log /var/log/nginx/youtrack.domain.com/error.log warn;
}

Swap usage by pid

for i in `find /proc -maxdepth 1 -type d|xargs -i basename {}`;do echo -n "PID: $i SWAP: ";cat /proc/$i/smaps|fgrep Swap|awk '{s=s+$2}END{print s}';done|sort -k4n

Bitwise operators in Python

>>> x = 10
>>> y = 6
>>> print "x={x} ({x:08b}), y={y} ({y:08b})".format(x=x, y=y)
x=10 (00001010), y=6 (00000110)

>>> print "x XOR y = {r:d} ({r:08b})".format(r=x ^ y)
x XOR y = 12 (00001100)

>>> print "x OR y = {r:d} ({r:08b})".format(r=x | y)
x OR y = 14 (00001110)

>>> print "x AND y = {r:d} ({r:08b})".format(r=x & y)
x AND y = 2 (00000010)

Python dict.fromkeys() magic

I just got some magic from python and what to share with the world.
Imagine we have some python code:

keys = ("T1", "T2", "T3")
td = dict.fromkeys(keys, [])
print td

{'T2': [], 'T3': [], 'T1': []}

Ok. Now lets try to populate new dictionary with some data:

for i in range(0,5):
 for v in td.itervalues():
  v.append(1)
print td

I think you are expecting to get something like:
{'T2': [1, 1, 1, 1, 1], 'T3': [1, 1, 1, 1, 1], 'T1': [1, 1, 1, 1, 1]}

But Nope! You will see this one:
{'T2': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'T3': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'T1': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

The magic comes from fromkeys() function. It seems the function links all dictionary keys to one list object. Lets check it:

for k in td.itervalues():
 print id(k)

140567008377312
140567008377312
140567008377312

Yes, it's true! So be carefully with fromkeys() or use something like:

td = dict((k, []) for k in keys)

P.S.
Thanks to Alexey Gusev for this case.
And yes, when we have understood the problem I easily found this discussion on stackoverflow.

Simple moving average in RRD

It's quite easy to add simple moving average (SMA) to RRD with rrd PREV function.
PREV allows you to get data from previous time stamp. The magic is that you can run that function on sequences of variables to dig deeper in the past.

Here is SMA from 10 variables, current value and 9 from the past (php template for PNP).

$def[1] .= "DEF:idle=$rrdfile:$DS[3]:AVERAGE " ;

$def[1] .= "CDEF:idle1=PREV(idle), " ;
$def[1] .= "CDEF:idle2=PREV(idle1), " ;
$def[1] .= "CDEF:idle3=PREV(idle2), " ;
$def[1] .= "CDEF:idle4=PREV(idle3), " ;
$def[1] .= "CDEF:idle5=PREV(idle4), " ;
$def[1] .= "CDEF:idle6=PREV(idle5), " ;
$def[1] .= "CDEF:idle7=PREV(idle6), " ;
$def[1] .= "CDEF:idle8=PREV(idle7), " ;
$def[1] .= "CDEF:idle9=PREV(idle8), " ;
$def[1] .= "CDEF:idle_sma=idle,idle1,idle2,idle3,idle4,idle5,idle6,idle7,idle8,idle9,+,+,+,+,+,+,+,+,+,10,/ " ;