WHAT WE THINK?
Loading...
6 Tips That Every MySQL User Should Know
Over the last 3 years, I have been working with MySQL almost every day. Even though non-relational databases like MongoDB are gaining more and more popularity every year, traditional SQL solutions are still widely used for many purposes. In this post, I will share some of my tricks I have been using to make my life easier. Note: most of those tips apply only to development machines, for production you should take more care.
1. Run MySQL in Docker
Seriously, in 2018 there is absolutely no need to run MySQL server natively by installing it, setting users and passwords, performing upgrades etc. You are wasting your client's time if you are still doing it. Just use the sample Docker Compose file as a working starting point: [code] version: '3' services: mysql: image: mysql:5.7 environment: - MYSQL_ROOT_PASSWORD=root - MYSQL_DATABASE=myproject ports: - 127.0.0.1:3306:3306 volumes: - ./varlibmysql:/var/lib/mysql [/code] After docker-compose up, you will have a working server with localhost-only port bound on standard port 3306, one user root/root and a pre-created "myproject" database. Throw in restart: always into Compose file if you want to keep the server running across reboots. That is enough for 95% of software projects, this solution is completely disposable and easy to recreate. Note: I still have MySQL client installed natively on my development machine. Technically speaking, there is a way of avoiding this too and running the client itself from Docker image, but that is a matter of preference.2. Store MySQL data in RAM
In Docker world, the best practice for handling data storage is to store it outside of the container filesystem - in case of a development machine, in some mounted directory on the host. A cool trick, at least on Linux, is to create a RAM-based filesystem called tmpfs and use it as data storage for MySQL. This will, of course, result in data loss after machine restart, but who cares about development data? If it is important, you should have a recent and verified backup anyway, right? In my case, I am mounting a folder /tmp/varlibmysql into container /var/lib/mysql, since I am using ramdisk for the whole temporary directory to limit SSD wearing. So the relevant part of Compose file is: [code] ... volumes: - /tmp/myproject/varlibmysql:/var/lib/mysql ... [/code] There is a noticeable performance gain with this configuration: I measured the time it takes to run a few hundred Liquibase migrations on application startup and the time it takes to import ~1GB database dump.- migrations: SSD 0:44, ramdisk 0:07 - huge speedup
- import: SSD 5:23, ramdisk 4:14 - small but noticeable speedup
3. Manage database dumps like a boss
I personally dislike edge case bugs, which often appear in web applications. One of the ways for a tester to describe such rarely occurring bug is to provide a list of bazillion intricate steps, that have to be carefully performed in order for the bug to appear. It is much easier to create a bug report with a database dump attached, which contains the whole state of the application. As a result, explaining and more importantly reproducing a bug is much easier and faster. However, if every time this happens there is a need to go to StackOverflow to recall mysqldump syntax, no one will want to do this. So let's fix the issue once and for all: [code] $ cat export.sh #! /bin/bash set -e DATABASE=myproject if [ "$#" -ne 1 ]; then echo "Usage: export.sh <filename.sql.bz2>" exit 1 fi echo "Exporting to $1..." mysqldump --protocol tcp -h localhost -u root -proot ${DATABASE} \ | pv \ | bzip2 > "$1" echo "Export finished." [/code] [code] $ cat import.sh #! /bin/bash set -e DATABASE=myproject if [ "$#" -ne 1 ]; then echo "Usage: import.sh <filename.sql.bz2>" exit 1 fi echo "Importing from $1..." bzcat "$1" \ | pv \ | mysql --protocol tcp -h localhost -u root -proot ${DATABASE} echo "Importing finished." [/code] [code] $ cat drop.sh #! /bin/bash set -e DATABASE=myproject echo "Dropping and recreating ${DATABASE} ..." mysql --protocol tcp -h localhost -u root -proot ${DATABASE} \ -e "drop database ${DATABASE}; create database ${DATABASE};" echo "Done." [/code] These are 3 scripts I use every day for SQL export, import and one extra for recreating a database for testing purposes. Those scripts use bzip2 compression for minimum file size and pv tool for visualising data flow.4. Log executed queries
Recently I have been fixing some performance problems in one of our projects. Our business contact reported that "this specific webpage is slow when there are lots of users present". I started looking around in Chrome Developer Tools and it became clear that the issue is on the backend side, as usual... I could not see any obvious bottlenecks in Java code, so I went a layer down into the database and yep, there was a performance problem there - some innocent SQL query was executed thousands of times for no reason. In order to debug such cases, query logging is a must, otherwise we are shooting in the dark. You can enable basic query logging using those commands in MySQL console: [code] MySQL [myproject]> SET global general_log = 1; Query OK, 0 rows affected (0.00 sec) MySQL [myproject]> SET global log_output = 'table'; Query OK, 0 rows affected (0.00 sec) [/code] From now on, all queries will be logged in special table mysql.general_log. Fun fact - this table is actually a real physical table, it can be searched, exported etc. - good for documenting bugfixes. Let's create some sample database structure: [code] MySQL [myproject]> create table random_numbers(number float); Query OK, 0 rows affected (0.00 sec) MySQL [myproject]> insert into random_numbers values (rand()), (rand()), (rand()), (rand()); Query OK, 4 rows affected (0.00 sec) Records: 4 Duplicates: 0 Warnings: 0 [/code] And now run a few queries and see if they are captured in the log: [code] MySQL [myproject]> select * from random_numbers where number < 0.1; Empty set (0.00 sec) MySQL [myproject]> select * from random_numbers where number < 0.5; +----------+ | number | +----------+ | 0.254259 | +----------+ 1 row in set (0.00 sec) MySQL [myproject]> select * from random_numbers where number < 0.9; +----------+ | number | +----------+ | 0.777688 | | 0.254259 | +----------+ 2 rows in set (0.00 sec) MySQL [myproject]> select event_time, argument from mysql.general_log; +----------------------------+-------------------------------------------------+ | event_time | argument | +----------------------------+-------------------------------------------------+ | 2018-11-26 12:42:19.784295 | select * from random_numbers where number < 0.1 | | 2018-11-26 12:42:22.400308 | select * from random_numbers where number < 0.5 | | 2018-11-26 12:42:24.184330 | select * from random_numbers where number < 0.9 | | 2018-11-26 12:42:28.768540 | select * from mysql.general_log | +----------------------------+-------------------------------------------------+ 4 rows in set (0.00 sec) [/code] Perfect! Keep in mind that "all queries" means all of all, so if you are using graphical database tools for viewing query logs, those "query querying logs" will be also there.5. Use remote servers
MySQL protocol runs over TCP/IP (note to nitpickers: yes, it can also work through UNIX sockets, but the principle is the same). It is perfectly fine to use some remote MySQL server/service for local development instead of a local server. This is useful for working with big databases that would not fit onto tiny laptop SSD, or if we need more performance. The only concern is network latency, which can sometimes diminish any performance gains - but I think it is still a useful trick. Let's try Amazon RDS. It is pretty expensive for long-term usage but affordable for one-off tasks. Graphical setup wizard in AWS web console is pretty easy to use so I will not cover it here in full, however, keep attention on:- DB instance class - pick the cheapest one for start, you can always upgrade it later if needed
- Allocated storage - the minimum (20 GiB) is actually a huge amount of space for a typical project, but you can increase it now if you need
- Username and password - pick something secure, because our instance will be publicly visible from the Internet
- Security group - pick/create a security group with full access from the Internet
We can look up its details:
And finally, find the server hostname to connect to...
...like usual:
[code]
$ mysql -uroot -pmysqlletmein --protocol tcp \
-h mydbinstance.cizrsk4vjj5m.eu-central-1.rds.amazonaws.com
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 15
Server version: 5.6.41-log Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> select rand();
+--------------------+
| rand() |
+--------------------+
| 0.8418013517202823 |
+--------------------+
1 row in set (0.06 sec)
[/code]
Now we can configure our backend to use it and it should work exactly like a local server. Amazon RDS gives us some useful development features like real-time monitoring, automatic backups, spinning new database instances from snapshots and much more. Go try it!
6. Learn some SQL!
Finally, even if you love ORMs and you can't live without Spring Data and sexy dynamic finders (which can sometimes be long enough to wrap on 4k screen), it is very beneficial to learn at least some SQL to understand how everything works underneath, how ORMs are mapping one-to-many and many-to-many relationships with the use of extra join tables, how transactions work (very important in high load systems) and so on. Also, some kinds of performance problems ("N+1 select" being the most common) are completely undiscoverable without knowing how underlying SQL is generated and subsequently executed. And that is all. Thanks for reaching this point and I hope you've learnt something.15 April 2019
Let’s shake some trees – how to enhance the performance of your application
Nowadays JavaScript applications are getting bigger and bigger. One of the most crucial things while developing is to optimise the page load time by reducing the size of the JavaScript bundle file.
JavaScript is an expensive resource when processing and should be compressed when it is about to be sent over the network.
One of the most popular techniques to improve the performance of our applications is code splitting. It is based on splitting the application into chunks and serving only those parts of JavaScript code that are needed at the specified time. However, this article is going to be about another good practice called tree shaking.
Tree shaking is used within the ES2015 import and export syntax and supports the dead-code elimination. Since Webpack 4 released it is possible to provide the hint for a compiler by the “sideEffects” property to point the modules that can be safely pruned from the application tree if unused. Function may be supposed to have side effects if it modifies something outside its own scope.
Real life example
In order to introduce tree shaking concept more precisely, let’s start with creating a new project including the application entry point (index.js) and the output bundle file (main.js).
In the next step, a new JavaScript file (utils.js) is added to the src directory...
[code]
export function foo() {
console.log('First testing function')
}
export function bar() {
console.log('Second testing function')
}
[/code]
...and imported in the index.js.
[code]
import { foo } from './utils.js'
foo()
[/code]
Webpack 4 introduces the production and development mode. In order to get a not minified output bundle, we are supposed to run a build process in the development mode what can be defined in the package.json file.
[code]
"scripts": {
"dev": "webpack --mode development",
"build": "webpack --mode production"
}
[/code]
Now, just get the terminal and run: npm run build script.
Despite, only the foo function has been required in the entry point, our output bundle still consists of both foo and bar methods. Bar function is known as a “dead code” since it is unused export that should be dropped.
[code]
console.log('First testing function');\n\nfunction bar() {\n console.log('Second testing function')
[/code]
To fix this problem we are about to set the “sideEffects” property in package.json file.
[code]
{
"name": "tree-shaking",
"version": "1.0.0",
"sideEffects": "false",
}
[/code]
That property just tells the compiler that there are not any side effects files and every unused code can be pruned. It accepts also absolute and relative paths to the files that should not be dropped due having some side effects.
[code]
{
"name": "tree-shaking",
"version": "1.0.0",
"sideEffects": "./src/file-wth-side-effects.js",
}
[/code]
Minification
After we pruned unused ES6 imports and exports, we still need to remove “dead code” from the application bundle. The only thing we have to do is to set the mode configuration to production and execute npm run build.
[code]
([function(e,t,n){"use strict";n.r(t),console.log("First testing function")}]);
[/code]
As we can see, the second testing function is no more included in the bundle minified file.
Use three shaking with the ES6
It is crucial to keep in mind that tree shaking pattern can be used only within the ES6 import and export modules. We cannot “shake the tree” while using the CommonJS without the help of special plugins. To solve this issue, setting the babel-preset-env to leave the ES6 modules on their own should be performed.
[code]
{
"presets": [
["env", {
"modules": false
}]
]
}
[/code]
Exception
Removing unused modules does not work while dealing with lodash. Lodash is one of the most popular utility library. If you import a module in the way it is depicted below, it will still pull all the lodash library.
[code]
import { join } from 'lodash'
[/code]
To go around that, we need to install lodash-es package and require the modules in the following way:
[code]
import join from 'lodash-es/join'
[/code]
Conclusion
Let’s prove the statement from the beginning of this article! Let’s take a closer look at the sizes of the bundle file (main.js) before and after the tree shaking and minification process.
As we can see we minimized the output size significantly. When you start using tree shaking in your projects it may seem not taking a big advantage of application performance at all. You will notice how it really boosts up your work while having a more complex application tree.
14 January 2019
0
Kubernetes for poor – how to run your cluster and not go bankrupt
Since the last few years, Kubernetes has proved that it is the best container orchestration software on the market. Today, all 3 biggest cloud providers (Amazon, Google and Azure) are offering a form of managed Kubernetes cluster in form of EKS, GKE and AKS respectively. All of those offerings are production-ready, they are fully integrated with other cloud services and they include commercial support. There is one problem, though, and it is the price. Typically, cloud offerings like this are targeted for big organizations with a lot of money. For example, Amazon Elastic Kubernetes Service costs 144$/mo for just running cluster management ("control plane"), and all compute nodes and storage are billed separately. Google Kubernetes Engine does not charge anything for the control plane, but instances to run nodes at aren't cheap either - a reasonable 4 GiB machine with only a single core costs 24$/mo. The question is, can we do it cheaper? Since Kubernetes itself is 100% open source, we can install it on our own server of choice. How much we'll be able to save? Big traditional VPS providers (like OVH or Linode for instance) give out decent machines at prices ranging from 5$/mo. Could this work? Well, let's try it out!
Setting up the server
Hardware
For running our single-master single-node host we don't need anything too fancy. The official requirements for a cluster bootstrapped by kubeadm tool, which we are going to use, are as follows:- 2 GiB of RAM
- 2 CPU cores
- reasonable disk size for basic host software, Kubernetes native binaries and it's Docker images
Installing Docker and kubeadm
Here we will be basically following the official guide for kubeadm, which is a tool for bootstrapping a cluster on any supported Linux machine. First thing is Docker - we'll install it through the default Ubuntu repo, but generally, we should pay close attention to Docker version - only specific ones are supported. I've run into many problems creating a cluster with the current newest version (18.06-*), so I think this is worth mentioning. [code] $ apt-get update $ apt-get install -y docker.io [/code] And let's run a sample image to check if the installation was successful: [code] $ docker run --rm -ti hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 9db2ca6ccae0: Pull complete Digest: sha256:4b8ff392a12ed9ea17784bd3c9a8b1fa32... Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. (...) $ [/code] Good. The next step is kubeadm itself - note that the last command (apt-mark hold) will prevent APT from automatically upgrading those packages if a new version appears. This is critical because cluster upgrades are not possible to be done automatically in a self-managed environment. [code] $ apt-get update && apt-get install -y apt-transport-https curl $ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg \ | apt-key add - $ cat <<EOF >/etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF $ apt-get update $ apt-get install -y kubelet kubeadm kubectl $ apt-mark hold kubelet kubeadm kubectl [/code]Setting up a master node
The next step is to actually deploy Kubernetes into our server. Those 2 basic commands below initialize the master node and select Flannel as an inter-container network provider (an out-of-the-box zero-config plugin). People with experience with Docker Swarm will immediately notice the similarity in init command below - here we specify two IP addresses:- apiserver advertise address is the address at which, well, apiserver will be listening - for single-node clusters we could put 127.0.0.1 there, but specifying external IP of the server here will allow us to add more nodes to the cluster later, if necessary,
- pod network CIDR is a range of IP addresses for pods, this is dictated by the network plugin that we are going to use - for Flannel it must be that way, period.
Testing our setup
Testing - pod scheduler
Kubernetes is a really complex piece of container orchestration software, and especially in the context of a self-hosted cluster, we need to make sure that everything has been set up correctly. Let's perform one of the simplest tests, which is to run the same hello-world image, but this time, instead of running it directly through Docker API, we'll tell Kubernetes API to run it for us.
[code]
$ kubectl run --rm --restart=Never -ti --image=hello-world my-test-pod
(...)
Hello from Docker!
This message shows that your installation appears to be working correctly.
(...)
pod "my-test-pod" deleted
[/code]
That's a bit long command. Let's explain in detail what are the parameters and what just happened.
When we are using kubectl command, we are talking through the apiserver. This component is responsible for taking our requests (under the hood, HTTP REST requests, but we are using it locally) and communicating our needs to other components. Here we are asking to run an image of name hello-world inside temporary pod named my-test-pod (just a single container), without any restart policy and with automatic removal after exit. After running this command, Kubernetes will find a suitable node for our workload (here we have just one, of course), pull the image, run its entrypoint command and serve us it's console output. After the process finishes, pod is deleted and the command exits.
Testing - networking
The next test is to check if networking is set up correctly. For this, we will run Apache web server instance - conveniently, it has a default index.html page, which we can use for our test. I'm not going to cover everything in YAML configurations (that would take forever) - please check out pretty decent official documentation. I'm just going to briefly go through concepts. Kubernetes configuration mostly consists of so-called "resources", and here we define two of them, one Deployment (which is responsible for keeping our Apache server always running) and one Service of type NodePort (which will allow us to connect to Apache under assigned random port on the host node).
[code]
$ cat apache-nodeport.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: apache-nodeport-test
spec:
selector:
matchLabels:
app: apache-nodeport-test
replicas: 1
template:
metadata:
labels:
app: apache-nodeport-test
spec:
containers:
- name: apache
image: httpd
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: apache-nodeport-test
spec:
selector:
app: apache-nodeport-test
type: NodePort
ports:
- port: 80
$ kubectl apply -f apache-nodeport.yml
[/code]
It should be up and running in a while:
[code]
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
apache-nodeport-test-79c84b9fbb-flc9p 1/1 Running 0 25s
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
apache-nodeport-test NodePort 10.108.114.13 <none> 80:31456/TCP
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
[/code]
Let's try curl-ing our server, using a port that was assigned to our service above:
[code]
$ curl http://139.59.211.151:31456
<html><body>
<h1>It works!</h1>
</body></html>
[/code]
And we are all set!
How to survive in the Kubernetes world outside a managed cloud
Most DevOps people, when they think "Kubernetes" they are thinking about a managed offering of one of the biggest cloud providers, namely Amazon, Google Cloud Platform or Azure. And it makes sense - such provider-hosted environments are really easy to work with, they provide cloud-expected features like metrics based autoscaling (both on container count and node count level), load balancing or self-healing. Furthermore, they provide well-integrated solutions to (in my opinion) two biggest challenges of containerized environments - networking and storage. Let's tackle the networking problem first.
Kubernetes networking outside the cloud
How do we access our application running on the server, from the outside Internet? Let's assume we want to run a Spring Boot web hello world. In a typical traditional deployment, we'll have our Java process running on the host, bound to port 8080 listening to traffic, and that's it. In case of containers and especially Kubernetes, things get complicated really quickly.
Every running container is living inside pod, which has its own IP address from pod network CIDR range. This IP is completely private and invisible for clients trying to access the pod from the outside world.
The managed cloud solution, in case of Amazon for example, is to create a Service with type LoadBalancer, which will end up creating an ELB instance in your customer account (a whooping 20$/mo minimum), pointing to the appropriate pod. This is, of course, impossible to do in a self-hosted environment, so what can we do? Well, a hacky solution is to use NodePort services everywhere, which will expose our services at high-numbered ports. This is problematic because of those high port numbers, so we slap an Nginx proxy before them, with appropriate virtual hosts and call it a day. This will definitely work, but for such use case Kubernetes provides us something called Ingress Controller, that can be deployed into a self-hosted cluster. Without going into much detail, it's like installing Nginx inside Kubernetes cluster itself, which makes it possible to control domains and virtual hosts easily through the same resources and API.
Deploying official Nginx ingress is just a few kubectl apply commands away as usual, however, there is a catch - official YAML-s actually are using a NodePort service for Nginx itself! So that won't really work, because all our applications would be visible on this high port number. The fix to this is to do what Kubernetes explicitly discourages, which is to use fixed hostPort binding for 80 and 443 ports. This way, we'll be able to access our apps from the browser without any complications. Ingress definitions with those changes can be found at my GitHub, but I encourage you to see the official examples and start from there.
[code]
$ kubectl apply -f .
$ (...)
$ curl localhost
default backend - 404
$
[/code]
If you have a domain name for your server already (or you made a static entry in /etc/hosts), you should be able to point your browser at the domain and see "404 - default backend". That means our ingress is running correctly, serving 404 as a default response, since we haven't defined any Ingress resource yet. Let's use our Nginx Ingress to deploy the same Apache server, using domain names instead of random ports. I set up a static DNS entry in hosts file with name kubernetes-for-poor.test here.
[code]
$ cat apache-ingress.yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ingress-test
spec:
selector:
matchLabels:
app: ingress-test
replicas: 1
template:
metadata:
labels:
app: ingress-test
spec:
containers:
- name: ingress-test
image: httpd
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: ingress-test
spec:
selector:
app: ingress-test
ports:
- port: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-test
spec:
rules:
- host: kubernetes-for-poor.test
http:
paths:
- path: '/'
backend:
serviceName: ingress-test
servicePort: 80
$ kubectl apply -f apache-ingress.yml
deployment.apps/ingress-test created
service/ingress-test created
ingress.extensions/ingress-test created
$
[/code]
And pointing your browser at your domain should yield "It works!" page. Well done! Now you can deploy as many services as you like. An easy trick is to create DNS CNAME record, pointing from *.yourdomain.com to yourdomain.com - this way there is no need to create additional DNS entries for new applications.
Kubernetes storage outside the cloud
The second biggest challenge in the container world is storage. Containers are by definition ephemeral - when a process is running in a container, it can make changes to the filesystem inside it, but every container recreation will result in loss of this data. Docker solves it by the concept of volumes, which is basically mounting a directory from the host at some mount point in container filesystem, so changes are preserved.
Unfortunately, this by definition works only for single-node deployments. Cloud offerings like GKE solve this problem by allowing to mount a networked storage volume (like GCE Persistent Disk) to the container. Since all nodes in the cluster can access PD volumes, storage problem goes away. If we add automatic volume provisioning and lifecycle management provided by PersistentVolumeClaim mechanism (used to request storage from the cloud in an automated manner), we are all set.
But none of the above is possible on a single-node cluster (outside of managing a networked filesystem like NFS or Gluster manually, but that's not fun). The first thing that comes to mind is to go the Docker way and just mount directories from the host. Since we have only 1 node, it shouldn't be a problem, right? Technically true, but then we can't use PersistentVolumeClaims and we have to keep track of those directories, make sure they exist beforehand etc. There is a better way, though. We can use so-called "hostpath provisioner", which uses Docker-like directory mounts under the hood, but exposes everything through PVC mechanism. This way volumes are created when needed and deleted when not used anymore. One of the implementations which I've tested can be found here, it should work out of the box after just another kubectl apply.
Let's test it. For this example, we'll create a single, typical PVC.
[code]
$ kubectl apply -f .
$ (...)
$ cat test-pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Mi
$ kubectl apply -f pvc.yml
$ kubectl get pvc
test-pvc Bound pvc-a5109ca0... 200Mi RWO hostpath 5s
[/code]
We can see that our claim was fulfilled and the new volume was bound to the claim. Let's write some data to the volume and check if it is present on the host.
[code]
$ cat test-pvc-pod.yml
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
spec:
containers:
- name: myapp-container
image: ubuntu
command: ['bash', '-c', 'while true; do sleep 1 && echo IT_WORKS | tee -a /my-volume/test.txt; done']
volumeMounts:
- mountPath: /my-volume
name: test
volumes:
- name: test
persistentVolumeClaim:
claimName: test-pvc
$ kubectl apply -f test-pvc-pod.yml
(...)
$ ls /var/kubernetes
default-test-pvc-pvc-a5109ca0-b289-11e8-bc89-fa163e350fbc
$ cat default-test-pvc-pvc-a5109ca0-b289-11e8-bc89-fa163e350fbc/test.txt
IT_WORKS
IT_WORKS
(...)
[/code]
Great - we have our data persisted outside. If you can keep all your applications running on the cluster, then backups become a piece of cake - all the state is kept in a single folder.
Let's actually deploy something
We have our single-node cluster up and running, ready to run any Docker image, with external HTTP connectivity and automatic storage handling. As an example, let's deploy something everyone is familiar with, namely Wordpress. We'll use official Google tutorial with just one small change: we want our blog to be visible under our domain name, so we remove "LoadBalancer" from service definition and add an appropriate Ingress definition.
[code]
$ cat wp.yml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: wordpress-mysql
spec:
selector:
matchLabels:
app: wordpress-mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress-mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claim
---
apiVersion: v1
kind: Service
metadata:
name: wordpress-mysql
spec:
ports:
- port: 3306
selector:
app: wordpress-mysql
---
apiVersion: v1
kind: Service
metadata:
name: wordpress-web
labels:
app: wordpress-web
spec:
ports:
- port: 80
selector:
app: wordpress-web
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: wp-pv-claim
labels:
app: wordpress
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: wordpress-web
spec:
selector:
matchLabels:
app: wordpress-web
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress-web
spec:
containers:
- image: wordpress:4.8-apache
name: wordpress
env:
- name: WORDPRESS_DB_HOST
value: wordpress-mysql
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 80
name: wordpress
volumeMounts:
- name: wordpress-persistent-storage
mountPath: /var/www/html
volumes:
- name: wordpress-persistent-storage
persistentVolumeClaim:
claimName: wp-pv-claim
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: wordpress-web
spec:
rules:
- host: kubernetes-for-poor.test
http:
paths:
- path: /
backend:
serviceName: wordpress-web
servicePort: 80
$
[/code]
The only prerequisite is to create a Secret with the password for MySQL root user. We could easily and safely hardcode 'root/root' in this case (since our database isn't exposed outside the cluster), but we'll do it just to follow the tutorial.
[code]
$ kubectl create secret generic mysql-pass --from-literal=password=sosecret
[/code]
And we finally deploy.
[code]
$ kubectl apply -f wp.yml
[/code]
After a while, you should see a very familiar Wordpress installation screen at the domain specified in the Ingress definition. Database configuration is handled by dockerized Wordpress startup scripts, so there is no need to do it here like in a traditional install.
Conclusion
And that's it! Fully functional single-node Kubernetes cluster, for all our personal projects. Is this an overkill for a few non-critical websites? Probably yes, but it's the learning process that's important. How would you become a 15k programmer otherwise?
13 November 2018
0
Creating Countinous Integration system with GitLab and Unity
After spending days and nights working in the sweat of our brows and making tons of application builds, we finally decided to take a step forward and make some part of our job automatic. But... why ? The main reason of making our Continuous Integration system was shortening amount of time spent on building applications. In our last project it lasted even above half an hour, so we decided to make our life easier. We had a list of features we wanted to achieve:
- builds available online to download
- working on multiple unity versions
- building for iOS, Windows and Android on our Windows device
- builds available on our local server
( Our continuous integration system in few simple steps)
About GitLab CI
We chose GitLab CI solution because we had already possessed repositories there. How to configure GitLab and a Runner we described in a separate article:
When we set up our GitLab Runner we had to configure GitLab CI itself by .gitlab-ci.yml
Our system works in two stages : build and deploy. Job named “build” is a template for other jobs. It contains a powershell script which run the unity in batchmode and execute script we wrote in C#.
To do such a thing we need Unity command lines :
| C:\’Program Files’\Unity\Editor\Unity.exe | call unity executable |
| -batchmode | execute in batchmode to prevent the editor from opening |
| -nographics | needed on windows to really make sure there is no GUI |
| -executeMethod BuildScript.Build | calls the Method “Build()” in the Unity BuildScript |
| -projectPath %CI_PROJECT_DIR% | sets the unity project path to default GitLab directory |
| -quit | quit Unity after execution of the build |
Keep it planned
All builds can be made on demand, but also after every commit in master branch and everyday at 2 AM. Beside that we are able to configure it in GitLabCI/Schedules.
We are also setting a unity version as a variable on the top of our file to be easily configurable at the start of the project.
Where we can find our builds?
We wanted to save builds directly to our local server, but we found out that it was impossible to achieve. We had to workaround it, which we didn't want to use in the first place - copying from a temporary folder on our disc. We were afraid that it will fill the disc too fast with unnecessary data so we had to also clean the folder after completion. That’s what we are doing in the deploy stage. Builds are also available as the artifacts in GitLab.
As fast as possible
To make the whole process faster we decided to start using unity cache server which helps us to save time at changing between different platforms in unity. When it’s working Unity is downloading converted assets from cache server instead of converting them everytime.
Building iOS apps on Windows
One of our goals was to build everything on one machine and didn’t involve our mac. The major problem was iOS building, because we thought it was impossible to build an .ipa file on Wndows machine, but we have discovered a plugin that helped us fix the problem. Of course it wasn’t one click configurable, we had to transfer all the certificates from our mac etc. Never the less it was worth the pain. We are also able to use it by command lines so it is perfect for our purposes.
Try it yourself
So this is our story of creating continuous integration system with GitLab and Unity. It was harder than it seems to be, but I’m pretty sure that we will keep improving it. The system makes our workflow much more simpler and faster. Don’t wait, go and make yourself your own one!
21 September 2018
0
I’ve got the power – how to control remotely your PC using Grails
In this article I will show you how to control remotely your windows computer via the local network using Linux machine and Grails application. First we will turn on a device with Wake-On-LAN (WOL) method and then turn it off using RPC shutdown command call. After this few steps, you won't have to push the power button on your computer any more. 1. Turn on a computer - wake up, darling!!! To begin with we have to check a few configuration things:
- check if BIOS allows to use Wake-On-LAN on the computer. Go to BIOS power management settings and find the Wake-On-Lan configuration there. If related option does not exist it probably means that BIOS automatically supports WOL - most of new hardware does that. When you find an appropriate option check if it is enabled. Depends on your computer motherboard, it is automatically enabled or you will have to set it manually.
- check your system configuration. Open Windows Device Manager, find your local network device. Right click on a device and select "Properties". Next, select Advanced tab, find a "Wake on Magic Packet" option and check if it is enabled.
Now it's time to write a piece of code which will send a Wake-On-LAN magic packet to your device. For this purpose I have created project in Grails 3.2.0. Create a new service called "WakeOnLanService".
First method converts device's MAC address from String to byte array:
[code]
private static byte[] getMacBytes(String macAddress)
throws IllegalArgumentException {
byte[] bytes
String[] hex = macAddress.split("(\\:|\\-)")
if(hex.length != 6) {
throw new IllegalArgumentException("Invalid MAC address.")
}
try {
bytes = hex.collect { (byte) Integer.parseInt(it, 16) }
} catch (NumberFormatException e) {
throw new IllegalArgumentException("Invalid hex digit in MAC address.")
}
return bytes
}
[/code]
This method allows to pass MAC address as an argument in two formats "AA:AA:AA:AA:AA:AA" or "AA-AA-AA-AA-AA-AA". Second method generates a magic packet from received MAC address byte array:
[code]
private static byte[] generateMagicPacket(byte[] macBytes) {
ByteArrayOutputStream byteStream = new ByteArrayOutputStream()
6.times { byteStream.write(0xff)}
16.times { byteStream.write(macBytes) }
return byteStream.toByteArray()
}
[/code]
Finally the last method is used for broadcasting magic packet to all devices in your local network:
[code]
static final int PORT = 9
static final String DEFAULT_BROADCAST_IP = '255.255.255.255'
boolean sendWOLPacket(String macAddress, String broadcastIp=DEFAULT_BROADCAST_IP) {
if(!macAddress) {
return false
}
byte[] wol= generateMagicPacket(getMacBytes(macAddress))
InetAddress address = InetAddress.getByName(broadcastIp)
DatagramPacket packet = new DatagramPacket(wol, wol.length, address, PORT)
DatagramSocket socket = new DatagramSocket()
socket.send(packet)
socket.close()
return true
}
[/code]
Now you should be able to start your windows machine remotely. PORT and BROADCAST_IP are the default values and can be changed e.g. when you want to send magic packet through a router.
2. Turn off the computer - just stop talking to me
To turn off the Windows machine we will use a shutdown method executed by remote procedure call (RPC). To do that we execute a samba net command:
[code]
net rpc shutdown -I {ipAddress} -U {username}%{password}
[/code]
IpAddress is a remote windows machine IP address, username and password are credentials of a windows user with administrative privileges.
Below are the configuration steps needed:
- Firstly, check if your Linux system has samba packages installed. We will use "net rpc" command to communicate with remote computer. For Ubuntu/Debian Linux distribution it is included in samba-common-bin package. To install samba use console command "sudo apt-get install samba-common-bin".
- Configure your windows machine to disable UAC remote restrictions. Locate the following registry subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System. Check if the LocalAccountTokenFilterPolicy registry entry does exist. If not, create a new entry with DWORD value "1".
- check if firewall has opened port 445 for TCP connection. When not exist then you should add a new role in the firewall.
21 August 2018
0
Populate database in Spring
Once upon a time there was BootStrap class. The class was a very friendly class, therefore it befriended many more classes. With more classes inside, BootStrap grew bigger and bigger, expanding itself at the rate of the entire universe. This is not a fairy tale.
This is not a fairy tale, because this is exactly what happened. But first things first, you may wonder what the BootStrap is. It's a mechanism coming from Grails Framework, which executes code on application startup; I'm using it mostly to populate database. Just put the BootStrap.groovy in grails-app/init folder and add some goodies to the init closure. Having a background in Grails, this is something I missed in Spring, especially because as I mentioned before, the code grew fairly big. I wanted to rewrite the whole BootStrap logic in Java, because its older Groovy version somehow reminded me of poorly written tests you may see here and there: verbose and ugly. It just wasn't a first-class citizen of the production code.
[java]
@Log4j @AllArgsConstructor
@Component
public class BootStrap {
private final BootStrapService bootStrapService;
@EventListener(ApplicationReadyEvent.class)
private void init() {
try {
log.info("BootStrap start");
bootStrapService.boot();
log.info("BootStrap success");
} catch (Exception e) {
log.error("BootStrap failed," + e);
e.printStackTrace();
throw e;
}
}
}
[/java]
Surprise, surprise! There's an EventListener annotation that you can put on a method in order to track the ApplicationReadyEvent and run some code on application startup. Job's done, right? Well, not really, you CAN do that, but do you WANT TO do that? I prefer to keep the business logic in a service, therefore I created and injected BootStrapService, that just leaves logging and error handling here and Lombok's annotations make the whole thing even neater.
[java]
public abstract class BootStrapService {
@Autowired
protected BootStrapEntryService entryService;
@Autowired
protected MovieService movieService;
@Transactional
public void boot() {
writeDefaults();
}
protected void writeDefaults() {
entryService.createIfNotExists(BootStrapLabel.CREATE_MOVIE, this::createMovie);
}
private void createMovie() {
movieService.create("Movie");
}
}
[/java]
I made the boot method @Transactional, because it's our starting point to populate database, later in the project you might want to add here some data migration as well. The REAL rocket science begins in writeDefaults method! The example createMovie method reference is passed as a parameter, with double colon as a syntactic sugar, to the createIfNotExists method of the injected BootStrapEntryService. The other parameter is a simple BootStrapLabel enum, with a value used as a description for a given operation. I prefer to add a verb as a prefix, just not to be confused later, when a possibility of other operations comes up.
[java]
@Log4j @AllArgsConstructor
@Transactional
@Service
public class BootStrapEntryService {
private final BootStrapEntryRepository bootStrapEntryRepository;
public void createIfNotExists(BootStrapLabel label, Runnable runnable) {
String entryStatus = "already in db";
boolean entryExists = existsByLabel(label);
if(!entryExists) {
runnable.run();
create(label);
entryStatus = "creating";
}
log(label, entryStatus);
}
public boolean existsByLabel(BootStrapLabel label) {
return bootStrapEntryRepository.existsByLabel(label);
}
public BootStrapEntry create(BootStrapLabel label) {
BootStrapEntry bootStrapEntry = new BootStrapEntry();
bootStrapEntry.setLabel(label);
return bootStrapEntryRepository.save(bootStrapEntry);
}
private void log(BootStrapLabel label, String entryStatus) {
String entryMessage = "processing " + label + " -> " + entryStatus;
log.info(entryMessage);
}
}
[/java]
Finally, createIfNotExists method is the place to call the actual methods to populate database, however, in a generic way. Method passed as a reference may be called, however, we don't want to write the data that was written to the database before, at least considering a not in-memory database, so we check if an entry for a given label already exists. We have to create an entity, a pretty simple entity, in this case the BootStrapEntry, with just a label field to keep the labels in database. existsByLabel and create are simple generic methods responsible for basic database operations on the labels.
[java]
@Profile("development")
@Service
public class DevelopmentBootStrapService extends BootStrapService {
@Autowired
private BookService bookService;
@Override
protected void writeDefaults() {
super.writeDefaults();
entryService.createIfNotExists(BootStrapLabel.CREATE_BOOK, this::createBook);
}
private void createBook() {
bookService.create("Book");
}
}
[/java]
Now we're getting somewhere! If you wondered why I made the BootStrapService abstract, now is the answer. I wanted to make it possible to run some code only in a given environment, like development. The Profile annotation with environment name as a parameter comes in hand. Overriding the writeDefaults method provides a way to initialize some data only in development.
[java]
@Profile("production")
@Service
public class ProductionBootStrapService extends BootStrapService {
}
[/java]
If there is the development, there may be the production as well. In the given example, I just wanted to run the default data from the parent class, without filling the environment with some random data.
That's all, my little guide to populate database in Spring. Hopefully, it wasn't THAT bad and even if it was, feel free to leave the feedback nonetheless!
14 May 2018
0
How GitLab helps us move fast at itSilesia
Continuous Integration and Continuous Delivery have taken the world by storm. The nature of our business forces development teams to move quickly and be as efficient as possible, both in regards to standard software development, but also it’s delivery and quality assurance. In order to achieve these goals, we need tools that will enable us to reach them in a simple way. Thankfully, today we have access to a lot of CI/CD solutions, both free and paid, all very different, with different features and different goals in mind. In essence they all do the same thing - allow us to work smarter, not harder.
Our previous development infrastructure consisted of three parts. For issue tracking and Scrum process we used Redmine with Scrum Board plugin. It worked fine and served us great for years, but from today’s perspective it's UI is seriously outdated and generally hard to use. We probably could give it a try to upgrade it, but sadly it is so old that we really do not want to try it. For hosting Git repositories we used Gitolite Redmine integration. It also worked fine, but it's functionality is nonexistent compared to modern solutions, it lacks really basic functionalities like Pull Requests or commenting on commits. For continuous integration we were (and somewhat still are) using Jenkins. Now, don't get me wrong - Jenkins is great and very powerful, requires minimal setup, can be scaled and we achieved some great success with it, especially with use of Groovy Pipelines. But here is the question, if there is a tool (GitLab, if you didn't read the title) that could integrate all of the above, even for some loss of customizability and potential vendor lock-in, is it worth it? Well, let's check it out!
1. Setup
How do we start then? The first decision to make is whether we want to use GitLab.com hosted service, or create our private GitLab instance on company infrastructure. This decision was simple to make - we don't want to store the most valuable company asset (the source code) on public infrastructure, and given the recent problems with GitLab accessibility - this is just a better option. There are multiple ways of installing GitLab, but the easiest way for us was using prebuilt "batteries included" Docker image, which contains all services in one big bundle. This way of packaging multiple applications into one monolithic Docker image is a bit controversial in the community (the best practice is to join individual processes running in separate containers using Docker Compose, for example), but in my opinion it works great for such big and complex software packages, because all the internal complexity is completely hidden away. As far as installation is concerned, we already have a few Docker hosts, so running an additional container somewhere isn’t a big deal.
Documentation specifies that the following Docker invocation is a good start:
[code]
sudo docker run --detach \
--hostname gitlab.example.com \
--publish 443:443 --publish 80:80 --publish 22:22 \
--name gitlab \
--restart always \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
[/code]
As with everything shipped in a form of a Docker image, we could just blindly copy and paste it into a privileged console... but let's slow down. Do we really understand what these parameters mean? First of all, on most Linux systems port 22 is already used by SSH server, so another binding on port 22 will surely conflict. SSH port is used by GitLab to run it's internal SSH Git server. Since we are perfectly happy with using only HTTPS as before, we can remove this binding altogether. Also, in our case (and in most cases on typical production systems) HTTP(S) ports (80 and 443) are already taken by some Apache or Nginx web server running natively. Since we wanted to use our external Apache proxy (which was also doing SSL termination) on the company edge router, we had to change HTTP port binding to some other random-ish value. Also, we can remove the hostname, it seems not to affect anything.
After a few minutes of some heavy internal provisioning and initialization, you can visit the main page. It will ask for an administrator password first, then it will allow to log in. I have to admit that I was very pleasantly surprised by how easy this setup process was.
I will not get deep into typical administrative stuff. As a first thing you probably want to create users, assign them to groups, create repositories and hand out permissions. It's not really interesting to be honest.
2. Projects
The next topic is migrating projects. Since Git is a decentralized system, it doesn't really matter how many repository servers are used for a single project. However, even though we could work this way (and technically we are), we generally don't think in a decentralized way - typically there is just one central repository. This becomes a challenge, when you want to migrate existing projects when developers are working on them at the same time. The first step is fairly easy - open a console (you are using Git from console, right?), create a new Git remote, and then push all branches to newly created GitLab repository. In the following examples I'll use a very simple Spring Boot application (you can find it here).
[code]
cd ~/luckynumber
git remote add gitlab \
https://gitlab.gliwice.itsilesia.com/agrzybowski/luckynumber.git
git push -u gitlab --all
git push -u gitlab --tags
[/code]
After some uploading, you will end up with two identical repositories.
And...
The issue is that now we have 2 remotes, which means that if you previously had branch develop (and it's remote tracking branch counterpart origin/develop), now there is a new tracking branch gitlab/develop. And that's only on your local repository - other team members can't know (by definition of decentralized model) that there is another remote somewhere. There are two ways of dealing with this. The easiest way is to go around the office and yell "Guys, please copy your files, delete the project and reclone from scratch". And this might work, but expect a lot of hasty copy-pasting and useless bulk commits after this procedure.
There is of course a better way. Git has an option to simply change the remote URL. Send your coworkers a Slack message to run this:
[code]
cd ~/luckynumber
git remote set-url \
https://gitlab.gliwice.itsilesia.com/agrzybowski/luckynumber.git
[/code]
Be careful, though - all remote branch references (origin/xxx) will immediately get out of sync, so push their local counterparts as soon as possible. It’s a good idea to treat remote branches as volatile and not get too emotionally attached to them, because they can be overwritten at any time (remember, git rebase is your best friend, if used correctly). If they are important, don’t use them - just creating a local branch pointing to some commit will make it persistent forever.
3. The interesting part
Now let's focus on CI/CD part. Glancing over the docs, the first thing that pops up is that GitLab CI is very tightly integrated to the source code repository. All configuration (in form of YAML file) is stored directly in the repository, jobs are based on branches and triggered on pushes, pull requests have little red/green checkmarks with build statuses and so on. Is this a good idea altogether? In my opinion it is, but only for the CI part, like running tests or collecting code quality metrics. Deployment (especially on production environments) should be decoupled from any version control system the application happens to use. But since we can get away with this in our internal projects (we have relatively little risk and hopefully responsible developers), we decided to go 100% GitLab and do our production deployments also there.
GitLab CI configuration is stored in .gitlab-ci.yml file. It has a specific structure, which we will not cover in full here. The documentation is very comprehensive, and you can also find some examples online. We will be building an almost barebones Spring Boot application with a few unit tests.
[code]
image: openjdk:8-jdk
stages:
- test
- package
- deploy
test:
stage: test
script:
- ./gradlew test
package:
stage: package
only:
- master
- develop
script:
- ./gradlew bootRepackage
artifacts:
paths:
- build/libs/luckynumber-0.0.1-SNAPSHOT.jar
deploy-test:
stage: deploy
only:
- develop
script:
- ./deploy-test.sh
deploy-prod:
stage: deploy
only:
- master
script:
- ./deploy-prod.sh
[/code]
There is a lot of stuff going on here, let's break it down. First of all, the recommended way of building/testing/packaging your application is by using ephemeral Docker containers for creating temporary and 100% reproducible build environments. We went through a lot of pain on traditional Jenkins because of conflicting JVM versions, badly installed Grails distributions or even globally installed NPM packages (which "no one remembers installing"...). Using Docker containers removes this problem completely - every time a build is triggered there is a guaranteed fresh environment available. Here we specify OpenJDK public image, but you can use any, even your own. After setting up the image name, we define build stages. We use very typical steps - build/package/deploy, but they are of course arbitrary. For each stage we can define:
- branch constraints - for example, deploy to production only from master,
- artifacts - they are preserved across stages and stored internally in GitLab for later download,
- a Bash script - defining what to actually do.
After committing .gitlab-ci.yml file to repository you can head to the tab CI/CD and look at a newly triggered job. It will be in a paused state... because we didn't tell GitLab yet how to run our builds! For this, we will have to add a so-called GitLab Runner. What is a Runner? In essence, it is a machine which will execute our jobs. It can be a physical server, a virtual machine, or even a throwaway AWS EC2 instance running Docker Machine on autoscaling cloud infrastructure! Okay, getting back on the ground, since our needs are rather modest, we decided to use a small KVM virtual machine, running on the same server GitLab itself is running on. It's explicitly discouraged by the documentation for management and security reasons, but we have 24 GiB of RAM and a lot of disk space there, so there shouldn't be any capacity problems, and top performance isn't really needed. We will be using a Docker executor, which is just a small service that needs access to Docker daemon on the underlying host.
[code]
sudo wget -O /usr/local/bin/gitlab-runner \
https://gitlab-runner-....../gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd \
--comment 'GitLab Runner' \
--create-home gitlab-runner \
--shell /bin/bash
sudo gitlab-runner install \
--user=gitlab-runner \
--working-directory=/home/gitlab-runner
sudo gitlab-runner start
[/code]
And as a last step, we need to register the runner, so it becomes visible in GitLab and it can be used to schedule jobs. Of course, you can add more than one runner - GitLab will automatically load-balance jobs across them. You can also do some fancy stuff, like locking a runner for specific project or attaching tags for easier administration. Here we just answer the questions one by one. We can obtain GitLab-CI token for a shared runner from administration page (Admin area > Runners) and paste it here, it needs to be done only once. We can safely enable this runner to run multiple projects and run all jobs.
[code]
$ sudo gitlab-runner register
Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
https://gitlab.gliwice.itsilesia.com
Please enter the gitlab-ci token for this runner
...
Please enter the gitlab-ci description for this runner
[gitlab] its-runner
Please enter the gitlab-ci tags for this runner (comma separated):
Whether to run untagged jobs [true/false]:
[false]: true
Whether to lock Runner to current project [true/false]:
[true]: false
Please enter the executor: ssh, docker+machine, docker-ssh+machine,
kubernetes, docker, parallels, virtualbox, docker-ssh, shell:
docker
Please enter the Docker image (eg. ruby:2.1):
debian:sid
Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!
[/code]
Right after applying those settings our newly created pipeline should kick off automatically. Opening CI/CD tab, we can see the individual pipeline stages, and show the console output.
And what about deployment? Surely we need to get access to some production servers via SSH or in some other way? That's the problem we discussed earlier. Should GitLab have access to external servers at all? That's a tough question, and there is no easy answer for all use cases. We decided that the gained simplicity and full CI/CD integration is worth the extra risk. But in any case we absolutely shouldn't store any credentials or passwords inside repository, ever. GitLab has a simple Secrets feature, which allows us to define special variables, that at the time of job execution will get injected into environment. We defined a single SSH_PRIVATE_KEY variable, containing a private key, that will be subsequently used to access our test server. But there is a catch! All jobs always run in containers, and we need to make sure that our container has all the tools necessary to perform a deployment, even basic ones like ssh-agent or bzip2. We can use a package manager to quickly install what’s missing in the image, or simply use a bigger image (like ubuntu) with all tools preinstalled.
For our SSH deployments, we created simple script (idea was taken from the official docs) that configures ssh-agent with a private key required to access the server. This simple script starts ssh-agent, disables SSH Host Key Checking and adds a private key to the in-memory store. Disabling host checking is sadly necessary, otherwise we would have to manually approve connections every time (not once - remember the part about fresh containers?), and we can't do that - there is no keyboard/terminal input when job runs.
[code]
#! /bin/bash
set -e
echo "Configuring ssh access..."
which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
eval $(ssh-agent -s)
ssh-add <(echo "$SSH_PRIVATE_KEY")
mkdir -p ~/.ssh
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
echo "SSH configured successfully, added ssh keys:"
ssh-add -l
[/code]
And we are basically done! From this moment, every push to the repository will trigger a pipeline run. Tests will be executed for all branches (including feature ones), and an appropriate package+deploy procedure will be ran for develop and master branches.
There is still one last thing missing: the actual deploy script. This is of course dependent on the type of application and target environment. For frontend applications, we might build necessary assets using NPM and Webpack, then upload them through (S)FTP to a specific folder for Apache to serve. For Java apps deployed in Tomcat, we might build the .war file and put it into Tomcat directory, where the change will be detected and the app restarted. For Docker Swarm environments, we can build all necessary images, push them to private registry and trigger docker service update or docker stack deploy on the remote cluster manager node.
That’s everything we have for now. We’ve been using this setup for one of our legacy projects for a few months and it works very well. Our development team is very happy to have one, integrated experience and very visible progress. Stay tuned for updates!
But wait, there’s more!
We wanted to host GitLab on company network and expose it to the world through external HTTPS proxy. At first, it seemed that setting instance URL in administration panel to its HTTP equivalent is enough, because it worked for cloning and browsing. Unfortunately, there is a nasty hidden problem in doing only this. You see, when GitLab is accessed through external HTTPS proxy, it can't be aware of this fact, and as a result all links inside GitLab have http:// and not https:// prefix. Not a big deal - developers can change the URL once when cloning and then forget about it, yes? Well, that's technically true, however if you try to kick off any CI build you will find out that it mysteriously fails with empty page without any errors. It turns out that in some very rare cases the frontend part (and also the runner part) of GitLab actually uses the real configured URL of GitLab to retrieve data from the server. We need to tell GitLab it's real external URL - but there is another catch! If you just specify https:// URL per documentation, it actually disables HTTP completely. Argh! The following manual configuration solves this problem.
[ruby]
external_url 'https://gitlab.gliwice.itsilesia.com'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['proxy_set_headers'] = {
'X-Forwarded-Proto' => 'https',
'X-Forwarded-SSL' => 'on'
}
[/ruby]
5 March 2018
0
Star Wars opening crawl based on CSS animations and transformations
10 years ago nobody expected how frontend is going to evolve in the future. Developers from around the world were using CSS for basic styling because then only that was possible with CSS. Complex things, like animations or transformations, were made thanks to plain Javascript or, very popular at this time, jQuery framework.
Currently, with CSS 3, HTML 5 and tons of Javascript frameworks for every possible use, we are in a totally different programming world. CSS has immense ecosystem of styles, which help us create, color, filter, transform or even animate objects on a screen. In this article I want to show a simple way to create animated Star Wars opening crawl using only HTML and CSS. There are a couple of methods to achieve this goal but we will try to choose the one that has the best performance. You may ask: "Why Star Wars opening crawl? There are so many interesting topics to choose from." Yeah, it's true but this task is relatively easy to implement and, what is more important, we need the most important CSS styles for animations and transformations to achieve it. And I love Star Wars.
1. Setup
At first, we have to prepare HTML and basic CSS for displaying opening crawl. The only thing we need in HTML body is this simple piece of code:
[html]
<p>Here put your opening crawl story</p>
[/html]
Now we need to apply basic styles to the background of our scene and to the crawl's content. Original Star Wars crawl contains blackish background and yellow text sliding on a screen. This can be done simply by adding this CSS code:
[css]
body { background: #111; }
p {
margin: 0 auto;
color: #fcd000;
font-size: 30px;
font-weight: bold;
width: 600px;
}
[/css]
Nothing special - just yellow text on a black background, you probably did it in your primary school. Fortunately, now we are going to use all the magic of CSS to make it ride.
2. Transformations
As you can see on the featured photo, our text must be leaned. To make it work we need one simple but very powerful style - transform. It prpvides us a lot of transformation functions but the most important are these, which are connected with translation, rotation and scaling. Currently, on most browsers we can use it in 3 dimensions (x, y, z).
In our case try to imagine that you put our <p> text into XYZ plot. We want to lean this text forward, in mathematical words, rotate around X axis.
And this is also pretty easy to do with CSS - just add this chunk of code to <p>:
[css]transform: rotateX(30deg)[/css]
If you try to write the code along with me, you can see that this is certainly not what we expected. It looks more like shrinking than rotating with perspective. Why? The keyword here is the perspective. We didn't declare this perspective, so browser did it for us and set perspective to none. As a result, there isn't any visible perspective. How can we declare a perspective? There are two methods which are a little bit different. We can attach perspective to the transformed object or a parent of this object. In that case there is no difference but it's extremely important to use the second approach if you want to transform more than one object. If you attach perspective to the parent, all children will belong to one perspective. And this is how it works in real life! On the other hand, you may find situations where "perspective per object" approach is better - mostly when you don't need realistic behaviour but only nice looking effect.
In our task we will use parent's perspective. To do this we have to put the code below into our styles (in our case these are <body> styles). For the sake of testing try this value:
[css]perspective: 50px[/css]
I must be honest, for a long time (waaaaay too long) I was writing random pixels' value into this property and checking if it's correct. But inside, I've always wanted to understand what these pixels mean. Nowadays, my beard is thicker, I switched from tea to coffee and finally I understand perspective property, yay! MDN documentation explains, that this value is a distance between user's eyes and z = 0 value on invisible plot. I will try to explain it more clearly. If you make perspective value very small, it looks like you are standing very close to the origin of object. Vice versa - if you provide really big value, you see an object from a really far distance. With this knowledge you can estimate what value you should type here. In our case our block of text is 600 pixels wide. We want to get the effect of seeing the crawl very close, so we should consider value, which is much lower than this 600 pixels. In our test we used 50 pixels but we can see that we "are standing" almost inside the block of text, so we should increase it to at least 150 pixels.
That should create a pretty nice effect of static Star Wars opening crawl.
[css]
body {
background: #111;
perspective: 200px;
}
p {
font-size: 30px;
color: #fcd000;
font-weight: bold;
width: 600px;
margin: 0 auto;
transform: rotateX(30deg);
}
[/css]
3. Animation
We've got the text on a screen, it's leaned as we wanted but we still need to make it move. At this moment we have to decide which way we choose - and this will result in how efficient our code will be. Practically, we have three significant methods to do this - 1. margin-top property, 2. position absolute/fixed with top property or 3. transform property using translate relative to Y axis.
Modifying margin is the worst choice. Seriously, never ever do that! Margin property was never created by browsers developers to use it in complex animations. Okay, in this simple example of one animated block of text, it's possible that you won't see any difference. But try to animate hundreds of objects on a screen with margin and you will see that you should forget about this method. And if you need just a simple animation it's still not a good choice. Why would you use something worse if you have something better? Speaking about something better..
..top property. Certainly, it is a better option, but still not the best. In the current frontend world there are many popular applications which use top/left properties to animate movement on scrollbars, sliders or animated dropdowns. But these examples are relatively simple tasks for our powerful devices (don't tell me your 2014 smartphone is not powerful, it is powerful enough). It has one advantage over next method - it's more compatible so if you need an effect which works on older browsers maybe it's better.
For complex visualizations or simple games using many animations you should always choose transform over margin or positioning. Browsers have different approaches to render DOM but mostly they use a different way to render elements transformed with transform property. For example, on Chrome, elements are taken into its own layer of GPU (RenderLayer), what causes frames to be drawn quicker. If you are interested in understanding it more precisely, try to make a simple animation with position absolute and check "Performance" section in Chrome Developer Tools. Then do the same with transform property. You will see differences in rendering times and GPU layers used to render it.
Okay, so we all agree that in that case transform is the best choice. We will stay with the transform property because it will be rendered on a separated GPU layer and we will create a crazy fast opening crawl animation. But you may ask: "Where is this animation mechanism?" Transform on its own just sets specific translation, rotation or scale, STATICALLY, nothing more. To make it move we need animation property - next fancy CSS feature. And this is quite tricky and complex feature. With animation property we can define duration, delay, name, iteration count, direction, even timing function of animation! But to simplify, we will leave most of these properties default. Without more ado, let's make it moooove. Add this code into the <p> styles:
[css]animation: animateCrawl 20s[/css]
We declared that this block will use animation called "animateCrawl" and it will take 20 seconds. But we still didn't define this animation and to do this we need to fill our animation's keyframes.
[css]
@keyframes animateCrawl {
0% { transform: rotateX(30deg) translateY(400px); }
100% { transform: rotateX(30deg) translateY(-300px); }
}
[/css]
For those who don't understand what does this syntax mean - first of all we need to enter keyframes expressed in percents. Then we define object's styles in these moments. Animation mechanism will automagically interpolate the difference in time what causes the animation effect. In result, our block of text will move according to Y axis, from the bottom to the top.
If you are observant, you may see that I also added rotateX value into transform property, which was added before. At the beginning of my frontend adventure I was asking myself: "Why can't I use only translation here if I want to animate only translation?". Answer is simple - the consequence. CSS treats every style as a pair - property name and value. Transform is just a single property name, so if you want to make a few transformations, you must put them all into this transform style. If you type only translateY transformation, you will simply override previously added rotateX with a default value. Not very comfortable but we must accept what CSS gives us.
If you launched this animation, you may see a small problem - at the beginning the movement is really fast and it slows down after every frame. From physical perspective it is done properly but original Star Wars opening crawl was moving uniformly. To fix it we need to change animation timing function. It gives us a lot of freedom, we can even define our own cubic bezier function! To simplify we will use built-in timing function called "ease-in". Mechanism of "ease-in" is pretty easy to understand - every frame animation is faster than frame before. And this is exactly what we need. Perspective gives an impression of slowing down, "ease-in" function accelerates in time so in result we get steady movement. Of course, if you want to make it very precisely, you should consider using custom cubic bezier function.
[css]animation: animateCrawl 20s ease-in;[/css]
And it's done. Star Wars opening crawl. Now you can use this knowledge for various CSS animations.
If you want to see working solution, check this codepen: https://codepen.io/Mossar/pen/rGpWqX
17 November 2017
0
Refreshed image
A polish classic says: "sit deeply in an armchair, tighten your belts and start". We do not have a ribbon to cut, so another chapter in company's history will be immortalized by an inaugural entry on a blog. itSilesia has metamorphosed and returns with a new image. We performed fundamental changes - rebranding met a building renovation and a new blood in our team. We count on that the new website and company materials' design will simplify access to any information you need.
31 October 2014
0
0