Category Archives: Ubuntu

Running PHP 5.3/5.6 in a Docker container with Nginx on Ubuntu 16.04

The docker installation procedure depends on the system architecture that can be determined with the following commands:

cat /proc/cpuinfo
dpkg --print-architecture

After I installed Docker, I successfully run its test application:

docker run hello-world
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
b04784fba78d: Pull complete
Digest: sha256:f3b3b28a45160805bb16542c9531888519430e9e6d6ffc09d72261b0d26ff74f
Status: Downloaded newer image for hello-world:latest
 
Hello from Docker!
This message shows that your installation appears to be working correctly.
 
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
 
To try something more ambitious, you can run an Ubuntu container with:
 
docker run -it ubuntu bash
 
Share images, automate workflows, and more with a free Docker ID:
 
For more examples and ideas, visit:

(more…)

Using “apitrace” with an OpenGL application

The basic commands for using apitrace with an OpenGL application are:

apitrace trace --api=gl MyApp
qapitrace MyApp.trace
apitrace dump --blobs MyApp.trace

The last command generates blobs that can be inspected with the following command, for example:

tail -c $((12*10)) ~/repos/MyApp/build/blob_call2325.bin | xxd -g1

Or by writing a simple C++ program that converts them into CVS format:

(more…)

Securing Nginx with Let’s Encrypt on Ubuntu 16.04

First we need to install certbot utility:

apt-get install software-properties-common
add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python-certbot-nginx

After that, we can easily generate SSL certificates for all the domains listed in Nginx ‘server_name’ attributes in alive (working) virtual hosts. The examples are:

certbot --nginx certonly -d slogpost.ru -d www.slogpost.ru
certbot --nginx certonly -d psiholog-s-vami.ru -d www.psiholog-s-vami.ru
certbot certonly --nginx --cert-name sharlines.com -d sharlines.com -d www.sharlines.com
certbot certonly --nginx --cert-name developernote.com -d developernote.com -d www.developernote.com -d herb.developernote.com -d mastermag.developernote.com -d geographx.developernote.com -d geographx.net -d www.geographx.net -d xn--80acc2atiigge7h.xn--p1ai -d www.xn--80acc2atiigge7h.xn--p1ai -d git.developernote.com -d gitweb.developernote.com

(do not forget to run the commands above each time you add or remove a subdomain)

We cannot use wildcard domains line *.developernote.com with Let’s Encrypt, so we should list all the subdomains. And I do not see anything wrong in combining multiple domains in a single certificate.

To remove the certificate we do something like this:

certbot revoke --cert-path /etc/letsencrypt/live/developernote.com/fullchain.pem
certbot delete --cert-name developernote.com

Updating all the generated certificates:

certbot renew

After changing the website URL from HTTP to HTTPS, probably it makes a sense to update all the hyperlinks in MySQL database:

show tables;
show columns from wp_posts;
SELECT ID, post_title, post_date, post_name FROM wp_posts WHERE INSTR(post_content, 'http://slogpost.ru') <> 0;
UPDATE wp_posts SET post_content=REPLACE(post_content, 'http://slogpost.ru', 'https://slogpost.ru') WHERE INSTR(post_content, 'http://slogpost.ru') <> 0;
UPDATE wp_posts SET post_content=REPLACE(post_content, 'http://developernote.com', 'https://developernote.com') WHERE INSTR(post_content, 'http://developernote.com') <> 0;

The final step is adding certbot-renew.sh file to /etc/cron.monthly with the following content:

certbot renew
service squid reload

It seems like the service … command is completely ignored. Nothing in syslog, nothing in nginx logs. I switched to using

certbot renew
systemctl reload squid

instead, and this seems to work.

Backing up and restoring all the PHP MySQL websites on a Linux server

All the following commands assumes we saved MySQL root password into MROOTPASS variable:

export MROOTPASS=<mysql root password>

The most straight forward method to backup all the MySQL databases and all the website files (PHP scripts, images, etc..) stored in the /home directory is the following:

mysqldump --all-databases -u root -p$MROOTPASS | gzip > all-databases-$(date '+%Y-%m-%d_%H-%M-%S').sql.gz
tar -cvzf home.tar.gz /home

If we backup some individual database (probably not as root) and change its user while restoring it, it might make a sense to remove DEFINER from the output script:

sed -e 's/*]*\*/\*/'

The following commands restore all the websites from the archives:

gunzip -c all-databases-2017-05-23_15-31-00.sql.gz | mysql -u root -p$MROOTPASS
cd /
sudo tar -xvzf home.tar.gz

After migration from MySQL version  14.14 Distrib 5.5.54 to 14.14 Distrib 5.7.18 (I do not know what is the difference between them) I got the following error: “ERROR 1805 (HY000) at line 1: Column count of mysql.user is wrong. Expected 45, found 42. The table is probably corrupted” while trying to drop some user, and fixed it by running:

mysql_upgrade -u root -p$MROOTPASS
service mysql restart

How to rename multiple files in Linux

The following bash script renames all the files matching src_name.* with dst_name.*:

src_name=$1
dst_name=$2
unset $1
cp $src_name.* ~/temp/
cd ~/temp/
rename "s/$src_name\.([a-z]+)/$dst_name\.\$1/" *
cd -
mv ~/temp/$dst_name.* .

If you have some files like src1.h and src1.cpp, you can save this script into dupsource.sh file, for example, and then rename those files with src2.h and src2.cpp using the following command:

dupsource.sh src1 src2

The script requires ~/temp directory to exist and be writable.

What packages QAudioDecoder may require on Ubuntu and CentOS?

If QAudioDecoder does not decode mp3, reporting a format error (GStreamer; Unable to start decoding process), the following package can help:
On Ubuntu:

apt-get install gstreamer0.10-fluendo-mp3

On CentOS:

yum -y install http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
yum install gstreamer{,1}-plugins-ugly

(more…)

How to convert mp3 to wav in Ubuntu

Install sox tool with mp3 support:

apt-get install sox
apt-get install libsox-fmt-mp3
sox file.mp3 file.wav

Basically that is all, but if you need to check resulting wav file parameters like channels, sample rate, precision, duration, bit rate, etc…, use soxi command:
(more…)

Detecting memory leaks of C++ application in Ubuntu

First, I tried Valgrind tool using the following command:

valgrind --tool=memcheck --leak-check=yes ./app

With some large QT application started for some short period I got the following output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
==7090== HEAP SUMMARY:
==7090==     in use at exit: 5,623,365 bytes in 36,268 blocks
==7090==   total heap usage: 32,454,680 allocs, 32,418,412 frees, 12,822,939,874 bytes allocated
................................
==7090== LEAK SUMMARY:
==7090== definitely lost: 20,163 bytes in 74 blocks
==7090== indirectly lost: 60,053 bytes in 1,273 blocks
==7090== possibly lost: 396,167 bytes in 2,169 blocks
==7090== still reachable: 4,834,822 bytes in 31,576 blocks
==7090== suppressed: 0 bytes in 0 blocks
==7090== Reachable blocks (those to which a pointer was found) are not shown.
==7090== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==7090==
==7090== For counts of detected and suppressed errors, rerun with: -v
==7090== Use --track-origins=yes to see where uninitialised values come from
==7090== ERROR SUMMARY: 20905 errors from 1583 contexts (suppressed: 15 from 2)

When I left this app running for the night (approximately 16 hours), I got the following summary:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
==3816== HEAP SUMMARY:
==3816==     in use at exit: 7,746,701 bytes in 36,248 blocks
==3816==   total heap usage: 827,800,342 allocs, 827,764,094 frees, 105,404,761,516 bytes allocated
..................................
==3816== LEAK SUMMARY:
==3816== definitely lost: 19,835 bytes in 38 blocks
==3816== indirectly lost: 59,805 bytes in 1,237 blocks
==3816== possibly lost: 396,167 bytes in 2,169 blocks
==3816== still reachable: 6,958,734 bytes in 31,628 blocks
==3816== suppressed: 0 bytes in 0 blocks
==3816== Reachable blocks (those to which a pointer was found) are not shown.
==3816== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==3816==
==3816== For counts of detected and suppressed errors, rerun with: -v
==3816== Use --track-origins=yes to see where uninitialised values come from
==3816== ERROR SUMMARY: 13022 errors from 1574 contexts (suppressed: 10 from 2)

It means that the app leaks about 0.126 MB/hour ( (6958734 – 4834822) / 16 / 1024.0 / 1024.0) and totally for the night 2.02 MB, and probably it is not a leak, because the app has pointers to the allocated memory (it is reachable), but it does not clean exit.

To figure out how it works I built the following trivial program:

1
2
3
4
int main()
{
  int *a = new int[100];
}

with debug information and started the tool:

g++ -g -o ex ex.cpp
valgrind --tool=memcheck --leak-check=yes ./ex

In the output the tool shows line number 3 where memory leak is occurred:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
==22042== Memcheck, a memory error detector
==22042== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==22042== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==22042== Command: ./ex
==22042==
==22042==
==22042== HEAP SUMMARY:
==22042==     in use at exit: 73,104 bytes in 2 blocks
==22042==   total heap usage: 2 allocs, 0 frees, 73,104 bytes allocated
==22042==
==22042== 400 bytes in 1 blocks are definitely lost in loss record 1 of 2
==22042==    at 0x4C2B800: operator new[](unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==22042==    by 0x40061B: main (ex.cpp:3)
==22042==
==22042== LEAK SUMMARY:
==22042==    definitely lost: 400 bytes in 1 blocks
==22042==    indirectly lost: 0 bytes in 0 blocks
==22042==      possibly lost: 0 bytes in 0 blocks
==22042==    still reachable: 72,704 bytes in 1 blocks
==22042==         suppressed: 0 bytes in 0 blocks
==22042== Reachable blocks (those to which a pointer was found) are not shown.
==22042== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==22042==
==22042== For counts of detected and suppressed errors, rerun with: -v
==22042== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Links:

  1. How I can detect memory leaks of C++ application in Linux (Ubuntu OS)?
  2. How to check for memory leaks in a large scale c++ Linux application?
  3. Still Reachable Leak detected by Valgrind
  4. With Memcheck’s memory leak detector, what’s the difference between “definitely lost”, “indirectly lost”, “possibly lost”, “still reachable”, and “suppressed”?
  5. Valgrind: can possibly lost be treated as definitely lost?

Another alternative is calling mallinfo() function in C++ code. The following code gets something like total heap size of the process:

1
int usedmem = mallinfo().uordblks;

Using this function I wrote a QT widget that shows memory usage in application status bar:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
#include <QLabel>
#include <malloc.h>
 
class MemoryStatusWidget : public QLabel
{
public:
    MemoryStatusWidget(QWidget *parent = nullptr);
 
public:
    void timerEvent(QTimerEvent *) override;
};
 
MemoryStatusWidget::MemoryStatusWidget(QWidget* parent)
    : QLabel(parent)
 
{
    startTimer(1000);
}
 
void MemoryStatusWidget::timerEvent(QTimerEvent *)
{
    int usedmem = mallinfo().uordblks;
 
    setText(tr("Memory Usage:%1 MB").arg(QString::number((double)usedmem / (1024 * 1024), 'f', 3)));
}

Detecting leaks on Windows

Just in case if you need to detect leaks on Windows (where Valgrind doesn’t work and will never work by design), give Deleaker a try. By the way it can be integrated with Qt Creator, see video on YouTube.

Compiling GDAL on Ubuntu Linux with SQLite and MySQL support

Fist install MySQL client libraries and check their location:

1
2
apt-get install libmysqlclient-dev
find / -name '*libmysqlclient*'

Then install SQLite:

1
sudo apt-get install sqlite3 libsqlite3-dev

Extract GDAL sources and configure supported modules (MySQL and SQLite are not compiled by default, so we need to specify them explicitly):

1
2
3
unzip gdal211.zip
cd gdal-2.1.1/
./configure --with-sqlite3 --with-mysql

this will output “MySQL support: yes” and “SQLite support: yes” along with other information.

(more…)

How to find files by permission on Ubuntu

Today I accidentally changed permissions of all the files in some folder /usr/lib/git-core with the following command:

chmod o-rx *

At first I thought that I have broken my system, but in few minutes I remembered that I have the same distribution on some other machine, I listed the same folder on this machine and used the following commands to restore original permissions:

chmod o+rx *
find -maxdepth 1 -type f -perm 645 -exec chmod o-x {} \;

645 means the files that initially were -rw-r–r— and became -rw-r–r-x. This two commands saved me a lot of time. After that I even count the number of files with all permission combinations:

find -maxdepth 1 -type f -perm 644 | wc -l
find -maxdepth 1 -type f -perm 755 | wc -l
find -maxdepth 1 -type f -perm -o=x | wc -l