Sunday, December 1, 2019

Automating apache bench results plotting

As part of my work with some Akamai resellers in Latin America, I help improve their customers websites performance. This process not only includes doing an optimal CDN implementation, but also sometimes includes doing load and performance testing to take metrics and recommend improvements. 

One of the open source tools we use for load testing is apache bech, a benchmarking tool that has been around for a long time. Apache bench is a simple tool that can help you have an initial idea of things like: 

  • Find out the amount of concurrent users a website can handle before response times start to spike up.
  • How many concurrent connections (users) a website can handle before it stops responding .
  • Help you do server dimensioning for an application.
  • Test cloud infrastructure autoscaling.
Granted, there are other tools than apache bench (propietary and open source) that can emulate a user more realistically, but many of these open source tools alternatives lack an important feature in my opinion that is plotting the test results. These graphs can be an important complement and can show you things that are not that obvious looking at the text test results.

So I wrote a tool called ab_graph.sh that can help plotting apache bech results. This tool is basically a wrapper script that sets up the parameters to save the results for plotting (-g and -e options), and calls gnuplot with those files. 

Here are all the configuration options:


    Usage: $0 OPTIONS

    OPTIONS:
    -c    Concurrent connections  (default: 1)
    -k    Enable keepalive        (defalt: no)
    -E    Extra parameters
    -n    Number of requests      (default: 1)
    -u    Url to test             (mandatory)
    -h    Print help.
    -V    Debug mode.
 

To use av_graph.sh Apache bench and gnuplot installed. Lests do an example. First clone the repository:


    $ git clone https://github.com/juanluisbaptiste/apachebench-graphs
    $ cd apachebecnh-graphs


If you were going to test the url www.testsite.com we could use the following command:

    $ ./ab-graph.sh -u http://www.testsite.com -c 50 -n 1000 -k
    
That command will run an apache bench against www.testsite.com, sending 50 concurrent corrections until 1000 requests are sent. If you need to pass more parameters to apachebench you can use the -E option. Refer to apachebench documentation for more details. Results are stored with the following structure:

    $PWD/results/www.testsite.com/2019-12-01-13-17/


 Inside that directory you can find the following files:

  • values.tsv and percentages.csv: apachebench results files for plotting.
  • values.p and percentages.p: gnuplot files created using a template.
  • values.tsv.png and percentages.csv.png: plotted results.
  • summary.txt:  apachebench summary results printed on the screen at the end of the test.
And these are the plotted results:

Values.tsv:



 percentages.csv:




You can customize the resulting graphs by modifying the gnuplot templates  found in the templates directory. This project is still in early development, so there may be some bugs that you can report at the repository issues page.

Wednesday, November 6, 2019

Improvements on the Unofficial OTRS docker images

Since the last post about the development of my unofficial OTRS docker image, appart from having the image updated to include OTRS's latest version which currently is 6.0.23 released some days ago, there have been some small improvements that I want to talk about.


Switching the article storage type

 

The first one is that you can switch the article storage type (ticket attachments) using a new environment variable called OTRS_ARTICLE_STORAGE_TYPE. This variable can have only two values: ArticleStorageDB (OTRS default) or ArticleStorageFS. When setting OTRS_ARTICLE_STORAGE_TYPE to a value different to the currently configured, the article type will be changed to the new one and articles will be moved to the new location.

For example, if you are switching from ArticleStorageDB to ArticleStorageFS, you would set: 

    OTRS_ARTICLE_STORAGE_TYPE=ArticleStorageFS 

in your env file. When the container starts it will:
  1. Change Ticket::Article::Backend::MIMEBase::ArticleStorage sysConfig setting to the new dtorage type
  2. Run ${OTRS_ROOT}bin/otrs.Console.pl Admin::Article::StorageSwitch to export articles from the databse and put them on ${OTRS_ROOT}/var/article.
If you want to swtich back to ArticleStorageDB you only need to remove OTRS_ARTICLE_STORAGE_TYPE from your env file and restart the container, articles will be reimported into the database. Also, the directory ${OTRS_ROOT}/var/article must be mounted on the host as a volume, if not then the exported articles will be lost upon container restart, you have been warned.


Changing default backup script for cron


When enabling scheduled backups, a script to call the OTRS backup script is added to /etc/cron.d. This script loads up the environment variables set at container startup and calls /otrs_backup.sh (which is simply a wrapper to the backup script that comes included with OTRS, that makes it possible control it with env variables and creates a tarball of the backup directory).

Sometimes you need to do some pre-backup stuff before doing the backup, for example excluding old articles (ticket attachments) on the filesystem when using OTRS_ARTICLE_STORAGE_TYPE=ArticleStorageFS

So to change the script being run by cron you can set the OTRS_CRON_BACKUP_SCRIPT environment variable to point to the new script. This file must be present in the /etc/cron.d directory, you will have to create a custom image that uses this one as base and add the custom script, or mount it as a volume on the host.


Disable email fetching


This is a partial reintroduction of an old feature available on OTRS 4 images were you could change the email fetch time or disable fetching using an environment variable. Now there's a new variable called OTRS_DISABLE_EMAIL_FETCH=(yes|no) to at least disable email fetching (is no longer possible to change fetch time from command line, which was the reason for this feature to be removed after OTRS 5 images were released). This is a useful feature when doing debugging with a local OTRS install or while doing initial software configuration.


Please checkout the README file for a more in depth documentation of these new features.

Tuesday, June 18, 2019

Development Update on the Unofficial OTRS docker images

Lately there have been some work with the unofficial OTRS docker images with the help of some new contributors. I have added some improvements and some new Pull Requests have been merged (thanks for those !). I'm going to review the latest updates done to the images:

OTRS container

Docker secrets support


Merged PR #60 that adds support for docker swarm secrets, allowing to use files to store variable values instead of adding them as environment variables when in a docker swarm cluster.

Email sending configuration is more flexible

Merged PR #64 that makes OTRS SMTP settings completely configurable. Now these settings aren't hardcoded to use this SMTP relay container to send emails (It is still added in the example .env file as an example if you plan to use it). 

Now addtional to the SMTP server, port, username and password values, you can also set the email module to use, using the variable OTRS_SENDMAIL_MODULE=(SMTP, SMPTS, Sendmail) to one of the supported values by OTRS.

The reason why we are not setting a default anymore for this settings is because any setting set on Config.pm becomes read-only on the SysConfig GUI, and some people like using SysConfig instead of modifying configuration files.

So now if you don't manually configure the SMTP settings, email sending will not work until you configure an SMTP server through SysConfig.

Major updates improved

 There also have been some improvements in the major version upgrade process: part of the code was reorganized to avoid some rare upgrade cases I was facing, now addons are upgraded before running the database upgrade script.

Also some new variables were added to improve the control during the upgrade:
  • New environment variable OTRS_UPGRADE_BACKUP=(yes|no) to enable/disable automatic backup before starting a major version upgrade. Default is yes. 
  • Load additional sql files before db upgrade.
  • Added new environment variable OTRS_UPGRADE_XML_FILES=(yes|no) to enable upgrading of XML configuration files during an major version upgrade.

Some more general changes

  • Another new feature is that addons can be installed at container start. Just map  /opt/otrs/addons to a directory in your host, download the addons from whatever repository you like and put the .opm files there. The container will pick them up on boot and install them (and automatically upgraded when a new version of OTRS is released).
  • Also now all pasword values that are printed into stdout are masked by default so they aren't displayed on container boot.
  • Merged PR #57 that adds a new variable MYSQL_ROOT_USER to configure the database root username that was hardcoded to root before.
  • Merged PR #66 that adds an example systemd service file to run the containers using docker-compose at server boot.

Database container

For a long time there was an issue with the database container when starting a new OTRS  service. The container expected the user owner of the database mount point to be the same as the user running the database process, so you had to manually set the correct permissions first on the docker host prior to starting it.

Now that's not needed to be done, the container will work without the need to set up filesystem permissions. We also moved from CentOS/MariaDB image to the official MariaDB image.



That's all for now, check the complete CHANGELOG for the complete list of changes done.

Monday, May 20, 2019

New docker images for upcoming mageia 7

I have added new docker images for the upcoming mageia 7 release. Thanks to the latest work on our image build tools, the images are available in all architectures mageia 7 supports:
  • x86_64
  • armv7hl
  • aarch64
The images are based on mageia 7 beta3, and will be periodically updated when new releases are available.

Next step, automation.

Friday, May 10, 2019

armv7hl support for mageia docker official images

After some months of on and off work with @Conan-Kudo on improving mageia's docker images build tools to support multi-arch builds, we finally were able to add armv7hl support to mageia 6.

Usage is completely transparent to the user, when pulling the image, the docker daemon will take care to download the correct image according to the host server architecture.

Also, now that our build tools support multiarch builds, the moment mageia 7 is available armv8 images will be available too, at the same time of the x86_64 image.

We are also working on having a periodically updated cauldron build, but we are still working on that. With the latest changes to the build tools it should be easier to automate a cauldron build for example, a weekly or daily.

Sunday, March 17, 2019

Some improvements on my docker simple SMTP relay

I have not done any improvements to one of my first docker containers, a SMTP relay, despite the fact that it has resulted quite popular on docker hub (a 1M+ downloads to date !!). Lately I have done some work and received some pull requests with some improvements that I want to talk about.

The first feature I want to introduce is the addition of rsyslog to enable logging the container output to stdout and to log to remote logging systems. This means that now you can see the logs of the emails being processed by the container on its standard output with the docker logs command.

Also, for the modification of postfix's config file, now instead of using sed to modify it we use the postconf command so it is safer. There are also some new configuration options:

SMTP_PORT 

Added by PR #4. Adds a new variable to configure the server SMTP port to use.


SMTP_HEADER_TAG

Added by PR #7. This will add a header for tracking messages upstream. Helpful for spam filters. It will appear as:
RelayTag: yourheadertag

in the email headers.


SMTP_NETWORKS
Added by PR #12 . This variable will set postfix parameter mynetworks, allowoing you to add additional, comma seperated, subnets to use the relay. The default value allows the following networks:

'10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16'

The value you set will append the entered values to that list, so no need to re-add them to the variable.

Many thanks to all the authors for them !!