Maintaining Docker-based Redash

For over a year now, I have provisioned and been maintaining three (3) docker-based Redash instances for different purposes at work. Redash has become a convenient tool for us to manage user access to databases and monitor certain operations.

Here are some of my learnings in keeping Redash alive and optimised on AWS EC2.

Configuration and Logs

To check running docker processes, ssh into the server and run docker ps. This lists all running Redash docker containers. docker logs -f also provides logs on the container you pass to it ie. docker logs -f <container id>.

Restart

Navigate to /opt/redash on the server and run docker-compose restart to restart all containers.

You can also restart single processes using the container id or name. For instance, docker restart redash_server_1 will restart the Redash server.

Restarting Celery Workers & the Queries Queue

In case you are handling a problem, and you need to stop the currently running queries and reset the queue, follow the steps below;

you can flush redis by running docker exec -it redash_redis_1 redis-cli flushall. Restart container with docker restart redash_redis_1.
check redis dump size: docker exec -it redash_redis_1 ls -la.

Changing the Number of Workers

By default, Celery will start a worker per CPU core. Because most of Redash’s tasks are IO bound, the real limit for the number of workers you can use depends on the amount of memory your machine has. It’s recommended to increase the number of workers, to support more concurrent queries.

You can change the number of workers by changing WORKERS_COUNT for the scheduler service in docker-compose.yml located at /opt/redash/. Restart the services after, using docker-compose restart or docker-compose up -d.

DB

Accessing Redash’s DB:

Access Redash DB (Postgres) in docker using: docker exec -it redash_postgres_1 psql -U postgres. You can then perform any activity via the Postgres prompt.

SSL Renewal (Certbot)

I use Let's Encrypt and I renew the certificate using the command below.

sudo docker run -t --rm -v /opt/redash/nginx/certs:/etc/letsencrypt -v /opt/redash/nginx/certs-data:/data/letsencrypt deliverous/certbot renew --webroot --webroot-path=/data/letsencrypt

list certificates to check the renewal has taken effect and see the new expiry date;

sudo docker run -t --rm -v /opt/redash/nginx/certs:/etc/letsencrypt -v /opt/redash/nginx/certs-data:/data/letsencrypt deliverous/certbot certificates

Restart nginx after; go to /opt/redash and sudo docker-compose kill -s HUP nginx.

Redash NGINX Error

ERROR: for nginx Cannot start service nginx: b'driver failed programming external connectivity on endpoint redash_nginx_1 (e8c2089c3a09d7728fc79fb8eb7dde390ee9687d11a51d924d7224a28824ed52): Error starting userland proxy: listen tcp4 0.0.0.0:80: bind: address already in use'

When you encounter the above error, try the below commands to resolve it.


docker stop $(docker ps -a -q); docker rm $(docker ps -a -q); docker volume rm $(docker volume ls -qf dangling=true)

docker network rm $(docker network ls -q)

sudo lsof -nP | grep LISTEN

sudo kill -9 <>

If the above doesn't work, check the nginx service(docker container), stop it and restart with docker-compose up to restart all services/containers.

sudo service nginx status
sudo service nginx stop
sudo docker-compose restart

kobebigs