How to run CKAN tests

CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and europeandataportal.eu/data/en/dataset among many other sites. http://ckan.org/

Open source world is interesting and challenging. Sometimes it’s easy and cheap. Sometimes it’s hard to contribute and costs a lot. Looking into the CKAN I was surprised that it’s used by government portals. That’s why I tried to use it a bit. Here is my short manual that extends an official one how to run functional and unit tests.

Unfortunately, I don’t have enough time to make a pull request (maybe you can do it instead of me:) ), so I just make a blog now.

The main issues with tests run are:

 

  • Documentation doesn’t cover all steps
  • CKAN uses outdated versions of Solr and Node.JS
  • Some bugs in tests which will be described later

 

All these things were found (or even reverse-engineered) in sources and CKAN’s CI results. You can found all needed data in the manual, GitHub (https://github.com/ckan/ckan/blob/master/circle.yml and https://github.com/ckan/ckan/blob/master/.circleci-matrix.yml)  and CI report for any pull request (https://github.com/ckan/ckan/pulls). I use the same versions as CI does.

 

I use Ubuntu 16.04 LTS distro in my environment. I strongly recommend to do it inside some virtual machine or containers. So it won’t break anything on your desktop or laptop.

1. Getting sources

Let’s go! First of all, you need to clone sources:

git clone https://github.com/ckan/ckan.git

 

2. Node.JS installation

For UI integration test you need to install Node.JS v. 0.10.33. It won’t work on the latest version for sure. 

curl -O https://nodejs.org/dist/v0.10.33/node-v0.10.33.tar.g

tar pxzf  node-v0.10.33.tar.gz

cd node-v0.10.33

./configure && make

sudo make install

 

You can install required npm packages now to run tests in the future:

npm install -g mocha-phantomjs@3.5.0 phantomjs@~1.9.1

 

3. PostgreSQL installation

I used that version of PostgreSQL which is available on my Linux distro:

apt install postgresql

apt install postgresql-server-dev-9.5

4. Redis

It should be simple, just run:

apt install redis-server

 

5. Python dependencies.

I use virtualenv wherever it’s possible:

cd ~/ckan

virtualenv .venv && . .venv/bin/activate

Once virtualenv is ready and activated, it’s time to install python packages:

pip install -r requirement-setuptools.txt

pip install -r requirements.txt

pip install -r dev-requirements.txt

python setup.py develop 

 

6. Database configuration

Configure some environment variables to get everything working. I used the same values as we’ve got in the Circle CI configuration:

export CKAN_POSTGRES_DB=ckan_test

export CKAN_POSTGRES_USER=ckan_default

export CKAN_POSTGRES_PWD=pass

export CKAN_DATASTORE_POSTGRES_DB=datastore_test

export CKAN_DATASTORE_POSTGRES_WRITE_USER=ckan_default

export CKAN_DATASTORE_POSTGRES_READ_USER=datastore_default

export CKAN_DATASTORE_POSTGRES_READ_PWD=passexport

Create required databases and grant permissions:

sudo -E -u postgres ./bin/postgres_init/1_create_ckan_db.sh

sudo -E -u postgres ./bin/postgres_init/2_create_ckan_datastore_db.sh

sed -i -e ’s/.datastore.read_url./ckan.datastore.read_url = postgresql:\/\/datastore_default:pass@\/datastore_test/’ test-core.ini

paster datastore -c test-core.ini set-permissions | sudo -u postgres psql

 

7. Solr installation and configuration

To get tests passed I use Solr v. 4.3.1. There is a filed bug about Solr version. CKAN tests don’t work with Solr 6.x now:

curl-O  http://archive.apache.org/dist/lucene/solr/4.3.1/solr-4.3.1.tgz

tar zxvf solr-4.3.1.tgz

Now you have to start Solr. You can run it as a daemon or run it in a separate terminal:

cd solr-4.3.1/example/

java -jar start.jar

 

Solr initialization is required too:

export SOLR_HOME=~/solr-4.3.1

cd ~/ckan

./bin/solr_init/create_core.sh

 

8. Initialize test data

paster db init -c test-core.ini

 

9. Run test CKAN server

paster serve test-core.ini

10. Finally, run UI tests

mocha-phantomjs http://localhost:5000/base/test/index.html

11. Run unit and functional tests

To run all tests you need to execute the following command:

nosetests –ckan –reset-db –with-pylons=test-core.ini –nologcapture ckan ckanext

Unfortunately, you’ll have some failed bugs due to the https://github.com/ckan/ckan/issues/3675 :(. To successfully run all tests, you should use segments. E.g.:

nosetests –ckan –reset-db –with-pylons=test-core.ini –nologcapture –segments=abc ckan ckanext

 

Tags

.net .net-framework .net-framework-3.5 agile ajax ajax-control-toolkit ampq ansible apache asp.net asp.net-mvc axum bash benchmark blog blog-engine bootstrap buildout c# cache centos chrome ci cinder ckan cli cloud codeplex community debugger deface development-environment devstack devtime django dlr dns docker dockerimage dos easy_install elmah encoding environment-variables error event events exception exceptions fabrik firefox foreach forms gae gcc git github go google google-app-engine hack hacked hardware headless horizon html hugo iaas ienumerable iis internet iptables iron-python ironic iscsi java-script javascript jenkins jquery js kharkivpy kombu kvm kyiv lettuce libvirt linux lio loci logging mac-os mercurial microsoft microsoft-sync-framework mobile mono ms-office msbuild networking news nginx offtopic open-xml opensource openstack openvswitch os packages paraller-development patterns--practices php pika pip pnp podcast popup postgresql profiler project protocols pycamp pycharm pycon pykyiv pylint pypi python python-3 qcow quantum qumy rabbitmq rar react refactoring rfc rhel search-engine security selenium server shell silverlight socket source-control sourcegear-vault sql sql-server sql-server-express sqlalchemy ssh svg tests tgt tipfy tornado typescript uapycon ui uneta unit-tests usability virtualenv visual-studio vm vue.js vuejs web-development web-server web-service webpack windows windows-live word-press x32 x64 xcode xml xss xvfb интернет-магазин книги

Recent posts

Docker Image Collection

Not a DNS Server for Libvirt

Diskimage-builder in Docker

Everything as a Code

Vue.js, global variables and TypeScript


Archives

2019 (19)
2018 (2)
2017 (3)
2016 (2)
2015 (3)
2014 (5)
2013 (17)
2012 (22)
2011 (35)
2010 (25)
2009 (35)
2008 (32)
2007 (2)