How to run CKAN tests

CKAN is an open-source DMS (data management system) for powering data hubs and data portals. CKAN makes it easy to publish, share and use data. It powers datahub.io, catalog.data.gov and europeandataportal.eu/data/en/dataset among many other sites. http://ckan.org/

Open source world is interesting and challenging. Sometimes it’s easy and cheap. Sometimes it’s hard to contribute and costs a lot. Looking into the CKAN I was surprised that it’s used by government portals. That’s why I tried to use it a bit. Here is my short manual that extends an official one how to run functional and unit tests.

Unfortunately, I don’t have enough time to make a pull request (maybe you can do it instead of me:) ), so I just make a blog now.

The main issues with tests run are:

 

  • Documentation doesn’t cover all steps
  • CKAN uses outdated versions of Solr and Node.JS
  • Some bugs in tests which will be described later

 

All these things were found (or even reverse-engineered) in sources and CKAN’s CI results. You can found all needed data in the manual, GitHub (https://github.com/ckan/ckan/blob/master/circle.yml and https://github.com/ckan/ckan/blob/master/.circleci-matrix.yml)  and CI report for any pull request (https://github.com/ckan/ckan/pulls). I use the same versions as CI does.

 

I use Ubuntu 16.04 LTS distro in my environment. I strongly recommend to do it inside some virtual machine or containers. So it won’t break anything on your desktop or laptop.

1. Getting sources

Let’s go! First of all, you need to clone sources:

git clone https://github.com/ckan/ckan.git

 

2. Node.JS installation

For UI integration test you need to install Node.JS v. 0.10.33. It won’t work on the latest version for sure. 

curl -O https://nodejs.org/dist/v0.10.33/node-v0.10.33.tar.g

tar pxzf  node-v0.10.33.tar.gz

cd node-v0.10.33

./configure && make

sudo make install

 

You can install required npm packages now to run tests in the future:

npm install -g mocha-phantomjs@3.5.0 phantomjs@~1.9.1

 

3. PostgreSQL installation

I used that version of PostgreSQL which is available on my Linux distro:

apt install postgresql

apt install postgresql-server-dev-9.5

4. Redis

It should be simple, just run:

apt install redis-server

 

5. Python dependencies.

I use virtualenv wherever it’s possible:

cd ~/ckan

virtualenv .venv && . .venv/bin/activate

Once virtualenv is ready and activated, it’s time to install python packages:

pip install -r requirement-setuptools.txt

pip install -r requirements.txt

pip install -r dev-requirements.txt

python setup.py develop 

 

6. Database configuration

Configure some environment variables to get everything working. I used the same values as we’ve got in the Circle CI configuration:

export CKAN_POSTGRES_DB=ckan_test

export CKAN_POSTGRES_USER=ckan_default

export CKAN_POSTGRES_PWD=pass

export CKAN_DATASTORE_POSTGRES_DB=datastore_test

export CKAN_DATASTORE_POSTGRES_WRITE_USER=ckan_default

export CKAN_DATASTORE_POSTGRES_READ_USER=datastore_default

export CKAN_DATASTORE_POSTGRES_READ_PWD=passexport

Create required databases and grant permissions:

sudo -E -u postgres ./bin/postgres_init/1_create_ckan_db.sh

sudo -E -u postgres ./bin/postgres_init/2_create_ckan_datastore_db.sh

sed -i -e ’s/.datastore.read_url./ckan.datastore.read_url = postgresql:\/\/datastore_default:pass@\/datastore_test/’ test-core.ini

paster datastore -c test-core.ini set-permissions | sudo -u postgres psql

 

7. Solr installation and configuration

To get tests passed I use Solr v. 4.3.1. There is a filed bug about Solr version. CKAN tests don’t work with Solr 6.x now:

curl-O  http://archive.apache.org/dist/lucene/solr/4.3.1/solr-4.3.1.tgz

tar zxvf solr-4.3.1.tgz

Now you have to start Solr. You can run it as a daemon or run it in a separate terminal:

cd solr-4.3.1/example/

java -jar start.jar

 

Solr initialization is required too:

export SOLR_HOME=~/solr-4.3.1

cd ~/ckan

./bin/solr_init/create_core.sh

 

8. Initialize test data

paster db init -c test-core.ini

 

9. Run test CKAN server

paster serve test-core.ini

10. Finally, run UI tests

mocha-phantomjs http://localhost:5000/base/test/index.html

11. Run unit and functional tests

To run all tests you need to execute the following command:

nosetests –ckan –reset-db –with-pylons=test-core.ini –nologcapture ckan ckanext

Unfortunately, you’ll have some failed bugs due to the https://github.com/ckan/ckan/issues/3675 :(. To successfully run all tests, you should use segments. E.g.:

nosetests –ckan –reset-db –with-pylons=test-core.ini –nologcapture –segments=abc ckan ckanext

 

Tags

.net .net-framework .net-framework-3.5 agile ajax ajax-control-toolkit ampq ansible apache asp.net asp.net-mvc automation axum babel bash benchmark blog blog-engine bootstrap buildout c# cache centos chrome ci cinder ckan cli cloud code-review codeplex community config debugger deface dependencies development-environment devices devstack devtime disks django dlr dns docker dockerimage dos easy_install elmah encoding environment-variables error event events everything-as-a-code exception exceptions fabrik firefox flask foreach forms fstab gae gcc gerrit git github go google google-app-engine grep hack hacked hardware headless horizon hound html hugo iaas ienumerable iis internet iptables iron-python ironic iscsi java-script javascript jenkins jquery js jsx kharkivpy kiss kombu kvm kyiv lettuce libvirt linux lio loci logging loopback losetup lvm mac-os macos mercurial microsoft microsoft-sync-framework mobile mono ms-office msbuild networking news nginx npm npx offtopic oop open-source open-xml opensource openstack openvswitch os packages paraller-development patterns-practices performance php pika pip plugins pnp podcast popup postgresql profiler project protocols proxy pycamp pycharm pycon pykyiv pylint pypi python python-3 qcow quantum qumy rabbitmq rar react reactjs refactoring rfc rhel search-engine security selenium server shell silverlight socket software-engineering source-control sourcegear-vault sources sql sql-server sql-server-express sqlalchemy ssh static-site sublimetext svg tests tgt tipfy tornado typescript uapycon ui uneta unit-tests upgrades usability vim virtualenv visual-studio vitrage vm vue.js vuejs web-development web-server web-service web_root webpack webroot windows windows-live word-press x32 x64 xcode xml xss xvfb интернет-магазин книги


Archives

2019 (73)
2018 (2)
2017 (3)
2016 (2)
2015 (3)
2014 (5)
2013 (17)
2012 (22)
2011 (35)
2010 (25)
2009 (35)
2008 (32)
2007 (2)