RESTful APIs | Provisioning data

Topical content analysis using word counts / tag clouds is a feature in my Head-Up Data project. The tag clouds are presented in html pages using visualisation tools. The same data may need to be presented on a mobile device or that data may be requested by other web services. In such cases only the data is requested. How the data is presented by a third party is up to them. The web services must provision its data in formats that clients can understand and use.

Here we look at a set of REST apis that provision page word-count data. Normally a user would navigate to the app url in a browser, look at the list of web pages, select a web page, examine the list of word counts/clouds and finally select one tag cloud. The REST Apis are also designed likewise using Django rest framework.

First we take a quick look at the user interface for the steps mentioned. Then we look at the REST APIs to achieve the same.

Web UI presenting a list of pages.

Web interface presenting page details and selecting a cloud from its list

The REST APIs and their browse-able screens are given below. 

Having a browse-able API helps people to learn the APIs faster. Developers can copy the url endpoints to a browser and see the details for themselves.

When a mobile app or another service accesses these APIs, only the json responses are sent back.

1) The data format used is Json.

2) APIs include

a) An API index to start with.

b) API to get the list of available Web pages.

c) API to get details of a particular web page.

d) API to get a list of all Word counts.

e) API to get a specific word count data.

3) Pagination: Each api that presents a list of data points will show only a fixed number of data points per page. The responses will have next and previous links for navigation. This is a better option that sending all the data points with each response.

4) Linked relations: A related object is always presented as a link. For example, the 'GET web page details' API has links to its word clouds. Not the actual word count/cloud data. If the requesting service wants to get that word count data, it can simply go to the link. On top of saving response data length, this approach also allows the data to be cached on the client side, say on a mobile device. The url can be used as a key.

API index

API for Web Pages list. The json response also has next and previous links to navigate the list.

API for a single web page. This has links to the page's word clouds.

API for a specific word count.

Connection pooling | Database performance | pgBouncer | DB Server side Vs AppServer Side

In this iteration a database connection pooling solution is implemented. This enables the application to reuse database connections thus increasing performance at the database end. The pooling software used is pgBouncer.

> If the reader wants to compare the load test reports on pgBouncer on DB hosts Vs pgBouncer on Web app server scroll down to the end of this post.

The content of this post is discussed in the following video for those who prefer that to reading.

The changes at the db side is shown in application architecture below. Instead of connecting directly to the database applications connect to pgBouncer.

1) Setting up a connection consumes server resources and time. It requires a round trip to the database, authenticating a user and checking user privileges among others. Then the query is executed. Some client side frameworks hold on to a connection for a period of time and then release it. i.e The connection is closed after use. This means that a new request with the same credentials even from the same client has to go through the entire process again. This is can be avoided by returning the used connection to a pool and requesting new connections from a pool of active connections.

2) Each database has a configurable maximum concurrent connections limit. For Postgresql this is specified using max_connections in settings /var/lib/pgsql/data/postgresql.conf and has a default value of 100. Beyond this number connections are refused.

3) The load on the database can increase when a) when the user base increases resulting in increased activity b) new application features c) the application scales using load balanced web application servers to meet user load. 

Whatever the reason, an application that used to see a maximum of  60 concurrent connections to the database ends up getting 120. Some of those extra connections end up refused at the database end. Pooling solutions also have a maximum client connection limit. The objective to configure and use a set number of database connections efficiently. Say we have configured the database to allow 150 concurrent connections. The server may have one or more databases for different users. For example, database A and B for userA and userB respectively. Also assume Database A sees more hits/usage than database B. Of total connection allowed by the database say 10 connections may be needed for backend database admininstration and rest are free for application servers. On pgBouncer we can set the maximum allowed connections to say 280 (greater than the database maximum) and configure pgBouncer pools for each user + database pair. In this case we can do userA + database A pool with pool size of 90 and userB + database B with pool size 50. Thus regulate connections.

Adding a pooling solution is an overhead. An immediate increase in throughput is not evident until the application goes through a situation when there is contention for database connections. In that situation instead of getting rejected by the database and failing there, the request is fulfilled by the next available connection in pool. The application is configured to minimise database accesses using caching. So throughput may not be a direct indicator of performance gains. A contention can be simulated only by reducing the cache timeouts which is not ideal. Finally, if the maximum number of connections at pgBouncer is exceeded they are still refused.

The question of putting pgBouncer on the application server hosts or on the database: 

This depends on what works out in the end and is needed. For a small application with multiple databases we can host pgBouncer on the application server and tangibly save the round trip time to the db hosts. If there are many application servers we may be configuring large number of pooled connections at the source end and still end up with a deluge at the database end. This is like having the club bouncer stopping you at your door step because a headcount is reached at the club Vs you going to the club and getting stopped there.

Aggregate reports on load test with two runs are shown below. One with pgBouncer on DB hosts and second with pgBouncer on web server hosts. Both tests take 3-5 minutes to finish. 

Notice that with pgBouncer at web server hosts throughput increased by ~ 21.4% and errors dropped to near 0%. This is due to time saved with not having to setup connections to a remote host each time it is needed and having only a localhost connection. However the webserver hosts pgBouncer pools and has 100s of pooled connections ready to be used. As already mentioned multiple web servers with pools like this can cause deluge the database host.

Load test results with pgBouncer on DB hosts

 

Load test results with pgBouncer on web server hosts

Throughput | Reaching 250 requests per second

In this post we look at how the throughput of my data visualization web application was improved to 267 requests per second in this iteration. This is a significant improvement with additional caching and application architecture modifications. Note that this new throughput is for the web application with a lot more functionalities like language support, machine learning models etc.

New throughput on load test: 267 request per second

Background

Previous JMeter test results with load balanced deployment and model caching is here. The same load test is used again to see the improvement.


The load balanced architecture is described here.

JMeter test results with new throughput
 
JMeter Response time Graph is shown below.

JMeter Graph results is shown below.

Techniques so far and modifications applied

The previous caching technique focused on avoiding database hits. 

That employed a Django model based custom caching library. In addition to that, it also marked static files like js and images with down stream cache-control so that the browser does not download them each time. 

That far that is good. However

1) With the application there are a couple of scopes from improvement especially since a set of new features have been added among which language support is prominent. The Django application builds HTML templates to provision to clients. This includes all the html templates like navigation bar, user profile templates and page footer templates. Some template contents have to change based on say time, user, user language etc. However most templates once they have been generated based on one or more of the above can remain same and be reused. That's where template fragment caching comes in. A few things that can trip if not understood are

• The gain from a single template fragment being cached is tiny. The return on investing in caching templates will only show as the number of concurrent requests on the application go up.
• Also, locality of cache matters for template fragments. The savings to be made on time is small on each request. So even having to go to a cache on a different host will cost more than just building the template!. This mandates a local cache and is a modification to the architecture.

2) Each view generates a response based on the request. For most requests the response is the same. For example, the response to a request for getting 'Word counts for www.cnn.com at 3 PM on 25 Dec 2018' is going to be the same. Such views need to be identified and cached. This helps with improving throughput.

3) Finally, one physical aspect of your deployment that can affect performance is thermal throttling of CPUs. It is a good idea to check this too.

Lang support Dutch, Swedish... for WebApp | HUD

Supporting multiple languages for a Django web application is straight forward. 1) Add the LocaleMiddleWare to the list of middlewares. 2) Supply languages files with translations in PO files. 3) The PO files are compiled into MO files. When a request is encountered django checks into the following places for the required lannguage; the url for a prefix, the session, a language cookie and Accept-Language header in that order. This order can be seen in the locale middleware code at django > middleware > locale. Once the required language is identified, it is activated.

Within the application a user can have a preferred language. This is implemented the same way as preferred timezone. Users can go into their preferences page and choose from a list of supported languages. Currently English, Dutch, Swedish, Norwegian and French languages are supported. Two users with Dutch and Swedish language preferences are shown below.

When a user accesses the application for the first time the login page is shown in a language as on the Accept-Language header from the browser. Here Chrome was set to French (browser settings shown in the end) and Opera in default language settings. Login based on request headers (for users above) is shown below.

The following screens show the two users accessing the application after logging in. For both the browsers the first language was what the browser mentioned in the request header. Once the user logs in the user's preferred language is activated. This is shown below where the preferred language of user on left is Dutch and Swedish on the right.

Chrome language settings indicating French selection is as follows.

References

Django docs on translation

https://docs.djangoproject.com/en/2.1/topics/i18n/translation/

*Updates with Runtimes* Web Search Engine with Word Counts

Here we look at implementation of a web search engine. The project already has the data on word counts for web pages. Pages added to the project have been crawled and content word counts are stored periodically. This was primarily for generating word clouds and text content analysis. However the word counts can also be used build a search index for the set of web pages. Given a bunch of words the search index can give back the list of pages within which the words occur. In addition to that, the word counts are attached with the timestamp at which the page was processed. This helps to find more recent occurrences.

Quick overview of steps involved: 

A) Filter word counts within a time period. Past 4 (or N) days.

B) Build a trie data structure with the data. 

C) Compress the trie so that it can be held in memory. 

D) For a given search string made up of multiple words, find the set of web pages where the words occur. The compressed trie helps with this. Time complexity is described below. 

E) Find the intersection of the sets of web pages. 

F) Extract required information and send back results. This information includes as in other search engines the full url of the page, time of crawling (word count generation) and a title. 

G) Cache information as necessary to speed up the web view.

Runtimes with cProfile are as follows:

1) Building the trie takes 3.583 seconds for 173693 Words. Pickled Size is 119.2MB

2) Compressing the trie takes 2.38 seconds and Pickles size is 6.2MB

3) Searching including fetching resulting web pages ~4ms

4) Searching all 10 strings above including fetching results ~116ms

 

Some screenshots of the engine at work are shown

Apart from the trie index the rest of the data is already part of the project database. However the trie is not part of the database. It is generated when required, compressed and held in memory.

Quick overview of tries: At the core of the index is a data structure called Trie (compressed). A trie is an m-ary tree where each node branches out based on the character encountered in a key. The interesting thing about tries is this. For a set of K unique characters a node has K+1 pointers. Based on the keys that are inserted into a trie the number of nodes can change. For a given trie, if S is node count, key count is N and L is the length of longest key then the search for any key is in O(L) independent of K and N. Storage requirement is (K+1) x S x P bits independent of N, the number of keys in the trie. P is number of bits in a pointer.

Compressing the trie: Once the trie has been constructed it can be compressed. Multiple techniques such as Patricia Tries and de la Briandais trees can be used. However, here the project uses a different technique. Any trie with N nodes and a K character set can be represented by an M x K table. The table can be shrinked further using a sparse matrix. Here we see the difference in serialised size of the trie index.

For 79 web pages in the project, with in a week there can be minimum 2 crawls so ~ 160 word count data rows for the web pages. Sizes for objects were also monitored using pympler trackers for Python 3.

Uncompressed Trie size  25628388 bytes ~ 25MB

Compressed Trie size 21586721bytes ~ 21 MB

Compressed Trie with minimum selected data in leaf nodes 7281173 bytes ~ 7.2 MB

This 7.2 MB trie index can be held in memory or cached. 

The search results are in decreasing order of timestamps. 

The architecture of the crawler project was discussed previously here.  Crawling and word counts are executed in celery async tasks. This architecture is shown below.

Future work: 

1) Currently a set intersection of the words is used. More options like OR and NOT can be supported using expression trees. 

2) Storing the pages themselves in the filesystem for reference would be great. But this is not feasible at the present disk allowance.

3) Since the word counts are time stamped,  a date time search window option can be given to users. Holding the index over an increased period of time raises the size of the index too.

4) It would be feasible to rank the pages on a complex parameter than just time stamps. Relevant visits and count can be used along with timestamps.

5) Word edit distance can be used to correct words as in popular search engines.

HUD | UI updates to web pages and earth quake visualisations

Django custom caching library v2

In a previous post we looked at a very early version of a caching library used in my Django project. This has been enhanced to include new features as requirements came up. Although this library is based on practical requirements that showed up, the two primary api are documented well. This is so that the user is aware of what the library can handle well and avoid performance degradation. Coding up this library has been primarily to help with keeping caching code DRY. Compared to the previous version there are no changes at the models. There are three additions.

i) Prefetched relation support

Django documentation on Prefetch is available here.

In Django it is a common practice to prefetch related relations while querying a model. While this is a good idea, this can really degrade performance by increasing the number of sql queries by O(N) where N is the number of prefetched rows. To address prefetching, both apis will accept a tuple of Prefetch objects. Not the prefetch related names. The reason is as follows. Prefetch objects allow more control on what is prefetched. This helps with performance especially using the .only(*fields) api from queryset as shown below.

In the code we want to get a web page and prefetch its related page word counts. We control what columns are needed from the prefetched relation, PageWordCount, using a queryset. Then we pass the Prefetch to the api. This is important for caching as too much prefetched data will result in memory consumption at database and web server but also cause Django to silently fail when the data is set to memcached. Memcached has a configurable 1MB object size limit. Notice the foreign key reference to web page in the only fields.  

In order to understand the loop hole which will cause sql to be fired, we need to understand how Django handles prefetch. On the primary relation Django brings in the web pages and uses an IN SQL query to bring in the PageWordCounts. Now it does the join in Python i.e it tries to find the PageWordCounts that belong to each WebPage. For that you need the foreign key field. If you did not mention it in the only(*fields) Django will send out an sql query for exactly that, for each prefetched row. 

Prefetch support in the other api is shown below. Here we are pre-loading the cache with a list of all WebPages. This is a better example of where forgetting the above point will cost a lot.

The api signatures are shown below. First one allows fetching rows based on fields. Cache entry is set based on the specified fields. The second fetches all rows.

ii) select_related

Django doc on this is here.

This is a simple forwarding of required fields. Similar to prefetch but for one-to-one and foreign keys relations.

iii) Chunked bulk updates to memcached

Once all the rows are fetched using all_ins_from_cache api, we will have a list of instances. This list can be huge. The api loops through the list and sets the individual cache entries using set_many. However, set_many was silently failing with 100-120 entries. Possibly due to large amount of data being passed over a single call. To avoid this, the instances list is broken into manageable chunks and each chunk is passed to set_many. Chunk size can be configured.

The resulting library is more usable in the Django project data set. Cache set/get code is more sophisticated and helps to keep code DRY.

HUD | Enable/Disable Django Apps

This post describes how Django apps in a project can be enabled/disabled via settings. The requirement is to be able to enable/disable Djanga apps with flags. If the app is enabled in the project then it will be loaded, its urls and templates will be available to users. On the other hand if an app is disabled then its templates and urls are not available to the user.

For example in the screenshot below the word cloud app is enabled on the left. The app is available on the navbar and homepage . The deployment to the right does not have the word cloud app enabled. Notice that the templates have adjusted themselves based on configuration.

To prevent Django from loading an app is easy. Just do not add it to the list of INSTALLED_APPS in Django settings. However the root url conf, ROOT_URLCONF will need to be changed accordingly. Again, url references in templates cannot be enabled/disabled using just the INSTALLED_APPS. Editing templates and root url confs to tailor them just for a specific deployment is not recommended. This creates additional effort as that particular deployment will need to be tracked and maintained separately.

A better implementation is to specify whether an app is enabled/not and the project will load appropriately. For each app we need its

a) Switch name: This tells the rest of the project whether the app is enabled or disabled. i.e a flag like wordcloud_enabled to check against. This is particularly useful for templates. 

b) Url regex: This is the base url pattern for the app. For example all urls for articles app will have the "articles/" base pattern as prefix. And the url for posting articles can be https://www.server.com/article/post

c) Urls module: The Python module that holds the app's url patterns. In the above example the url patterns for posting, editing and deleting articles etc are specified in this module.

These 3 details are easily configurable in named tuples as shown.

1) Controlling apps in templates: 

Users should not be able to view or use urls of a disabled app. In order to achieve this templates should know whether an app is disabled. This is made possible using a list of Application tuples and a template context processor. The switches tell the template whether an app is enabled or not. The context that contains the app's state is generated from the Application tuples and made available by a context processor. 

In this solution, so long as the templates are as modular as possible, i.e utilising template hierarchy to separate template fragments, we can enable/disable parts of the user interface. The solution becomes as simple as the following check in a template for the navbar for the ml machine learning application.

Notice that the template uses the flag 'ml_enabled' to check ml app is enabled. Each application needs a flag that describes its state. This flag/label is also configurable.

2) Now comes the root url confs. 

For each enabled application we need the url regex and the urls module for the application. These are added to urlpatterns in root url conf. This is shown below.

Deploying application updates | Quick automation using shell scripts

This post describes how the visualisation and machine learning application is currently updated to servers. See the application demo on YouTube using the playlist.

There are 2 Ubuntu server host machines in the load balanced pool. Python 3.5.3 and Virtualenv are compiled and are available on the machines. System Python is not used. The web application updates are made available on a particular branch of the Bitbucket git repository. The repository is configured with read-only keys for release use. This key is different from the developer keys. On the Ubuntu hosts the read-only keys are configured such that the keys get added to ssh-agent on agent startup. Only the branch with tested final release is cloned. 

There is a root folder within which each release is cloned to a specific date-timed folder. So there will be multiple folders each representing a particular update or release. A symbolic link 'current' points to the latest release running in production. The modwsgi-express server config points to this symbolic link. This avoids having to modify the modwsgi_express configuration on every application feature update. Finally this makes rollback as easy as pointing back to the previous release folder.

All of this is done using a simple shell script. The root folder, branch to clone, number of processes, thread counts etc are configurable. Figure shows the file with list of server ips and update scripts.

This shell script logs into the deployment servers using ssh (another key for admins). 

Then it performs the tasks mentioned above. This deployment also ensures that each release runs on its own python virtual environment. After the cloning, the virtualenv's  requirements.txt is available and is used to create a new environment in the release folder. If needed each of the date-timed folder mentioned above can be put back in production. The script accesses the servers listed in a configuration file. Each machine is updated sequentially. The figure shows the script logging into the first server in list.

The ssh-agent is also used only for the duration of the update. The agent is started at the beginning of the script and stopped at the end. Finally, the modwsgi_express server is restarted in this script. This is not necessary and will be removed in the next iteration. The figure shows the script finishing up on the first server and moving onto the next server in list. 

Notes:

This script based approach configures everything from the virtualenv upwards. However the infrastructure can also be configured and kept in code using tools like Puppet, chef and the like. Here the focus is on the application side and is kept simple using shell scripting.

Machine Learning | Cancer prediction

In a previous post a number of machine learning models were trained on a dataset. Details of the data set are here. The relative importance of each of the 32 dimensions was also deduced from a multi-dimensional visualisation. This post builds on the results of that to finally use the models for prognosis.

video for post content

The models that were trained included a Gradient boosting classifier, Random forest, an SVC and a K-neighbour classifier. In addition to those models, another Gradient boosting classifier was trained with parameter tuning. For the algorithm a range of values for learning rate, max_features and max depth were specified in a parameter grid. A model selection and evaluation tool like GridSearchCV and a scoring parameter is used to choose a model with best quality.

Using the generated models:

After this model training and generation, we need to use the models. Also when there are a number of algorithms, samples and models, it would be great to select a model and apply an out of bag data/sample through it. 

To do that with this cancer data set, we serialise the models, samples and provision them over an application. The models and their details are stored in the database with a link to their respective files. The same is done for samples. Since each model utilises a particular algorithm, a list of algorithms is also maintained. So algorithms, models and samples can be added to the application. From there a user interface is presented to choose a sample and the model to apply.

This enables users in, say a medical institution, to add new models/samples as they are generated and conduct a prognosis. Models can also be revisited and looked up as they are updated.

Screens of conducting a prognosis on stored models and samples are shown below. 

1) Selecting a sample

2) Selecting a model

3) Result

Another prognosis with K-neighbour classifier on a different sample.

Details of a particular model can also be viewed.

Porting/Upgrading a Python 2 library to Python 3

In this post we look at porting the coils library from Python 2.7 to Python 3.5.3 and having it on both versions.  Coils is a data structure library that was coded in 2015. It has implementations of basic data structures like hash tables, binary search trees and splay trees, list and heaps. Coils code is available here. The steps described here are based on official documentation on porting here. It would be necessary to go through it before attempting to port your project.

The ported code can be installed from https://pypi.org/project/pycoils/

Lets see the steps for porting coils library:

1) Create 2 virtual environments, one for Python 2.7.15 and Python 3.5.3

2) pip install future on both environments.

3) Ensure that all tests pass in the Python 2.7.15 environment to begin porting. 

You need tests and good coverage in your project. One way to be sure that the code would work as expected after porting to Python 3 and also on Python 2 is by ensuring that all tests pass on both versions. Here we see a screen of 255 tests for the project on both.

coils also had good test coverage as shown below.

If you do not have any tests on your project, there is no way to identify regressions introduced by porting. This is because porting involves not only using tools to refactor but also, manual changes to pass tests. 

If your coverage is low, write or modify tests to increase it.

4) Run futurize --stage1 -w **/*.py

(When using globstar ** make sure that it is enabled)

This will modernize the code without introducing any dependencies. You will see a lot of output regarding the code that was refactored. This is shown below.

Once the run is complete without errors, you can use a diff tool to see what code was changed to support python 3 in stage 1. In this example we see python 3 style in operator brought in to replace has_key.

5) Commit the changes. Rerun tests on Python 2 environment and manually fix any errors to ensure tests pass.

6) Run  futurize --stage2 -w **/*.py

The aim is to ensure that code works on both Python 3 and then back on Python 2 with future. As before the command will output the refactoring done. 

The changes can be checked with a diff tool too. Here we see the support for the division method on Python 3 and Python 2 with future package.

7) Run the tests again on Python 3. Note that test cases may also have been ported to Python 3 and will need changes to run on both Python 3 and Python 2. 

Here we see some manual fixes to the test cases. This was to a) Drop deprecated key-word arguments and b) Drop deprecated TestCase class method aliases.

8) Once the tests pass on both Python 2 and Python 3, then you have a stable version that can be used in both environments. 

Coils library has an examples package which demonstrates how the library's data structures can be used in code. For example, creating a min heap, adding data and performing heap operations. 

This is also used to ensure that things work as expected on both.

There are some post conversion steps (depends on your code). After that as tests and examples run on both versions, we have Coils for Python3/Python2.