IQ sometimes takes a few hundred milliseconds to complete small queries, occasionally even a few seconds.

The numerous small queries can blur the real workload, and the deviation of their execution time can induce a feeling of slow performance. 

This caching technique can be used for any type of query, not just small ones, just make sure you understand the synchronization concepts, and ensure they are compatible with your requirements. The caching techniques described below are based on the proven and widely used HTTP caching techniques (even in this very web page). Some of the advantages are:

• Faster UI response time
• Less database workload
• Less network workload with client/proxy caching
• If database is down, some intermediate proxy cache can return stalled results instead of errors

Webservice for SAP IQ

It's possible to connect to IQ (and SQL Anywhere) over HTTP/HTTPS for many years, application architects don't always know this.

The database includes a lightweight webserver which allows to connect with the HTTP protocol and fetch results as XML or other formats.

Simply specify a port by adding the line  -xs "http{port=XXXX}" to the config file and restart IQ.

Webservices concepts are explained in further details in the documentation.

The HTTP client used here is a web browser for simplicity which has caching activated by default, in your environment, you need to make sure the client application connected to IQ has client HTTP caching activated, or connects to IQ through a proxy/CDN with caching capabilities.

Also, if you have a multiplex, it makes a lot of sense to get your results over webservices because HTTP load balancing is a lot simpler than TCP load balacing.

Security concepts

In the following samples, a user webadm is created. The webservices are configured public, and don't require password authentication.

Of course you would need to change this before going live, just like you would need to setup https. I might write a follow up document explaining how.

Install the sample webservice

The attached file install.sample1.txt creates a portfolio table, inserts random data, and defines a web service to fetch the details of a portfolio.

After executing it against a running IQ server, you should be able to connect using a web browser :

Firefox, chrome, and IE have a developer mode to debug HTTP protocol, open it by using the appropriate shortcut:

• Ctrl + Shift + Q on firefox
• Ctrl + **** + I on chrome
• Alt + F4  on IE :-)

This article explains how to view if the data is loaded from the network or from the cache

As you can see, by default the result sets includes a tag called "Expires" which is equivalent to the current date, thus forbidding any caching.

Simple result caching

In the second sample file, the webservice getptf2 returns the same results but the HTTP response header includes a caching information :

The property cache-control has a value of max-age=300which means the result can be cached for 300 seconds (or 5 minutes). Additional properties let you control the cache behavior in intermediate servers.

--5 minutes cache retention

call sa_set_http_header('cache-control','max-age=300') ;

If you query the webservice several time, you should the cache hits. In firefox, open this page to view cached resources and their associated cache hits.

about:cache?storage=disk&context=

Cache entry information

You can notice here that the fetch count is 8, meaning the result was fetched from the cache and not from IQ. Also, the cache entry expires 5 minutes after the last fetch, which is consistent with the 300 seconds max-age value we set for the cache control property. Also, the cache entry includes the parameter id, so if I request portfolio "PTF-7", it won't used the cached results for "PTF-6" 

The last section "proxy setup for automatic caching" explains how to automatically insert cache control header based on webservices name.

This is simple, robust and can improve many situations I've encountered where data doesn't change often. A variant could be to manually set a cache expiration date if you know the data is only updated at a fixed time.

In case you need to avoid the risk of using old results, then this other technique is for you.

Results version validation

With this technique, every call to the webservice hits IQ. If the client has a cache entry, it includes the version of this result in the call. On the IQ side, we look at the version the client has, if it is the most recent one, IQ sends a special status to confirm that the client can use the result in cache.

In the example sample3.txt, each portfolio entry has a valuation date, which represents when the data was last change. The value of this column is therefore used as a version marker. In addition to embedding the tag, the expiration is disabled, meaning the result will stay in the cache until the cache is cleared or the database updated.

call sa_set_http_header('ETag',@last_dt) ;

call sa_set_http_header('Expires',null) ;

If you call the webservice twice, you should get a similar view :

The request includes a new tag, "If-None-Match", and its value is the date of the last valuation. Also, the status code is different, status "304 not modified" tells the client to go ahead and use the cached version.

Also, you should see the messages in the server log :

I. 09/03 18:08:19. OK Client cache version for PTF PTF-6

Unlike the simple caching solution, this technique doesn't eliminate queries.

However, if you can validate the version of a result with a simple query, then you could improve performance by replacing a complex calculation by this simple version lookup.

HTTP caching software

While IQ has a webserver, and has the feature to change HTTP response headers, specific software for websites can do a much better job at it.

They can also provide additional features :

• protection against DDos, slowloris
• rewrite URLs
• return results even if database is down (minimize impact)
• load balancing (if multiple instances)

There are several software available to do this :

HaProxy is the best opensource loadbalancer on the market.

Varnish is the best opensource static file cacher on the market.

Nginx is the best opensource webserver on the market.

The configuration files for the following software do the following tasks :

• hide the origin by removing the name IQ from the headers
• insert a cache control header based on the webservice name for instance:
  • 5 min retention for every webservice which name has 05min in it
  • 1 day if the webservice name includes "01day"

NGINX

The interesting section of the configuration file is :

http {  proxy_cache_path  /var/www/cache levels=1:2 keys_zone=my-cache:30m max_size=100m inactive=60m;  proxy_temp_path /var/www/cache/tmp;  proxy_cache my-cache;  server_tokens off;  server {    location ~* 05min*{      add_header Cache-Control "max-age=300";      proxy_pass http://localhost:8181;    }    location ~* 01day*{      add_header Cache-Control "max-age=86400";      proxy_pass http://localhost:8181;    }    location /{      proxy_pass http://localhost:8181;    }  }
}

In this configuration nginx forwards HTTP requests to localhost:8181 which is the address of IQ http port.  localhost:8181

VARNISH

Simply change in /etc/varnish/default.vcl

backend default {    .host = "localhost";    .port = "8181";
}

sub vcl_backend_response {

   #remove origin database server identification   #   SybaseIQ/16.0.0.1226 - see 'iqsrv16 -v' for specifics   unset beresp.http.Server;   #set server and client ttl   if (bereq.url ~ "05min") {      set beresp.ttl = 5m;      set beresp.http.cache-control = "max-age=300";   } else if (bereq.url ~ "01day") {      set beresp.ttl = 1d;      set beresp.http.cache-control = "max-age=86400";   }
}

sub vcl_deliver {
   # Add cache hit information   if (obj.hits > 0) {      set resp.http.X-Cache = "HIT";   } else {      set resp.http.X-Cache = "MISS";   }
}

Conclusion

If you call stored procedures on IQ and can migrate some of your application code to use webservices instead of ODBC/JDBC, then I recommend doing it.

As you can see, caching is fairly easy to implement.

Even if you won't change your application code to use webservice, it is still worth activating the webservices port. For instance, a nagios monitoring probe connects much faster over a webservice than over odbc/jdbc.