(back to main page)

We're using kdb+ to manage a several-hundred gigabyte database of historical timeseries data. Originally, we built a soap web service in java with api's for retrieving timeseries from this database. The java code makes use of q's java connector to issue data requests. This allows non-q and non-java applications to perform these retrievals in a consistent manner.

It turns out that it's preferable for some applications to use a non-soap, tcp-based retrieval mechanism where the requests and responses consist of a simple, non-validated dialect of xml. We realized that it would be useful and more efficient if we could make those queries directly to the q server instead of routing them through an intermediate web service, and therefore, it became desirable to create a simple procedure for serializing and deserializing q entities that represent the requests and responses for these queries. Non-q apps could then retrieve this data directly from q via a simple tcp request without having to use any of the kdb+ client libraries.

I have written this xml serializer/deserializer for q entities. You can get the code here. The two principal functions are xfq ("xml from q") and qfx ("q from xml").

xfq:

converts a q entity into its xml representation

argument 0:char list or symbol representing the name that will be assigned to resulting xml
argument 1:the entity to be xmlized
returns:the resulting xml

example:

q)k)a:(+(`a;`b)!((`q;`r;`s);(`i;`j;`k)))!+(`c;`d)!((0;1;2);(1.1;1.2;1.3))
q)a
a b| c d
---| -----
q i| 0 1.1
r j| 1 1.2
s k| 2 1.3
q)axml:xfq[`newa;a]
q)-1 axml
<newa>
 <dictkey>
  <tablekey>
   <list>
    <symbol>a</symbol>
    <symbol>b</symbol>
   </list>
  </tablekey>
  <tablevalue>
   <list>
    <list>
     <symbol>q</symbol>
     <symbol>r</symbol>
     <symbol>s</symbol>
    </list>
    <list>
     <symbol>i</symbol>
     <symbol>j</symbol>
     <symbol>k</symbol>
    </list>
   </list>
  </tablevalue>
 </dictkey>
 <dictvalue>
  <tablekey>
   <list>
    <symbol>c</symbol>
    <symbol>d</symbol>
   </list>
  </tablekey>
  <tablevalue>
   <list>
    <list>
     <int>0</int>
     <int>1</int>
     <int>2</int>
    </list>
    <list>
     <float>1.1</float>
     <float>1.2</float>
     <float>1.3</float>
    </list>
   </list>
  </tablevalue>
 </dictvalue>
</newa>
-1
qfx:

converts the xml representation of a q entity back into its q form

argument:the xml to be converted to a q entity
returns:a symbol representing the q entity which now contains the converted xml

example (using `a and `axml as defined above):

q)qfx axml
`newa
q)newa
a b| c d
---| -----
q i| 0 1.1
r j| 1 1.2
s k| 2 1.3
q)newa~a
1b

software licensing:

Copyright (C) 2007, 2008, Lloyd Zusman <q.o@potam.us>

This program is free software; you can redistribute
it and/or modify it under the terms of Version 2 of
the GNU General Public License as published by the
Free Software Foundation.

This program is distributed in the hope that it will
be useful, but WITHOUT ANY WARRANTY; without even the
implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.  See the GNU General Public License
for more details: http://q.o.potam.us/license or
http://www.gnu.org/copyleft/gpl.html

You should have received a copy of the GNU General
Public License along with this program; if not, write to
the Free Software Foundation, Inc., 51 Franklin Street,
Fifth Floor, Boston, MA  02110-1301, USA.