WEB-SNAP: POMDP-based Knowledge Engineering for Assistive Systems

https://cs.uwaterloo.ca/~mgrzes/code/web-snap/

While  this code  is made  publicly  available, all  work that  uses this  code,
extends this  code or makes  a comparison  to this code  is required to  cite my
paper:

Marek Grzes,  Jesse Hoey, Shehroz  Khan, Alex Mihailidis, Stephen  Czarnuch, Dan
Jackson  and  Andrew Monk:  Relational  Approach  to Knowledge  Engineering  for
POMDP-based  Assistance  Systems as  a  Translation  of a  Psychological  Model.
International  Journal  of  Approximate   Reasoning  (IJAR).  Special  Issue  of
Applications of Bayesian Networks. Elsevier, 2013.

INFO

Software implemented by Marek Grzes.  A tool for designing POMDP-based assistive
systems by people who are not necessarily experts in artificial intelligence. No
POMDP related knowledge is required. The designer  works with an easy to use web
interface were domain details can be  specified. After that the full POMDP model
(in the SPUDD/SymbolicPerseus  format) is generated automatically and  it can be
used for decision making in prompting/assistive systems. Written in Java and PHP
and requires a web-server (e.g., Apache) and a PostgreSQL database.

1. PACKAGE CONTENTS

web-main -  contains the main  web application that  should be installed  on the
web  server.  In  order  to  import  this project  into  Eclipse  PHP,  you  can
use  "File->Import->General->Existing Projects  into workspace"  and select  the
web-main directory.  This way,  the entire web-snap  directory should  be loaded
into the  current Eclipse  workspace. I've  just checked that  it works  on Zend
Eclipse for PHP Developers Version: 3.2.0.

SnapGenerator - contains  Java files that generate the actual  POMDP file in the
SPUDD/SymbolicPerseus  format  using  the  data  found  in  the  database.  This
application  reads  in the  data  from  the  database,  and constructs  a  fully
specified POMDP model which can be solved using the SymbolicPerseus planner. The
planner also contains a simulator that can  be used to simulate the policy after
it has been solved. It can be further used to build arbitrary tools that execute
the POMDP policy and update the belief state.

lib - contains a copy of symbolicPerseus written in Java

examples - example models  that can be loaded to the database,  as well as POMDP
files in the symbolicPerseus format that were generated using this software

screenshots - several screenshots that show how the web interface looks like

2. INSTALLATION

web-main requires a  web server with PHP support and  a PostgreSQL database. The
following  shows how  to  configure the  Apache web  server  and the  PostgreSQL
database on Ubuntu 12.04 so that web-main can be executed.

2.1 Install the Apache server with PHP plugins

sudo apt-get install apache2 libapache2-mod-php5 php5-pgsql

Go to http://localhost  to see if Apache  is running. You should be  able to see
the default Apache web-sited that says "It works!".

2.1.1 set up htaccess in Apache

Go to /etc/apache2/sites-available/default and replace "AllowOverride None" with
"AllowOverride AuthConfig". Restart Apache using:

sudo /etc/init.d/apache2 restart

2.1.2 Set up PHP for Apache

PHP modules for Apache may require the following command to be executed:

a2enmod php5

2.2 Install PostgreSQL

sudo apt-get install postgresql

Change password for the postgres user. Type

sudo -u postgres psql

and then you can change password for user postgres using:

ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'mypass-here';

the same password as above should be set for the unix accout of user postgres:

sudo passwd -d postgres

sudo su postgres -c passwd

If only the  above is done, the  server should be accessible  from the localhost
only, the  problem is that with  this the custom user  has to exist also  in the
system,  to  avoid  this  limitation,  the  following has  to  be  done:  Go  to
/etc/postgresql/9.1/main/pg_hba.conf  and replace  "ident" with  "trust" in  the
following line:

local all all trust

Now, you  can create your actual  user account, "x721demouser", in  the database
engine:

sudo -u postgres createuser -D -A -P x721demouser

And then create the default database which is required by web-snap:

sudo -u postgres createdb -O x721demouser snap-config

At some point I also had to uncomment this line:

listen_addresses = 'localhost'

and set

password_encryption = on

in /etc/postgresql/9.1/main/postgresql.conf

After  performing all  steps  up to  here,  I could  connect  to snap-config  as
x721demouser using:

psql -d snap-config -U x721demouser -W

At this point, we have PostgreSQL  running with user x721demouser which is going
to be  our entry point  to the  database from Java  and PHP codes.  Example snap
databases will be created (loaded)  below. snap-config is used for configuration
only and does not store any data related to particular snap models.

2.3 Install web-main

web-main does not  require any compilation. After all the  required software has
been installed, the full web-snap package should be checked out as follows:

cd /var/www hg

clone https://bitbucket.org/mgrzes/web-snap

After that, the following steps are required:

2.3.1    Check    if    php    is   properly    configured.    You    do    this
using    the    link   http://localhost/web-snap/web-main/index.php/?test    and
http://localhost/web-snap/web-main/index.php/?phpinfo. If it  does now work, you
need to verify your installation of Apache and PHP.

2.3.2 Add  your password  to the /var/www/web-snap/web-main/.htpasswd  file. You
can encrypt your password using:

htpasswd -n snapuser

and then add the  encrypted version of your password to  that file. Ideally, for
security  reasons, you  should move  .htpasswd  to another  location and  update
the  .htaccess  file.  Right  now  they  are  in  the  same  directory  to  make
installation easier. Note that you need to change the path in .htaccess when you
are  installing web-snap  in  a  different directory.  The  current password  is
'MtNjxQHP4Y' in case you won't skip this step initially.

If this step is  done correctly, you should be prompted to type  a user name and
password when you go to:

http://localhost/web-snap/web-main/

2.3.3 In the above  web-site, there were no instances to work  with. Now, we are
going  to  create  several  empty  databases. Each  database  will  be  named  a
snap-instance  and  the idea  is  that  one POMDP  will  be  generated from  one
instance. Hence, one logical database stores one snap-instance, and one instance
generates one POMDP. New databases can be created using:

sudo -u postgres createdb -O x721demouser snap-demo

sudo -u postgres createdb -O x721demouser snap-test

2.3.3  Having two  empty logical  databased created  above, we  can load  schema
(empty tables) to those database using the following command:

cd /var/www/web-snap

psql -d snap-demo -U x721demouser -W < SnapGenerator/sql-files/create-snapdb.sql

psql -d snap-test -U x721demouser -W < SnapGenerator/sql-files/create-snapdb.sql

At this  point, if we  go to http://localhost/web-snap/web-main/, we  can select
one of the two snap-instances to work with.

2.3.4 Load example models to the database:

The  two  databases created  in  the  previous  step  are empty.They  are  empty
initially.  There  is one  small  example  than we  can  load  into an  existing
database. Let's say that we want to load that example to snap-demo.

psql -d snap-demo -U x721demouser -W < SnapGenerator/sql-files/snap-door4-example.sql

This is a very  simple prompting model where the person is  supposed to open the
door. In  order to do that,  the person needs  an ability of recognising  a door
handle. If the person  is not able to do the task, the  system should prompt her
to recognise the door handle. We will show below how to simulate this model, but
before that, we need to compile the snap generator.

Overall,  at  this   point,  the  web  interface  should  work   if  you  go  to
http://localhost/web-snap/web-main. You can work on  your model, but in order to
be able to generate the POMDP file, you need to compile the SnapGenerator.

2.4 Compile and Install the Snap Generator

If  the  repository was  checked  out  from  bitbucket  as I  mentioned  before,
SnapGenerator should be found here: /var/www/web-snap/SnapGenerator.

2.4.1 Required Libraries

PostgreSQL JDBC which can be installed as follows in Ubuntu

sudo apt-get install libpostgresql-jdbc-java

2.4.2 Compilation and Installation

cd /var/www/web-snap/SnapGenerator

bash create_jar.sh

snap-ubuntu.jar should appear as a result in the current directory.

Once compiled, there is  no need to move the jar  file; the web-main application
is going to use that jar file from its default location.

At this point entire application should be installed.

3. HOW TO USE IT?

In this system,  every model is stored  in a separate logical  database which we
call a  snap-instance; every  such instance  can be used  to generate  one POMDP
file. Normally, one POMDP should be  sufficient to model one assistive task. The
problem is that sometimes  large POMDPs cannot be solved by  the planner and the
model has  to be split  into a  number of POMDPs  and some form  of hierarchical
control is  required. In this  manual, I won't  go into details  of hierarchical
control, but  it is supported by  this software as well  because extra variables
required by  the hierarchical controller  can be  generated on request  (see the
check boxes located above the generate  button). For those who are interested in
hierarchical control, I would recommend looking into this paper:

Jesse Hoey and Marek Grzes: Distributed  control of situated assistance in large
domains with many tasks. In Proceedings of International Conference on Automated
Planning and Scheduling (ICAPS), Freiburg, Germany, 2011.

When the new  prompting model is designed using web-snap,  the designer performs
task analysis  which breaks down the  task into smaller elements.  In this work,
syndetic modelling form  psychology is used because it nicely  fits into dynamic
Bayesian networks. The  following paper describes the  underlying methodology in
more detail:

Jesse Hoey, Thomas  Ploetz, Dan Jackson, Patrick Olivier, Andrew  Monk and Cuong
Pham Rapid Specification and Automated Generation of Prompting Systems to Assist
People with Dementia. Pervasive and Mobile Computing, 7, 3, June, 2011.

A  practical and  detailed case  study that  shows how  to use  this methodology
through this software package can be found in Section 4 of

Marek Grzes,  Jesse Hoey, Shehroz  Khan, Alex Mihailidis, Stephen  Czarnuch, Dan
Jackson  and  Andrew Monk:  Relational  Approach  to Knowledge  Engineering  for
POMDP-based  Assistance  Systems as  a  Translation  of a  Psychological  Model.
International  Journal  of  Approximate   Reasoning  (IJAR).  Special  Issue  of
Applications of Bayesian Networks. Elsevier, 2013.

I would encourage anybody who wants to  use this software to read that paper and
refer to  section 4 when designing  own models. Additionally, here,  I summarise
this methodology at the very high level with hope that it will be useful for you
to start using the software.

We want  to use  this software  to design  a new  prompting system.  The example
challenge could be to design a prompting  system for the hand washing task where
the person  with a cognitive  disability should be able  to wash hands  with the
help  of  the promoting  system.  The  entire  prompting systems  would  include
hardware, software  to control the  hardware and collect readings  from sensors,
and software that implements decision making  engine. Here, we are talking about
designing  such an  intelligent decision  making  engine based  on POMDPs.  This
software adheres  to the rules  of the psychological interactive  unit analysis,
and the  overall resulting  methodology follows the  way psychologists  would do
their analysis.

When the  interactive unit  (IU) analysis  is performed by  a psychologist  or a
trained health  professional, the  outcome of  such an  analysis is  usually the
spreadsheet-like document that  describes the IU table (see Table  1 in our IJAR
paper for an example). An important observation is that the professional, before
creating  the IU  table,  writes down  its components,  where  the task  states,
abilities and  behaviours are distinguished.  The health professional  will also
determine how every  behaviour of the user  changes the state of  the world, and
what  kinds of  preconditions are  required for  those changes  to happen.  This
naturally corresponds to the  use of our tool that, in  this case, would require
accomplishing a number  steps that are explained  in detail in Section  4 of our
paper.

The designer normally starts the process  by watching the video showing the user
washing her hands with the help of a  human care giver who would prompt the user
when  required. The  designer  watches  the video  several  times an  identifies
elements required in order to complete the IU analysis.

3.1 Design Your Model

Assuming that  you want to  design a  new model, you  would start with  an empty
instance and you would  need to perform a number of steps  explained in the case
study  in Section  4  of our  IJAR  paper. This  includes  identifying task  and
environment  features,  dynamics  of  user behaviours,  the  IU  table,  rewards
and  costs  of prompting,  and  finally  identify  sensors and  their  precision
(information about sensors has to be provided by engineers who are familiar with
sensors that  the POMDP is going  to work with).  Each of these elements  can be
specified in corresponding tables through the web interface.

3.2 Generate POMDPs

Once the initial model has been defined,  the user can generate its formal POMDP
specification. For that, the user clicks  the "SNAP generator" option in the top
menu.  In  there,  there is  a  detailed  description  of  the process  and  the
"Generate" button can be found. Once  clicked, the POMDP model will be generated
on the server and  it can be saved on a local computer.  Having that POMDP file,
the user can go to the next step which solves the model using the POMDP planner.

3.3 Solve POMDPs

This software comes with the POMDP file for the snap-demo example used before in
this manual. In order to solve that  example (or any other POMDP model), one can
type:

cd /var/www/web-snap

java -cp lib/symbolicPerseus.jar pomdp.Solver examples/pomdp/snap-demo-door4.txt -g -b 1500 -r 2

where symbolicPerseus.jar  is a  jar file  that contains  symbolicPerseus. Other
planners could be used as long as they can read in the symbolicPerseus format.

3.4 Simulate POMDPs

To simulate the policy using command line

cd /var/www/web-snap

java -cp lib/symbolicPerseus.jar pomdp.Solver examples/pomdp/snap-demo-door4.txt -i examples/pomdp/snap-demo-door4.pomdp -s 10

The simulator starts  and the user can type observations  using command line and
overwrite (if necessary)  actions suggested by the policy. The  belief state and
Q-values are also printed  in order to provide more insight  into the quality of
the  POMDP policy  and  the behaviour  of  the overall  model.  Note, that  poor
prompting does not  mean that the policy is not  good enough (symbolicPerseus is
an approximated planner);  it usually happens when the model  is not correct and
needs to be adjusted. Normally, the  user would generate the POMDP, simulate it,
and  then go  back to  the  model to  refine  it. The  process requires  several
iterations of  these steps,  and usually  leads to  a model  that does  what the
designer wants to achieve.

3.5 Use POMDP Policies in Your Software

After  you've successfully  simulated your  model, you  can use  it in  your own
program.  For  that  you  would  need  to look  into  the  source  code  of  the
SymbolicPerseus and  you would need to  extract parts that load  the POMDP model
and the policy, and  parts that do belief update, query the  policy, and read in
observations. Having  this functionality, you will  need to write your  own code
that provides observations (sensors), and takes and executes actions returned by
the policy.

4. EXAMPLES

Folders examples/sql and  examples/pomdp contain several models  in the database
format and the symbolicPerseus format respectively. As for autumn 2013, I do not
remember very well which exact model was used  in our IJAR paper, but it is very
likely that  snap-hw_one_pomdp_version10 is the one  that was used for  the hand
washing experiment with the human actor. In order to load the model

examples/sql/snap-hw_one_pomdp_version10_full.sql

to the  database, the following  steps would be required.  First, we need  a new
database for that model:

sudo -u postgres createdb -O x721demouser snap-hw_one_pomdp_version10

Second, we need to load the data and the schema:

psql -d snap-hw_one_pomdp_version10 -U x721demouser -W < examples/sql/snap-hw_one_pomdp_version10_full.sql

After  that, the  POMDP file  for  that model  can  be generated  using the  web
interface. An example output for that model can be found in:

examples/pomdp/snap-hw_one_pomdp_version10.txt

It is a  valid POMDP model in  the symbolicPerseus format, and it  can be solved
and simulated according to our description in Section 3.

There are a few other models in examples. For example,

examples/sql/snap-hw_one_pomdp_version11_full.sql

is another version of the hand washing  problem that was used for testing. There
are a few  other models in examples/sql  from other projects. You  can load them
using the above steps.

DISCLAIMER

This program is distributed in the hope  that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty  of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

-----------------

Have fun!

Marek Grzes, Waterloo, 2013