)abbrev domain XHASHTBL XHashTable
)if LiterateDoc
\documentclass{article}
\usepackage{url}
\usepackage{color}
\definecolor{bgcode}{rgb}{1,1,0.7}

\usepackage{listings}
\lstdefinelanguage{spad}{basicstyle=\footnotesize\ttfamily,%
  numbers=left,firstnumber=last,backgroundcolor=\color{bgcode}}
\parindent0pt
\parskip\medskipamount

\begin{document}
\title{A hash table implementation via double hashing}
\author{Ralf Hemmecke}
\date{14-Sep-2012}
\maketitle
\begin{abstract}
A hash table domain is implemented. Hash tables are like arrays in the
sense that they allow to insert/extract/delete an element in almost
constant time.
\end{abstract}

\tableofcontents

\section{Overview}
We implement a hash table domain that uses a double hashing strategy.
The main idea is as follows. An extensible array of buckets is used to
store key/entry pairs where the index is computed using a hash
function. If two keys happen to have the same hash value $h_1$, i.e.,
another key/entry pair would have to be stored in the same bucket, a
second hash function is employed to compute another value $h_2$. Then
$h_1+i\cdot h_2$ modulo the array size is used to look for another
free bucket by increasing $i$.

With double hashing it is important to have a special distinct value
that marks an unused vacant bucket. Furthermore, because of the way
the entries are stored via double hashing, deletion of an entry can
just happen by overriding the position with a distinct marker. Thus a
second distinct value is necessary that marks a bucket that has once
been filled but whose value has subsequently been deleted. We call
these marker values \texttt{VACANT} and \texttt{DELETED},
respectively. \texttt{VACANT} and \texttt{DELETED} should not be part
of the key space.

Let's assume that the type of the marker values is \texttt{Marker}.
Naively, and type correctly, we would have to use as representation
like the following.
\begin{verbatim}
Union(Marker, Record(key: Key, entry: Entry))
\end{verbatim}

Of course, for efficiency reasons that is undesirable. For storing
one new key/entry pair, we would have to allocate memory in order to
form the record and another allocation to form the union of the
markers and pairs. Therefore, we assume that markers keys and entries
all need the same amount of memory (which is basically the size of a
pointer) and store markers, keys, and entries in a flat way.

Keys are distinguished from entries by storing keys only in the first
half of the array and entries in the second half. Markers can only
appear in the first half of the array (instead of a key). If the index
corresponding to a key is $p$ and $n$ is half the size of the
array, then the entry corresponding to the key is stored at index $p+n$.

A type in FriCAS that can hold a pointer is \texttt{None}. However,
since \texttt{None} is actually a technical domain that does not have
any inhabitants and is not expressing our idea logically, we introduce
a type \texttt{UMKE} to stand for \texttt{Union(Marker, Key, Entry)} and
represent it by \texttt{None}. For efficiency reasons, \texttt{UMKE}
and its coercions from/to the respective domains will be implemented
via macros.

\section{Implementation}
Let us start with a few common macros.

These two technical macros are necessary to distinguish between Rep
and \%.
)endif

rep x ==> (x@%) pretend Rep
per x ==> (x@Rep) pretend %

)if LiterateDoc
The following macros are just global abbreviations.
)endif

N ==> NonNegativeInteger
Z ==> Integer
I ==> SingleInteger

)if LiterateDoc
Let us now discuss the overall domain implementation.

Since we allow the user to set the hashing function for
the keys at creation time of the table, the \texttt{Key} parameter
would not necessarily have to provide a hashing function of its own.
)endif

++ Author: Ralf Hemmecke
++ Keywords: hash table
++ Description:
++ An implementation of a hash table that uses equality of the key domain
++ to decide upon equality of keys.
XHashTable(Key: SetCategory, Entry: Type):
  Join(TableAggregate(Key, Entry), finiteAggregate, shallowlyMutable) with
    table: (Key -> SingleInteger) -> %
      ++ table(h) creates an empty hash table that uses h instead of
      ++ hash$Key. Note that h should be a mathematical function in
      ++ the sense that from k1=k2 follows h(k1)=h(k2). If that is not
      ++ the case, k1 and k2 will internally be considered as being
      ++ different keys.
 == add
    KE ==> Record(key: Key, entry: Entry)
    UE ==> Union(Entry, "failed")

)if LiterateDoc
\subsection{Markers, MKey, UMKE, Buckets, and coercions}

We need a new type for markers and corresponding coercions.
In the following we assume that $mk$ is a value of type \texttt{MKey}.
)endif

    Marker ==> None
    toMarker mk ==> mk@Marker -- note that MKey==Marker==UMKE
    VACANT : Marker := (HASHTABLEVACANT$Lisp)  pretend Marker -- pos never used
    DELETED: Marker := (HASHTABLEDELETED$Lisp) pretend Marker -- pos is deleted
    vacant?(mk)  ==> EQ(toMarker mk, VACANT)$Lisp
    deleted?(mk) ==> EQ(toMarker mk, DELETED)$Lisp
    key?(mk) ==> not (vacant? mk or deleted? mk)

)if LiterateDoc
Whereas \texttt{vacant?} is true if the bucket is vacant, i.e., has
never been filled with a key, \texttt{deleted?} returns true if the
bucket is marked as being deleted.

We represent buckets as a flat array, thus we use the (virtual) type
\texttt{UMKE} to represent \texttt{Union(Marker,Key,Entry)}.
%
Since markers can only appear in the first half of this array (like
keys), it is advantageous to introduce another type \texttt{MKey}
which stands for \texttt{Union(Marker,Key)}.
)endif

    MKey ==> None
    UMKE ==> None
    Buckets ==> PrimitiveArray UMKE

)if LiterateDoc
Since keys and markers are stored in the first half of the buckets
array, we need a way to get the number of key/entry pairs that we can
store in the array. By construction, this number will be a prime
number.

Below, we reserve the identifier \texttt{a} to be of type \texttt{Buckets}.
)endif
    numOfBuckets(a) ==> shift(#a, -1)

)if LiterateDoc
The value $n$ represent \texttt{numOfBuckets a} and $i$ is the
index into the buckets array, $0 \le i < n$.
)endif

    toUMKE x ==> x pretend UMKE
    toKey k ==> (k@UMKE) pretend Key
    getMKey(a, i)    ==> ((a.i)@UMKE) pretend MKey
    setKey!(a, i, k) ==> (a.i := toUMKE k) pretend Key
    getEntry(a, n, i)     ==> a(n+i) pretend Entry
    setEntry!(a, n, i, e)  ==> (a(n+i) := toUMKE e) pretend Entry
    setKeyEntry!(a, n, i, k, e) ==> (setKey!(a, i, k); setEntry!(a, n, i, e))
    -- deleteKeyEntry!(a, n, i) ==> setKeyEntry!(a, n, i, DELETED, VACANT)
    deleteKeyEntry!(a, n, i) ==>
        (a.i := toUMKE DELETED; a(n+i) := toUMKE VACANT)

)if LiterateDoc
\subsection{Representation}

The representation of the hash table domain cannot simply be a
primitive array. A hash table is unbound in size just like a list.
Therefore we must be able to increase the number of the buckets
whenever a certain load factor is reached. The load factor is given by
the number of actually stored entries divided by the number of buckets.

Thus, we must (together with the array of buckets) store the number of
entries that are actually stored in the array. Furthermore, the
representation stores the maximal number of entries before the growth
of the buckets array is triggered. We call this the maximum load.

Deletion of an entry can only be done by overriding an entry with a
special \texttt{DELETED} marker. These markers are bad with respect to
the probing length in the search of a key. In fact, the number of
entries plus the number of \texttt{DELETED} entries gives a
\emph{virtual} load. If this load reaches a certain value (for the
search algorithm to work correctly there must be at least one
\texttt{VACANT} position) the table is rehashed so that the virtual
load equals the actual load again.
)endif

    maxLoad n ==> n*7 quo 10 -- load factor
    maxVirtualLoad n ==> n*9 quo 10  -- virtual load factor


)if LiterateDoc
Together with these maximal loads, we also store the number of
\texttt{DELETED} entries in the representation.

Each domain in FriCAS that provides a \texttt{hash} function, does
only provide one hash function. However, for double hashing. we
actually need two independent hashing functions. We achieve this by a
trick that is described in \emph{The Art of Computer Programming,
  Volume 3: Sorting and Searching} by D.~E. Knuth.

Instead of doubling the size of the buckets each time we have to make
space for more entries (and thus using a power of 2 as the size), we
actually choose a prime that is close to twice the previous size. If
$h$ is the hash value of a key and $n$ is the maximal number of
entries that can be stored in the array, then $h_1 = h \bmod n$ is the
index that should first be tried to look for the key. The second hash
is computed via $h_2 = h \bmod (n-2)$ under the assumption that $n-2$
is also a prime. In fact, we precompute the possible sizes of the
buckets array by looking for twin primes with appropriate size.
)endif

    arrayLengths: PrimitiveArray N := [[_
        7, 13, 31, 61, 109, 241, 463, 1021, 2029, 4093, 8089, 16363,_
        32719, 65521, 131011, 262111, 524221, 1048573, 2097133,_
        4193803, 8388451, 16777141, 33554011, 67108669, 134217439,_
        268435009, 536870839, 1073741719, 2147482951, 4294965841,_
        8589934291, 17179868809, 34359737299, 68719476391,_
        137438953273, 274877906629, 549755813359, 1099511626399]]

)if LiterateDoc
The above numbers have been computed by the following code chunk.
\begin{lstlisting}[language={},basicstyle=\footnotesize\ttfamily]
prevTwinPrime(x) ==
    p:=prevPrime(x)
    while p-(q:=prevPrime p) > 2 repeat p:=q
    p
[[prevTwinPrime(2^i)::N for i in 3..40]]
\end{lstlisting}

In order to know how many buckets our array has available, we simply
store an index to the above array in the representation.

To be even more flexible and also allow to use hash functions that are
given by the user, we also store such the hashing function in the
representation.

Putting everything together yields the following.
)endif

    Rep ==> Record(_
        numOfEntries: Z,_
        maxNumOfEntries: Z,_
        numOfDeletedEntries: Z,_
        maxNumOfVirtualEntries: Z,_
        idx: Z,_
        arr: Buckets,_
        hashFunction: Key -> I)

)if LiterateDoc
Note that for a value $r$ of type \texttt{Rep} it always holds:
\begin{verbatim}
2*arrayLengths(r.idx) = #(r.arr)
\end{verbatim}
and that for \texttt{n=arrayLengths(r.idx)}, \texttt{a=r.arr}, and
$0\le i < n$ the value \texttt{a.i} is either a marker or a key and
\texttt{a(i+n)} is either \texttt{VACANT} or of type \texttt{Entry}.

\subsection{Auxiliary local functions}

\subsubsection{Searching for a key}
With double hashing, one cannot simply empty the bucket where the key
is stored, i.e., overriding it with its initial value \texttt{VACANT}.
Suppose there are keys $k_1$ and $k_2$ with the same hash value stored
in the hash table and $k_1$ has been inserted before $k_2$ and now we
want to remove the entry that corresponds to $k_1$. If the entry where
$k_1$ is stored were simply overridden by \texttt{VACANT}, then we
would be unable to find $k_2$, because when looking for $k_2$ we would
find a vacant position which signals the end of the probing sequence
and thus that $k_2$ is not stored in the hash table. This is obviously
wrong.

Therefore, we override $k_1$ by \texttt{DELETED} and provide two
search functions. One that looks for the first free position to store
a new entry and another one that searches whether a key is stored in
the hash table.

Both search functions are, naturally, very similar. They differ only
in the while loop condition that decides when to stop the search.
Whereas for the \emph{search for a free position} it is enough to find
the first position where no key is stored, for the \emph{search for
  the existence of a key}, the search must ignore positions that are
marked with \texttt{DELETED}.

In fact, we can combine the two functions. If ever an entry for key
$k$ is to be inserted, it must be first checked whether the key is
already there. And only then we can look for the first free position
to store a new entry. However, the first search ignores all the
positions where \texttt{DELETED} is stored. If we remember the
position where \texttt{DELETED} was encountered for the first time,
there is no need to start searching again from the beginning.

We aim at a search function that takes an array $a$ and a key $k$ and
returns a number $p$ with the following specification. $n$ is supposed
to be the number of buckets, i.e., \texttt{n=numOfBuckets a}.
\begin{itemize}
\item $0 \le p < n$: The key is in the array $a$ at position $p$,
  i.e., \texttt{a.p=k}.
\item $-n \le p < 0$: The key is not in the array, $n+p$ is a free
  position where $k$ should be stored, and \texttt{a(n+p)=VACANT}.
\item $-2n \le p < -n$: The key is not in the array, $2n+p$ is a free
  position where $k$ should be stored, and \texttt{a(2*n+p)=DELETED}.
\end{itemize}

Instead of two separate functions, in the following code we have two
while loops. After the first while loop terminates, we know that
\texttt{mk} is \texttt{VACANT} (if \texttt{deletedPosition?=false}) or
\texttt{DELETED} (if \texttt{deletedPosition?=true}). The
free position is remembered for later.

If both loops are completed  without having found the key, then
depending on \texttt{deletedPosition?}, $q-n$ or $q-2n$ is returned.

If we find the key in the second loop, we move the entry from the
found place to the position of the first \texttt{DELETED} value in the
probing sequence. This ensures that in another search the key is found
in a shorter probing sequence.

Note that because the maximal number of virtual entries (i.e.
\texttt{maxNumOfVirtualEntries} is strictly smaller than the number of
buckets) there is at least one position where \texttt{VACANT} is
stored and thus the loops eventually terminate.
)endif

    localSearch(a: Buckets, k: Key, h: Key -> I): Z ==
        update!(p, mk) ==>
            p := p + h2
            if p>=n then p := p-n
            mk := getMKey(a, p)

        n: Z := numOfBuckets a
        h1: Z := (h k)::Z
        p: Z := positiveRemainder(h1, n) -- position in array
        h2: Z := 1 + positiveRemainder(h1, n-2)
        mk: MKey := getMKey(a, p)
        deletedPosition?: Boolean := false
        while not vacant? mk repeat
            deleted? mk => (deletedPosition? := true; break)
            k = toKey mk => return p -- key found
            update!(p, mk)
        q := p -- first position of a free entry
        -- We ignore DELETED entries when looking for the key.
        while not vacant? mk repeat
            not deleted? mk and k = toKey mk =>
                setKeyEntry!(a, n, q, k, getEntry(a, n, p))
                deleteKeyEntry!(a, n, p)
                return q -- entry has been copied to previous DELETED position
            update!(p, mk)
        if deletedPosition? then q := q-n
        q-n -- KEY not found.

)if LiterateDoc
\subsubsection{Creating a new array of buckets}
To create a new array for the buckets, we reserve memory for twice as
many objects, because we want to store keys in the first half and the
corresponding entries in the second half of this array. All entries
are initialized to being \texttt{VACANT}.
)endif

    newArr(n: N): Buckets == new(2*n, toUMKE VACANT)

)if LiterateDoc
\subsubsection{Growing and rehashing}
When the load factor reaches a certain threshold, the array of buckets
has to be resized, i.e., a new array of approximately twice the size
is reserved and all entries are inserted into the new array.

We start with the main function that inserts the entries from one
bucket array into another. The size of the target array is given by
the index into the \texttt{arrayLengths} array.
)endif

    rehashAux!(x: %, ix: Z): % ==
        m: N := arrayLengths ix
        r: Rep := rep x
        h: Key -> I := r.hashFunction
        a: Buckets := r.arr
        n: Z := numOfBuckets a
        c: Buckets := newArr m
        for i in 0..n-1 | key?(mk: MKey := getMKey(a, i)) repeat
            k: Key := toKey mk
            -- Note that k is not in c, and there are no DELETED positions.
            -- Thus, -m<=p<0.
            p := m + localSearch(c, k, h)
            setKeyEntry!(c, m, p, k, getEntry(a, n, i))
        r.arr := c  --destructively set new array
        r.idx := ix
        r.maxNumOfEntries := maxLoad m
        r.numOfDeletedEntries := 0
        r.maxNumOfVirtualEntries := maxVirtualLoad m
        x

)if LiterateDoc
With this function, the grow! function is simple.
)endif

    grow!(x: %): % == rehashAux!(x, rep(x).idx + 1)

)if LiterateDoc
Rehashing a table is, in fact, creating a new table and inserting all
entries from the original table into the new table.
)endif

    rehash!(x: %): % == rehashAux!(x, rep(x).idx)

)if LiterateDoc
\subsection{Exported functions}
\subsubsection{Creation of a new table}
The user is allowed to provide a specially tailored hash function.
If no function is given, the \texttt{hash} function of the
\texttt{Key} domain is used.

By an internal design decision for the SPAD compiler, at domain
creation the domain function vector is filled with code to actually
fill the respective domain slot. Here it means, if the \texttt{Key}
domain has been created and \texttt{hash} has never been invoked
before, calling \texttt{table(hash)} would store the domain slot
filling code into the representation of the table and not the hash
function itself. That is, of course, inefficient, so we basically
force the filling of the domain slot, and thus storing the true hash
function into the representation.
)endif

    table(hashfunction: Key -> SingleInteger): % ==
        n: N := arrayLengths 0
        maxEntries: Z := maxLoad n
        maxVirtualEntries: Z := maxVirtualLoad n
        per [0, maxEntries, 0, maxVirtualEntries, 0, newArr n, hashfunction]
    empty(): % ==
        table(forceLazySlot((hash$Key)@(Key -> I))$Lisp pretend (Key->I))

)if LiterateDoc
\subsubsection{Getting a random element}
)endif

    inspect(x: %): KE ==
        a := rep(x).arr
        n: Z := numOfBuckets a
        for i in 0..n - 1 | key?(mk: MKey := getMKey(a, i)) repeat
            return [toKey mk, getEntry(a, n, i)]
        error "table must be non-empty"

)if LiterateDoc
\subsubsection{How many entries are stored?}
)endif

    #(x: %): N == rep(x).numOfEntries :: N

)if LiterateDoc
\subsubsection{Searching and accessing the table}
)endif

    search(k: Key, x: %): Union(Entry, "failed") ==
        a: Buckets := rep(x).arr
        h: Key -> I := rep(x).hashFunction
        p: Z := localSearch(a, k, h)
        p<0 => "failed"
        getEntry(a, numOfBuckets a, p)::UE
    elt(x: %, k: Key): Entry ==
        a: Buckets := rep(x).arr
        h: Key -> I := rep(x).hashFunction
        p := localSearch(a, k, h)
        p<0 => error "key not in table"
        getEntry(a, numOfBuckets a, p)
    elt(x: %, k: Key, e: Entry): Entry ==
        a: Buckets := rep(x).arr
        h: Key -> I := rep(x).hashFunction
        p := localSearch(a, k, h)
        p<0 => e
        getEntry(a, numOfBuckets a, p)

)if LiterateDoc
\subsubsection{Inserting a new entry}

A table is never totally filled with entries. There are certain
thresholds that are given by the load factor. If that threshold is
reached, the underlying array of buckets is resized (approximately
twice its current size) and all entries will be (re-)inserted into the
new buckets array.

Since the probing length depends on the number of virtual entries,
i.e., the number of actual entries plus the number of deleted entries,
we will rehash the table (and in that process eventually get rid of the
\texttt{DELETED} entries), if a certain threshold is reached.
)endif

    setelt!(x: %, k: Key, e: Entry): Entry ==
        if rep(x).numOfEntries >= rep(x).maxNumOfEntries then grow! x
        a: Buckets := rep(x).arr
        h: Key -> I := rep(x).hashFunction
        p := localSearch(a, k, h)
        n: Z := numOfBuckets a
        p>=0 => setEntry!(a, n, p, e)
        r: Rep := rep x
        r.numOfEntries := inc(r.numOfEntries)
        p := n+p
        p<0 => -- fill DELETED position
            r.numOfDeletedEntries := dec(r.numOfDeletedEntries)
            setKeyEntry!(a, n, n+p, k, e)
        setKeyEntry!(a, n, p, k, e) -- fill VACANT position
        if r.numOfEntries + r.numOfDeletedEntries > r.maxNumOfVirtualEntries then
            rehash! x
        e

)if LiterateDoc
\subsubsection{Removing an entry}

We cannot simply override an entry with \texttt{VACANT}, because this
might break the chain of entries with the same hash value. One
solution to the problem is to simply mark the entry as
\texttt{DELETED} and adapt the search routine in such a way that a
deleted entry does not stop a probing sequence.
)endif

    remove!(k: Key, x: %): Union(Entry, "failed") ==
        a: Buckets := rep(x).arr
        h: Key -> I := rep(x).hashFunction
        p := localSearch(a, k, h)
        p<0 => "failed" -- key has not been found
        n: Z := numOfBuckets a
        e: Entry := getEntry(a, n, p) -- to be returned
        deleteKeyEntry!(a, n, p)
        rep(x).numOfEntries := dec(rep(x).numOfEntries)
        rep(x).numOfDeletedEntries := inc(rep(x).numOfDeletedEntries)
        e::UE

)if LiterateDoc
\subsubsection{Copying a table}
After copying, no exported operation from this domain performed on the
copy should have any destructive effect on the original.
)endif

    copy(x: %): % ==
        r: Rep := rep x
        per [r.numOfEntries, r.maxNumOfEntries,_
             r.numOfDeletedEntries, r.maxNumOfVirtualEntries,_
             r.idx, copy(r.arr), r.hashFunction]

)if LiterateDoc
\subsubsection{Setting common values}
)endif

    fill!(x: %, e: Entry): % ==
        a := rep(x).arr
        n: N := numOfBuckets a
        for i in 0..n-1 | key? getMKey(a, i) repeat setEntry!(a, n, i, e)
        x


)if LiterateDoc
\subsubsection{Applying a function to all entries}
)endif

    map!(f: Entry -> Entry, x: %): % ==
        a := rep(x).arr
        n: N := numOfBuckets a
        for i in 0..n-1 | key? getMKey(a, i) repeat
            setEntry!(a, n, i, f getEntry(a, n, i))
        x

)if LiterateDoc
\subsubsection{Extracting keys and entries}
)endif

    keys(x: %): List Key ==
        a := rep(x).arr
        l: List Key := empty()
        for i in 0..numOfBuckets a - 1 | key?(mk: MKey := getMKey(a, i)) repeat
            l := cons(toKey mk, l)
        l
    parts(x: %): List Entry ==
        a := rep(x).arr
        n: N := numOfBuckets a
        l: List Entry := empty()
        for i in 0..n-1 | key? getMKey(a, i) repeat
            l := cons(getEntry(a, n, i), l)
        l
    parts(x: %): List KE ==
        a := rep(x).arr
        n: N := numOfBuckets a
        l: List KE := empty()
        for i in 0..n-1 | key?(mk: MKey := getMKey(a, i)) repeat
            l := cons([toKey mk, getEntry(a, n, i)], l)
        l

)if LiterateDoc
\subsubsection{Removing duplicates}
We provide a more efficient function. If the hash function $h$ is a
mathematical function, i.e., if $k_1=k_2$ then $h(k_1)=h(k_2)$, then
there can be no duplicates in the table.
)endif

    removeDuplicates(x: %): % == x

)if LiterateDoc
\subsubsection{More efficient equality testing}

Equality is by default implemented via \texttt{DictionaryOperations}
which means that during testing records for the key/entry pairs will
be allocated. For that reason we implement equality testing here
without such allocation.

Equality is only exported if the \texttt{Entry} type is a \texttt{BasicType}.
)endif

    if Entry has BasicType then
        ((x: %) = (y: %)): Boolean ==
            #x ~= #y => false
            xa := rep(x).arr; xn := numOfBuckets xa
            ya := rep(y).arr; yn := numOfBuckets ya
            h := rep(y).hashFunction
            for i in 0..xn - 1 | key?(mk: MKey := getMKey(xa, i)) repeat
                p := localSearch(ya, toKey mk, h)
                p < 0 => return false
                getEntry(xa, xn, i) ~= getEntry(ya, yn, p) => return false
            true

)if LiterateDoc
\end{document}
)endif
