System.IO.Streams.Tutorial

io-streams-1.5.2.2: Simple, composable, and easy-to-use stream I/O

Safe Haskell | Safe-Inferred
Language | Haskell2010

System.IO.Streams.Tutorial

Contents

- Introduction
- Build Input Streams
- Build Output Streams
- Connect Streams
- Transform Streams
- Resource and Exception Safety
- Pushback
- Thread Safety
- Examples

Synopsis

# Introduction

The `io-streams` package defines two "smart handles" for stream processing:

- `InputStream`: a read-only smart handle
- `OutputStream`: a write-only smart handle

The `InputStream` type implements all the core operations we expect for a read-only handle. We consume values using `read`, which returns a `Nothing` when the resource is done:

    read :: InputStream c -> IO (Maybe c)

... and we can push back values using `unRead`:

    unRead :: c -> InputStream c -> IO ()

The `OutputStream` type implements the `write` operation which feeds it output, supplying `Nothing` to signal resource exhaustion:

    write :: Maybe c -> OutputStream c -> IO ()

These streams slightly resemble Haskell `Handle`s, but support a wider range of sources and sinks. For example, you can convert an ordinary list to an `InputStream` source and interact with it using the handle-based API:

    ghci> import qualified System.IO.Streams as S
    ghci> listHandle <- S.fromList [1, 2]
    ghci> S.read listHandle
    Just 1
    ghci> S.read listHandle
    Just 2
    ghci> S.read listHandle
    Nothing

Additionally, IO streams come with a library of stream transformations that preserve their handle-like API. For example, you can map a function over an `InputStream`, which generates a new handle to the same stream that returns transformed values:

    ghci> oldHandle <- S.fromList [1, 2, 3]
    ghci> newHandle <- S.mapM (\x -> return (x * 10)) oldHandle
    ghci> S.read newHandle
    10
    ghci> -- We can still view the stream through the old handle
    ghci> S.read oldHandle
    2
    ghci> -- ... and switch back again
    ghci> S.read newHandle
    30

IO streams focus on preserving the convention of traditional handles while offering a wider library of stream-processing utilities.

# Build Input Streams

The `io-streams` library provides a simple interface for creating your own `InputStream`s and `OutputStream`s.

You can build an `InputStream` from any `IO` action that generates output, as long as it wraps results in `Just` and uses `Nothing` to signal EOF:

    makeInputStream :: IO (Maybe a) -> IO (InputStream a)

As an example, let's wrap an ordinary read-only `Handle` in an `InputStream`:

    import Data.ByteString (ByteString)
    import qualified Data.ByteString as S
    import System.IO.Streams (InputStream)
    import qualified System.IO.Streams as Streams
    import System.IO (Handle, hFlush)

    bUFSIZ = 32752

    upgradeReadOnlyHandle :: Handle -> IO (InputStream ByteString)
    upgradeReadOnlyHandle h = Streams.makeInputStream f
      where
        f = do
            x <- S.hGetSome h bUFSIZ
            return $! if S.null x then Nothing else Just x

We didn't even really need to write the `upgradeReadOnlyHandle` function, because System.IO.Streams.Handle already provides one that uses the exact same implementation given above:

    handleToInputStream :: Handle -> IO (InputStream ByteString)

# Build Output Streams

Similarly, you can build any `OutputStream` from an `IO` action that accepts input, as long as it interprets `Just` as more input and `Nothing` as EOF:

    makeOutputStream :: (Maybe a -> IO ()) -> IO (OutputStream a)

A simple `OutputStream` might wrap `putStrLn` for `ByteString`s:

    import Data.ByteString (ByteString)
    import qualified Data.ByteString as S
    import System.IO.Streams (OutputStream)
    import qualified System.IO.Streams as Streams

    writeConsole :: IO (OutputStream ByteString)
    writeConsole = Streams.makeOutputStream $ \m -> case m of
        Just bs -> S.putStrLn bs
        Nothing -> return ()

The `Just` wraps more incoming data, whereas `Nothing` indicates the data is exhausted. In principle, you can feed `OutputStream`s more input after writing a `Nothing` to them, but IO streams only guarantee a well-defined behavior up to the first `Nothing`. After receiving the first `Nothing`, an `OutputStream` could respond to additional input by:

- Using the input
- Ignoring the input
- Throwing an exception

Ideally, you should adhere to well-defined behavior and ensure that after you write a `Nothing` to an `OutputStream`, you don't write anything else.

# Connect Streams

`io-streams` provides two ways to connect an `InputStream` and `OutputStream`:

    connect :: InputStream a -> OutputStream a -> IO ()
    supply  :: InputStream a -> OutputStream a -> IO ()

`connect` feeds the `OutputStream` exclusively with the given `InputStream` and passes along the end-of-stream notification to the `OutputStream`.

`supply` feeds the `OutputStream` non-exclusively with the given `InputStream` and does not pass along the end-of-stream notification to the `OutputStream`.

You can combine both `supply` and `connect` to feed multiple `InputStream`s into a single `OutputStream`:

    import qualified System.IO.Streams as Streams
    import System.IO (IOMode(WriteMode))

    main = do
       Streams.withFileAsOutput "out.txt" WriteMode $ \outStream ->
       Streams.withFileAsInput  "in1.txt" $ \inStream1 ->
       Streams.withFileAsInput  "in2.txt" $ \inStream2 ->
       Streams.withFileAsInput  "in3.txt" $ \inStream3 ->
       Streams.supply  inStream1 outStream
       Streams.supply  inStream2 outStream
       Streams.connect inStream3 outStream

The final `connect` seals the `OutputStream` when the final `InputStream` terminates.

Keep in mind that you do not need to use `connect` or `supply` at all: `io-streams` mainly provides them for user convenience. You can always build your own abstractions on top of the `read` and `write` operations.

# Transform Streams

When we build or use `IO` streams we can tap into all the stream-processing features the `io-streams` library provides. For example, we can decompress any `InputStream` of `ByteString`s:

    import Control.Monad ((>=>))
    import Data.ByteString (ByteString)
    import System.IO (Handle)
    import System.IO.Streams (InputStream, OutputStream)
    import qualified System.IO.Streams as Streams
    import qualified System.IO.Streams.File as Streams

    unzipHandle :: Handle -> IO (InputStream ByteString)
    unzipHandle = Streams.handleToInputStream >=> Streams.decompress

... or we can guard it against a denial-of-service attack:

    protectHandle :: Handle -> IO (InputStream ByteString)
    protectHandle =
        Streams.handleToInputStream >=> Streams.throwIfProducesMoreThan 1000000

`io-streams` provides many useful functions such as these in its standard library and you take advantage of them by defining IO streams that wrap your resources.

# Resource and Exception Safety

IO streams use standard Haskell idioms for resource safety. Since all operations occur in the IO monad, you can use `catch`, `bracket`, or various "`with...`" functions to guard any `read` or `write` without any special considerations:

    import qualified Data.ByteString as S
    import System.IO
    import System.IO.Streams (InputStream, OutputStream)
    import qualified System.IO.Streams as Streams
    import qualified System.IO.Streams.File as Streams

    main =
        withFile "test.txt" ReadMode $ \handle -> do
            stream <- Streams.handleToInputStream handle
            mBytes <- Streams.read stream
            case mBytes of
                Just bytes -> S.putStrLn bytes
                Nothing    -> putStrLn "EOF"

However, you can also simplify the above example by using the convenience function `withFileAsInput` from System.IO.Streams.File:

    withFileAsInput
     :: FilePath -> (InputStream ByteString -> IO a) -> IO a

# Pushback

All `InputStream`s support pushback, which simplifies many types of operations. For example, we can `peek` at an `InputStream` by combining `read` and `unRead`:

    peek :: InputStream c -> IO (Maybe c)
    peek s = do
        x <- Streams.read s
        case x of
            Nothing -> return ()
            Just c  -> Streams.unRead c s
        return x

... although System.IO.Streams already exports the above function.

`InputStream`s can customize pushback behavior to support more sophisticated support for pushback. For example, if you protect a stream using `throwIfProducesMoreThan` and `unRead` input, it will subtract the unread input from the total byte count. However, these extra features will not interfere with the basic pushback contract, given by the following law:

    unRead c stream >> read stream == return (Just c)

When you build an `InputStream` using `makeInputStream`, it supplies the default pushback behavior which just saves the input for the next `read` call. More advanced users can use System.IO.Streams.Internal to customize their own pushback routines.

{- NOTE: The library only exports pushback API for Sources, which are a completely internal type, so should we teach the user how to define custom pushback or not? Maybe that belongs in some sort of separate "advanced" tutorial for System.IO.Streams.Internal. -}

# Thread Safety

IO stream operations are not thread-safe by default for performance reasons. However, you can transform an existing IO stream into a thread-safe one using the provided locking functions:

    lockingInputStream  :: InputStream  a -> IO (InputStream  a)
    lockingOutputStream :: OutputStream a -> IO (OutputStream a)

These functions do not prevent access to the previous IO stream, so you must take care to not save the reference to the previous stream.

{- NOTE: Should I give specific performance numbers or just say something like "a slight cost to performance" for locking? -} {- NOTE: This could use a concrete example of a race condition that a user might encounter without this protection. -}

# Examples

The following examples show how to use the standard library to implement traditional command-line utilities:

    {-# LANGUAGE OverloadedStrings #-}

    import Control.Monad ((>=>), join)
    import qualified Data.ByteString.Char8 as S
    import Data.Int (Int64)
    import Data.Monoid ((<>))
    import System.IO.Streams (InputStream)
    import qualified System.IO.Streams as Streams
    import System.IO
    import Prelude hiding (head)

    cat :: FilePath -> IO ()
    cat file = withFile file ReadMode $ \h -> do
        is <- Streams.handleToInputStream h
        Streams.connect is Streams.stdout

    grep :: S.ByteString -> FilePath -> IO ()
    grep pattern file = withFile file ReadMode $ \h -> do
        is <- Streams.handleToInputStream h >>=
              Streams.lines                 >>=
              Streams.filter (S.isInfixOf pattern)
        os <- Streams.unlines Streams.stdout
        Streams.connect is os

    data Option = Bytes | Words | Lines

    len :: InputStream a -> IO Int64
    len = Streams.fold (\n _ -> n + 1) 0

    wc :: Option -> FilePath -> IO ()
    wc opt file = withFile file ReadMode $
        Streams.handleToInputStream >=> count >=> print
      where
        count = case opt of
            Bytes -> \is -> do
                (is', cnt) <- Streams.countInput is
                Streams.skipToEof is'
                cnt
            Words -> Streams.words >=> len
            Lines -> Streams.lines >=> len

    nl :: FilePath -> IO ()
    nl file = withFile file ReadMode $ \h -> do
        nats <- Streams.fromList [1..]
        ls   <- Streams.handleToInputStream h >>= Streams.lines
        is   <- Streams.zipWith
                    (\n bs -> S.pack (show n) <> " " <> bs)
                    nats
                    ls
        os   <- Streams.unlines Streams.stdout
        Streams.connect is os

    head :: Int64 -> FilePath -> IO ()
    head n file = withFile file ReadMode $ \h -> do
        is <- Streams.handleToInputStream h >>= Streams.lines >>= Streams.take n
        os <- Streams.unlines Streams.stdout
        Streams.connect is os

    paste :: FilePath -> FilePath -> IO ()
    paste file1 file2 =
        withFile file1 ReadMode $ \h1 ->
        withFile file2 ReadMode $ \h2 -> do
        is1 <- Streams.handleToInputStream h1 >>= Streams.lines
        is2 <- Streams.handleToInputStream h2 >>= Streams.lines
        isT <- Streams.zipWith (\l1 l2 -> l1 <> "\t" <> l2) is1 is2
        os  <- Streams.unlines Streams.stdout
        Streams.connect isT os

    yes :: IO ()
    yes = do
        is <- Streams.fromList (repeat "y")
        os <- Streams.unlines Streams.stdout
        Streams.connect is os