cbmconvert release notes

# `cbmconvert` release notes

`cbmconvert`
  create, extract and convert different Commodore binary archives

Copyright © 1993‒2006,2021‒2022,2024 Marko Mäkelä.

> This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

> This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

## Table of Contents

  1 Introduction
  2 Getting Started
  3 Using `cbmconvert`
    3.1 `cbmconvert`
      3.1.1 CP/M disk images
      3.1.2 GEOS disk images
      3.1.3 Commodore C2N tape archives
      3.1.4 Flaws of `cbmconvert`
    3.2 `disk2zip`
    3.3 `zip2disk`
  4 Credits
  5 Distribution
    5.1 Revision History

## 1 Introduction

There are many archiving programs for the Commodore 64, all of which are incompatible with archiving programs on other systems. This `cbmconvert` utility tries to address this problem. It extracts files from most known file formats and writes them to several different formats, including some formats used by some Commodore 64 emulators.

This release of `cbmconvert` does not contain LHARC support any more, since 1541 disk images (either plain or ZipCoded) and Lynx archives are much easier to handle. If you need LHARC support, get the older release, `cbmconvert 1.1`.

All code has been cleaned up. The file type management in `cbmconvert 2.0` is very flexible, and it is very easy to add file types later. Currently `cbmconvert` reads the following file types: Arkive, Lynx, ARC64/128, PC64, tape images (T64 and native Commodore C2N), disk images (1541, 1571, 1581, both native and C128 CP/M format), and raw files. Conversions between 1541 disk images and ZipCoded files can be made with the two supplied utilities `disk2zip` and `zip2disk`.

## 2 Getting Started

Compile `cbmconvert` as follows:

    mkdir build
    cd build
    cmake ..
    cmake --build .
    cmake --install .

The compilation should leave you with three executables: `cbmconvert`, `zip2disk` and `disk2zip`. The installation step is optional; you may also invoke the executables from the build directory.

16-bit compilers will probably have trouble with `cbmconvert`. For instance, the standard file I/O library in Borland C uses signed 16-bit integers for `size_t`, which is not enough.

## 3 Using `cbmconvert`

Starting with version 2.0.0, `cbmconvert` consists of three programs:

`cbmconvert`
  The main program that makes most conversions.
`disk2zip`
  ZipCode creator. Encodes a 35-track 1541 disk image to four ZipCode files (`1!filename`..`4!filename`).
`zip2disk`
  ZipCode dissolver. Decodes a 35-track 1541 disk image from four ZipCode files.

### 3.1 `cbmconvert`

Since version 2.0, `cbmconvert` has independent file reader and writer modules that are controlled through a main program. There is only a command-line interface, but adding an interactive user interface should be easy (any volunteers?).

`cbmconvert` reads all files in all input files you specify on the command line and writes them in the format you specify. If you want to copy only some files from a set of archive files to an archive file or disk image, you will have to extract all the files to a single-file format such as the PC64 format, and to copy the wanted individual files to the output archive with another invocation of `cbmconvert`.

You can get a list of `cbmconvert` options by invoking the program without any arguments. `cbmconvert` supports the following file formats:

Raw files
  Files with just the raw data. Written with the `-I` and `-N` options, read with the `-n` option.
PC64 files
  Also known as "P00" files. Written with the `-P` option, read with the `-p` option.
Lynx archives
  Written with `-L`, read with `-l`.
Commodore C2N tape archives
  Written with `-C`, read with `-c`. These files are raw dumps of the data format the Commodore KERNAL routines maintain. Block checksums and countdown leaders (0x89..0x81 for the first copy and 9..1 for the second) are omitted, and the blocks are not stored twice, but only once. The data consists of 192-byte tape header blocks, 192-byte data file blocks, and arbitrary-length program file blocks.
C128 CP/M disk images
  Written with `-M[478]`, read with `-m`.
CBM DOS disk images
  Written with `-D[478]`, read with `-d`. 1571 and 1581 support have not been tested properly, and not all 1581 features have been implemented.
ARC/SDA
  Read with `-a`. No write support.
Arkive
  Read with `-k`. No write support.
T64
  Read with `-t`. No write support.

#### 3.1.1 CP/M disk images

Since `cbmconvert` has been written exclusively with Commodore file names in mind, the support for CP/M disk images is a bit clumsy. You may want to rename the files after extracting them from the disks. For instance, if you have downloaded the CP/M disk images from http://www.funet.fi/pub/cbm/demodisks/c128/, you can use the following commands in `bash` to extract the files from the disk images:

    gunzip *.d64.gz

    for i in *.d64; do mkdir ${i%.d64}; cd ${i%.d64}; cbmconvert -m ../$i; cd ..; done

    for i in */*.prg; do mv $i "`echo ${i%.prg}|tr A-Z a-z`"; done

CP/M does not store the exact length of the files anywhere. That is why many CP/M programs create files whose length is a multiple of 128 bytes. Text files in CP/M commonly use `^Z` as an end-of-file delimiter, and many programs fill the extra bytes in the last 128-byte block of the file with this character. The CP/M routines in `cbmconvert` does so as well, and it removes the `^Z` padding when reading the files. So, you could use `cbmconvert` to remove the padding. But you may prefer the following Perl 5 command:

    perl -i -e 'undef $/; while (<>) { s/\032+\z//gs; print }' *

#### 3.1.2 GEOS disk images

GEOS files on disk images will be handled transparently by converting them to and from the Convert format on the fly. All unused areas in the Convert file will be zeroed out, and all possible integrity checks will be made in both conversions. If the file is interpreted to be in GEOS or Convert format, and it fails in any integrity check, it will be treated as a native Commodore file.

You should note that Convert does not provide much safety for the file. For sequential files, the original file length is not stored anywhere. For VLIR files, it is, but even then there are no checksums.

As a speciality, all GEOS files have a file information block. The contents of this block can be viewed with the simple Perl script `cvtinfo.pl` that reads files in Convert format. It was just a quick hack; feel free to make it more robust.

The GEOS routines have been tested with twelve 1541 disk images and with several megabytes of `*.cvt` files, and they are believed to be error free now. There were problems with the VLIR sector format; Mr. Boyce's reverse engineered information about it was incorrect, or at least inadequate.

#### 3.1.3 Commodore C2N tape archives

The Commodore C2N tape format uses 187-byte file names that are padded with spaces. On disks, file names are 16 bytes and padded with shifted spaces. Since `cbmconvert` was designed primarily for disk-based archives, it truncates tape file names to 16 bytes and changes the padding from regular spaces to shifted spaces and vice versa.

End-of-tape markers on C2N tapes are not converted.

There are two kinds of program files on tapes: relocatable program files (type code 1, typically written in BASIC) and absolute program files (type code 3, typically written in machine language). There is no such distinction among disk files. When writing program files to C2N archives, `cbmconvert` marks the files relocatable if the least significant byte of their start address is 1. This typically indicates a BASIC program.

On output, all file types expect PRG are converted to tape data files consisting of 191-byte fixed-length data blocks. The output to tape data blocks tries to mimic the Commodore KERNAL behaviour: a zero byte is written to the last data block after end-of-file, and the bytes following the zero byte are identical with the bytes in the preceding data or header block.

The data file padding in the last block is not detected or removed on input. Also, the padding is not added on output if the file length is an exact multiple of 191 bytes, even though the KERNAL routines would add an extra block whose first data byte is zero. This prevents data files from growing when they are imported from tape and exported again, but this may cause incompatibilities when writing a data file with exactly n·191 bytes of payload.

#### 3.1.4 Flaws of `cbmconvert`

The conversion may lose some file name or attribute information. In some cases, the filenames will be converted from PETSCII to ASCII or vice versa, and some PETSCII or ASCII specific characters will be lost in the conversions. Non-standard directory information available in some formats will be lost as well.

Lynx, Arkive and ARC allow duplicate filenames. By default, `cbmconvert` detects them and refuses to write multiple files with identical names to archives or to disk images. These duplicates are often used as directory separators, to separate groups of filenames.

By default, `cbmconvert` stops when it notices the first duplicated file name. If you specify the option `-o1`, files with duplicate names will be skipped (only the first such file will be copied). If you specify the option `-o2`, files with duplicate file names can be created. For writing disk images, the corresponding options are `-D4d` or `-M4d` and so on. The native file system output `-N` does not allow duplicate file names.

Arkive also works on illegal file types that cannot be created without directly modifying the directory blocks. Such files will be converted to a supported file type. Files with unknown types on disk images will be skipped.

Handling the errors on disk images could be more robust. Not all (typically BAM-related) errors on CBM DOS disk images are detected. For erroneous 1541 disk images that do not work with the new `cbmconvert`, you may want to try the `disk2files` program of the previous release.

### 3.2 `disk2zip`

This program converts a standard 35-track, 683-block 1541 disk image to ZipCode format. Because the ZipCode format stores the disk identifier, this conversion adds information. You can use the `-i` option to specify the disk identifier.

### 3.3 `zip2disk`

ZipCoded archives consist of four files prefixed with a number from 1 to 4, and an exclamation point. To extract an archive having the files `1!disk`, `2!disk`, `3!disk` and `4!disk`, type `zip2disk disk`. You may then delete the original files with `rm [1-4]\!disk` or similar. But note that some software (demos and copy-protected software) may rely on the disk identifier, which will not be stored in the 1541 disk image file.

Support for ZipCode SixPacks (`1!!disk`, `2!!disk`, ..., `6!!disk`) could be added easily, but this format is used mainly for really custom 1541 disks with abnormal low-level format, which `cbmconvert` does not support.

## 4 Credits

Most of the `cbmconvert` package was written by Marko Mäkelä. The original version appeared in 1993, and a complete rewrite was started in 1997 and finished in 1998.

The zip2disk module was originally programmed by Paul David Doherty. The ISO 9660 compliant filename truncation algorithm was taken from the T64TOP00 utility created by Wolfgang Lorenz as a part of his now discontinued PC64 emulator.

Support for 1581 CBM DOS disk images was programmed by Pasi Ojala.

The ARC/SDA dissolving code was originally written by Chris Smeets.

The necessary information for supporting GEOS was provided by Joe Forster/STA, Doug Cotton, Alexander Boyce (author of the unofficial GEOS Programmer's Reference Guide) and by the anonymous author of the file "convertfrmt.txt".

## 5 Distribution

The source code repository of `cbmconvert` is available at https://github.com/dr-m/cbmconvert/.

### 5.1 Revision History

Version 2.1.6:
  Fixed compatibility with the 2023 version of the C standard. Implemented `cbmconvert` options `-o1`, `-o2` and `-D4d` for more options regarding duplicate file names; by default (`-o0`), copying is stopped and an error returned.
Version 2.1.5:
  Bug fixes, and regression tests for everything except the read-only formats `-a`, `-k`, `-t`
Version 2.1.4:
  Bug fixes; public source code repository
Version 2.1.3:
  Bug fixes
Version 2.1.2:
  Bug fixes
Version 2.1.1:
  Bug fixes
Version 2.1:
  Added support for raw Commodore C2N tape archives. Renamed the CP/M-related options to `-m` and `-M`, so that the C2N-related options can be `-c` and `-C`.
Version 2.0.5:
  Mikko Suonio provided patches for tolerating truncated T64 and Lynx archives. His patches also handle single-file T64 archives that have number of entries set to zero, and Lynx archives that have file names starting with a space.
Version 2.0.5.1:
  It was brought to my attention that Lynx 17 on the Commodore 64 and some other utilities do not recognize the Lynx archives created by `cbmconvert`. I slightly modified the magic cookie to fix the problem.