Guide

From an unread file on disk to typed records, and back out again.

Installation

go get example.invalid/kelmscott@v2.4.1

The library has no dependencies outside the standard library, and it will not acquire any. That is a compatibility promise, not a preference: these files are read by programs that are rebuilt once every few years, on machines that are not allowed to reach a package proxy.

Declaring a layout

A layout is an ordered list of fields. Widths are in bytes, not runes — fixed-width formats are defined by byte positions, and a multi-byte character in a name field does not move the field after it.

layout := kelmscott.Layout{
    {Name: "id",      Width: 8,  Type: kelmscott.Integer, Pad: kelmscott.ZeroLeft},
    {Name: "station", Width: 12, Type: kelmscott.Text},
    {Name: "taken",   Width: 14, Type: kelmscott.Timestamp, Format: "20060102150405"},
    {Name: "reading", Width: 10, Type: kelmscott.Decimal, Scale: 3},
    {Name: "flags",   Width: 4,  Type: kelmscott.Text, Optional: true},
}

The sum of the widths is the record length. If the file declares a record length in its header and the two disagree, the reader stops at the header rather than producing 200 million records that are each shifted by one byte.

Reading a file

f, err := os.Open("readings-202608.dat")
if err != nil {
    return err
}
defer f.Close()

r := kelmscott.NewReader(f, layout)
r.Encoding = kelmscott.Latin1

for r.Next() {
    rec := r.Record()
    id, _ := rec.Int("id")
    val, _ := rec.Decimal("reading")
    fmt.Println(id, val)
}
return r.Err()

Next reports whether another record was decoded. It returns false both at the end of the file and on the first error, which is why Err has to be checked after the loop and not inside it. Record is only valid until the next call to Next; the reader reuses its buffer, and a record kept across the loop boundary will change under you. Call rec.Clone() if you need to keep one.

Writing a file

w := kelmscott.NewWriter(out, layout)
w.Encoding = kelmscott.Latin1

for _, m := range measurements {
    if err := w.Write(kelmscott.Values{
        "id":      m.ID,
        "station": m.Station,
        "taken":   m.Taken,
        "reading": m.Value,
    }); err != nil {
        return err
    }
}
return w.Close()

A value that does not fit its declared width is an error, not a truncation. Truncating a station code silently is how two stations become one in the downstream report, and there is no way to find out afterwards which records were affected.

Close flushes and must be checked. A writer that is only deferred and never checked will happily lose the last buffered block.

Handling errors

Every error returned by the reader wraps a *kelmscott.PositionError, which carries the byte offset, the one-based record number and the field name.

var pe *kelmscott.PositionError
if errors.As(r.Err(), &pe) {
    log.Printf("record %d, field %q, offset %d: %v",
        pe.Record, pe.Field, pe.Offset, pe.Unwrap())
}

Recovery, where recovery makes sense, is a decision for the caller. Set r.OnError to a function returning kelmscott.Skip and the reader will drop the offending record and continue; return kelmscott.Stop and it behaves as it does by default. There is deliberately no "repair" mode.

Performance

On a 2.6 GHz core the reader sustains roughly 240 MB/s for a nine-field layout of 48-byte records, which is faster than any spinning disk these files usually live on. The two things that cost real time are per-record allocation, which the reader avoids by reusing its buffer, and charset conversion, which it avoids for ASCII and UTF-8 by not doing any.