note: this is incomplete documentation and is more of a brain dump whilst herald is being developed...
Processes
Herald tags samples and experiments with processes. Processes included "sequencing", "basecalling", "pipeline X" etc.
Adding a process
note: the UI will only be updated if the process is for samples (I've left the experiment processes harcoded in the UI for now)
- add a process definition in
processes.go
This file is located in the source code (src/data/processes.go
). An example function call is:
createProcessDefinition("mypipeline", []string{"sequence", "basecall"}, false, true)
This will create a process called mypipeline
which depends on the processes sequence
and basecall
being complete.
Message passing
dependencies
Herald uses Google Protocol Buffers v3 for serialisation and gRPC for message passing. Current versions:
- Protobuf 3.6.1
- gRPC 1.16.1
protocol buffers
The protocol buffer definitions are stored under protobuf/*.proto
.
updating the schema
If editing the *.proto
files, the Go bindings will need to be re-compiled. For example, the bindings for the sample message:
protoc -I=protobuf --go_out=plugins=grpc:src/ protobuf/*.proto
go generate
sh build-osx.sh
To update the MinKNOW api:
protoc --go_out=plugins=grpc:src/ont_rpc -I protobuf/minknow/rpc protobuf/minknow/rpc/rpc_options.proto
Database
Sample records are stored via bitcask db, which is currently hardcoded to live in /tmp/db
.
Sample records are stored in a keyvalue store, where the label is the key and the sample protobuf message is the value.