Custom Transaction Processor not receiving request

Question

Why is my transaction processor not receiving the request I post via the rest API?

I have built a client and Transaction Processor (TP), in Golang, which is not much different to the XO example. I have successfully got the TP running locally to the Sawtooth components, and sending batch lists from a separate cli tool. Currently the apply method in the TP is not being hit and does not receive any of my transactions.

EDIT: In order to simplify and clarify my problem as much as possible, I have abandoned my original source code, and built a simpler client that sends a transaction for the XO sdk example.*

When I run the tool that I have built, the rest api successfully receives the request, processes and returns a 202 response but appears to omit the id of the batch from the batch statuses url. Inspecting the logs it appears as though the validator never receives the request from the rest api, as illustrated in the logs below.

sawtooth-rest-api-default | [2018-05-16 09:16:38.861 DEBUG    route_handlers] Sending CLIENT_BATCH_SUBMIT_REQUEST request to validator
sawtooth-rest-api-default | [2018-05-16 09:16:38.863 DEBUG    route_handlers] Received CLIENT_BATCH_SUBMIT_RESPONSE response from validator with status OK
sawtooth-rest-api-default | [2018-05-16 09:16:38.863 INFO     helpers] POST /batches HTTP/1.1: 202 status, 213 size, in 0.002275 s

My entire command line tool that sends transactions to a local instance is below.

package main

import (
    "bytes"
    "crypto/sha512"
    "encoding/base64"
    "encoding/hex"
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "math/rand"
    "net/http"
    "strings"
    "time"

    "github.com/hyperledger/sawtooth-sdk-go/protobuf/batch_pb2"
    "github.com/hyperledger/sawtooth-sdk-go/protobuf/transaction_pb2"
    "github.com/hyperledger/sawtooth-sdk-go/signing"
)

var restAPI string

func main() {
    var hostname, port string

    flag.StringVar(&hostname, "hostname", "localhost", "The hostname to host the application on (default: localhost).")
    flag.StringVar(&port, "port", "8080", "The port to listen on for connection (default: 8080)")
    flag.StringVar(&restAPI, "restAPI", "http://localhost:8008", "The address of the sawtooth REST API")

    flag.Parse()

    s := time.Now()
    ctx := signing.CreateContext("secp256k1")
    key := ctx.NewRandomPrivateKey()
    snr := signing.NewCryptoFactory(ctx).NewSigner(key)

    payload := "testing_new,create,"
    encoded := base64.StdEncoding.EncodeToString([]byte(payload))

    trn := BuildTransaction(
        "testing_new",
        encoded,
        "xo",
        "1.0",
        snr)

    trn.Payload = []byte(encoded)

    batchList := &batch_pb2.BatchList{
        Batches: []*batch_pb2.Batch{
            BuildBatch(
                []*transaction_pb2.Transaction{trn},
                snr),
        },
    }

    serialised := batchList.String()

    fmt.Println(serialised)

    resp, err := http.Post(
        restAPI+"/batches",
        "application/octet-stream",
        bytes.NewReader([]byte(serialised)),
    )

    if err != nil {
        fmt.Println("Error")
        fmt.Println(err.Error())
        return
    }

    defer resp.Body.Close()
    fmt.Println(resp.Status)
    body, err := ioutil.ReadAll(resp.Body)
    fmt.Println(string(body))
    elapsed := time.Since(s)
    log.Printf("Creation took %s", elapsed)

    resp.Close = true
}

// BuildTransaction will build a transaction based on the information provided
func BuildTransaction(ID, payload, familyName, familyVersion string, snr *signing.Signer) *transaction_pb2.Transaction {
    publicKeyHex := snr.GetPublicKey().AsHex()
    payloadHash := Hexdigest(string(payload))

    addr := Hexdigest(familyName)[:6] + Hexdigest(ID)[:64]

    transactionHeader := &transaction_pb2.TransactionHeader{
        FamilyName:       familyName,
        FamilyVersion:    familyVersion,
        SignerPublicKey:  publicKeyHex,
        BatcherPublicKey: publicKeyHex,
        Inputs:           []string{addr},
        Outputs:          []string{addr},
        Dependencies:     []string{},
        PayloadSha512:    payloadHash,
        Nonce:            GenerateNonce(),
    }

    header := transactionHeader.String()
    headerBytes := []byte(header)
    headerSig := hex.EncodeToString(snr.Sign(headerBytes))

    return &transaction_pb2.Transaction{
        Header:          headerBytes,
        HeaderSignature: headerSig[:64],
        Payload:         []byte(payload),
    }
}

// BuildBatch will build a batch using the provided transactions
func BuildBatch(trans []*transaction_pb2.Transaction, snr *signing.Signer) *batch_pb2.Batch {

    ids := []string{}

    for _, t := range trans {
        ids = append(ids, t.HeaderSignature)
    }

    batchHeader := &batch_pb2.BatchHeader{
        SignerPublicKey: snr.GetPublicKey().AsHex(),
        TransactionIds:  ids,
    }

    return &batch_pb2.Batch{
        Header:          []byte(batchHeader.String()),
        HeaderSignature: hex.EncodeToString(snr.Sign([]byte(batchHeader.String())))[:64],
        Transactions:    trans,
    }
}

// Hexdigest will hash the string and return the result as hex
func Hexdigest(str string) string {
    hash := sha512.New()
    hash.Write([]byte(str))
    hashBytes := hash.Sum(nil)
    return strings.ToLower(hex.EncodeToString(hashBytes))
}

// GenerateNonce will generate a random string to use
func GenerateNonce() string {
    return randStringBytesMaskImprSrc(16)
}

const (
    letterBytes   = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
    letterIdxBits = 6                    // 6 bits to represent a letter index
    letterIdxMask = 1<<letterIdxBits - 1 // All 1-bits, as many as letterIdxBits
    letterIdxMax  = 63 / letterIdxBits   // # of letter indices fitting in 63 bits
)

func randStringBytesMaskImprSrc(n int) string {
    rand.Seed(time.Now().UnixNano())
    b := make([]byte, n)
    // A rand.Int63() generates 63 random bits, enough for letterIdxMax letters!
    for i, cache, remain := n-1, rand.Int63(), letterIdxMax; i >= 0; {
        if remain == 0 {
            cache, remain = rand.Int63(), letterIdxMax
        }
        if idx := int(cache & letterIdxMask); idx < len(letterBytes) {
            b[i] = letterBytes[idx]
            i--
        }
        cache >>= letterIdxBits
        remain--
    }

    return string(b)
}

Answer 1

There were a number of issues with this, and hopefully I can explain each in isolation to help shed a light on the ways in which these transactions can fail.

Transaction Completeness

As @Frank C. comments above my transaction headers were missing a couple of values. These were addresses and also the nonce.

// Hexdigest will hash the string and return the result as hex
func Hexdigest(str string) string {
    hash := sha512.New()
    hash.Write([]byte(str))
    hashBytes := hash.Sum(nil)
    return strings.ToLower(hex.EncodeToString(hashBytes))
}


addr := Hexdigest(familyName)[:6] + Hexdigest(ID)[:64]
transactionHeader := &transaction_pb2.TransactionHeader{
    FamilyName:       familyName,
    FamilyVersion:    familyVersion,
    SignerPublicKey:  publicKeyHex,
    BatcherPublicKey: publicKeyHex,
    Inputs:           []string{addr},
    Outputs:          []string{addr},
    Dependencies:     []string{},
    PayloadSha512:    payloadHash,
    Nonce:            uuid.NewV4(),
} 

Tracing

The next thing was to enable tracing in the batch.

return &batch_pb2.Batch{
    Header:          []byte(batchHeader.String()),
    HeaderSignature: batchHeaderSignature,
    Transactions:    trans,
    Trace: true, // Set this flag to true
}

With the above set the Rest API will decode the message to print additional logging information, and also the Validator component will output more useful logging.

400 Bad Request
{
"error": {
"code": 35,
"message": "The protobuf BatchList you submitted was malformed and could not be read.",
"title": "Protobuf Not Decodable"
}
}

The above was output by the Rest API once the tracing was turned on. This proved that there was something awry with the received data.

Why is this the case?
Following some valuable advice from the Sawtooth chat rooms, I tried to deserisalise my batches using the SDK for another language.

Deserialising

To test deserialising the batches in another SDK, I built a web api in python that I could easily send my batches to, which could attempt to deserialise them.

from flask import Flask, request

from protobuf import batch_pb2

app = Flask(__name__)

@app.route("/batches", methods = [ 'POST' ])
def deserialise():
    received = request.data
    print(received)
    print("\n")
    print(''.join('{:02x}'.format(x) for x in received))

    batchlist = batch_pb2.BatchList()

    batchlist.ParseFromString(received)
    return ""

if __name__ == '__main__':
    app.run(host="0.0.0.0", debug=True)

After sending my batch to this I received the below error.

RuntimeWarning: Unexpected end-group tag: Not all data was converted

This was obviously what was going wrong with my batches, but seeing as this was all being handled by the Hyperledger Sawtooth Go SDK, I decided to move to Python and built my application with that.

Answer 2

[Edit]

Managed to get document updated with information on writing a client in Go https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide/go_sdk.html

[Original Answer]

Answering to the question a bit late, hope this will help others who are facing similar issues with Go client.

I was recently trying out a sample Go client for Sawtooth. Faced similar issues as you have asked here, currently it's hard to debug what has gone wrong in the composed batch list. Problem is lack of sample code and documentation for using Go SDK in client application development.

Here's a link for the sample Go code that's working: https://github.com/arsulegai/contentprotection/tree/master/ContentProtectionGoClient

Please have a look at the file src/client/client.go which composes single Transaction and Batch, puts it to batch list and sends it to the validator. Method I followed for debugging is to compose the expected batch list (for specific transaction) in another language and comparing each step with the result from equivalent step in Go code.

Coming to the question, along with missing information in the composed transaction header other problem could be the way protobuf messages are serialized. Please use protobuf library for serializing.

Example: (extending the answer from @danielcooperxyz)

transactionHeader := &transaction_pb2.TransactionHeader{
    FamilyName:       familyName,
    FamilyVersion:    familyVersion,
    SignerPublicKey:  publicKeyHex,
    BatcherPublicKey: publicKeyHex,
    Inputs:           []string{addr},
    Outputs:          []string{addr},
    Dependencies:     []string{},
    PayloadSha512:    payloadHash,
    Nonce:            uuid.NewV4(),
}
transactionHeaderSerializedForm, _ := proto.Marshal(transactionHeader)

(protobuf library methods can be found in github.com/golang/protobuf/proto)