Discussion on External Signal Reception of FMZ Platform: Extended API VS Strategy Built-in HTTP Service

Preface

There are several articles in the platform “Digest” about connecting to Trading View webhooks, which allows strategies to drive tradings with signals from external systems. At that time, the platform did not support the built-in http service function of the JavaScript language. The platform’s extended API interface was used: CommandRobot, in simple terms, the http/https request of the external signal is sent to the FMZ platform, and the platform forwards the signal as a strategy interaction message to the strategy program.

As the platform develops and iterates, many new features have been updated and upgraded. There are also new solutions for receiving external signals. Each solution has its own advantages. In this article, we will discuss this topic together.

Use FMZ Platform to Expand API Interface

The advantages of using this method to connect to external systems are that it is relatively simple, highly secure, and relies on the stability of the platform’s extended API interface.

The process of receiving external signals:

External system (Trading View webhook) –> FMZ extended API service –> Strategy live trading

1. External system (Trading View webhook): For example, the PINE script running on Trading View can set an alarm, and when triggered, it will send an http request to the set webhook url address as a signal.
2. FMZ extended API service: After accessing the interface successfully, the platform forwards the information and sends it to the strategy live trading as an interactive message.
3. Strategy live trading: In the strategy live trading, you can design the GetCommand function to listen to the interactive message, and execute the specified operation after detecting the message.

Compared with using the built-in Http service to create a service to receive signals directly, there is an extra step in the middle (platform transfer).

Strategy Built-in Http Service

After the platform supports the built-in Http service function of the JavaScript language, you can create a concurrent service to listen to external signals directly. The advantages are: the created Http service is a separate thread and does not affect the main function logic. It can listen to messages like the GetCommand function and listen to external signals directly. Compared with the use of the extended API solution, it eliminates the need for transit.

The process of receiving external signals:

External system (Trading View webhook) –> Strategy live trading

1. External system (Trading View webhook): For example, the PINE script running on Trading View can set an alarm, which will send an http request to the set webhook url address as a signal when triggered.
2. Strategy live trading: The strategy runs a Http service concurrently to receive external signals directly.

This solution saves a step, but in order to improve security, it is better to configure https service, which requires some effort. It is a bit more troublesome than using the extended API solution.

Test Code

Test two solutions. The following strategy will send 10 Http/Https requests in parallel in each cycle to simulate external signals. Then the strategy monitors “interaction messages” and “messages pushed by Http service threads”. Then the strategy program matches external signal messages and received signals one by one, detects whether there is signal loss, and calculates the time consumption.

var httpUrl = "http://123.123.123.123:8088/CommandRobot"
var accessKey = ""
var secretKey = ""

function serverFunc(ctx) {
    var path = ctx.path()
    if (path == "/CommandRobot") {
        var body = ctx.body()
        threading.mainThread().postMessage(body)
        ctx.write("OK")
        // 200
    } else {
        ctx.setStatus(404)
    }
}

function createMsgTester(accessKey, secretKey, httpUrl) {
    var tester = {}
    
    tester.currentRobotId = _G()
    tester.arrSendMsgByAPI = []
    tester.arrSendMsgByHttp = []
    tester.arrEchoMsgByAPI = []
    tester.arrEchoMsgByHttp = []
    tester.idByAPI = 0
    tester.idByHttp = 0

    var sendMsgByAPI = function(msgByAPI, robotId, accessKey, secretKey) {
        var headers = {
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36",
            "Content-Type": "application/json"
        }
        HttpQuery(`https://www.fmz.com/api/v1?access_key=${accessKey}&secret_key=${secretKey}&method=CommandRobot&args=[${robotId},+""]`, {"method": "POST", "body": JSON.stringify(msgByAPI), "headers": headers})
    }

    var sendMsgByHttp = function(msgByHttp, httpUrl) {
        HttpQuery(httpUrl, {"method": "POST", "body": JSON.stringify(msgByHttp)})
    }

    tester.run = function() {
        var robotId = tester.currentRobotId

        for (var i = 0; i < 10; i++) {
            var msgByAPI = {"ts": new Date().getTime(), "id": tester.idByAPI, "way": "ByAPI"}            
            tester.arrSendMsgByAPI.push(msgByAPI)
            tester.idByAPI++
            threading.Thread(sendMsgByAPI, msgByAPI, robotId, accessKey, secretKey)

            var msgByHttp = {"ts": new Date().getTime(), "id": tester.idByHttp, "way": "ByHttp"}
            tester.arrSendMsgByHttp.push(msgByHttp)
            tester.idByHttp++
            threading.Thread(sendMsgByHttp, msgByHttp, httpUrl)
        }
    }

    tester.getEcho =function(msg) {
        if (msg["way"] == "ByAPI") {
            tester.arrEchoMsgByAPI.push(msg)
        } else {
            tester.arrEchoMsgByHttp.push(msg)
        }
    }

    tester.deal = function() {
        var tbls = []
        for (var pair of [[tester.arrEchoMsgByHttp, tester.arrSendMsgByHttp, "ByHttp"], [tester.arrEchoMsgByAPI, tester.arrSendMsgByAPI, "ByAPI"]]) {
            var receivedMessages = pair[0]
            var sentMessages = pair[1]
            var testType = pair[2]

            var receivedMap = new Map()
            receivedMessages.forEach(message => {
                receivedMap.set(message["id"], message)
            })
            
            var matchedPairs = []
            var timeDifferences = []
            for (var sentMessage of sentMessages) {
                var receivedMessage = receivedMap.get(sentMessage["id"])
                if (receivedMessage) {
                    matchedPairs.push([JSON.stringify(sentMessage), JSON.stringify(receivedMessage), receivedMessage["ts"] - sentMessage["ts"]])
                    timeDifferences.push(receivedMessage["ts"] - sentMessage["ts"])
                } else {
                    Log("no matched sentMessage:", sentMessage, "#FF0000")
                }
            }
            
            var averageTimeDifference = timeDifferences.reduce((sum, diff) => sum + diff, 0) / timeDifferences.length
            
            var tbl = {
                "type": "table",
                "title": testType + " / averageTimeDifference:" + averageTimeDifference,
                "cols": ["send", "received", "ts diff"],
                "rows": []
            }

            for (var pair of matchedPairs) {
                tbl["rows"].push(pair)
            }

            tbls.push(tbl)
            Log(testType, ", averageTimeDifference:", averageTimeDifference, "ms")
        }

        tester.arrSendMsgByAPI = []
        tester.arrSendMsgByHttp = []
        tester.arrEchoMsgByAPI = []
        tester.arrEchoMsgByHttp = []

        return tbls
    }

    return tester
}

function main() {
    __Serve("http://0.0.0.0:8088", serverFunc)

    var t = createMsgTester(accessKey, secretKey, httpUrl)
    while (true) {
        Log("Test start...", "#FF0000")
        t.run()

        var beginTS = new Date().getTime()
        while (new Date().getTime() - beginTS < 60 * 1000) {
            var cmd = GetCommand()
            if (cmd) {
                try {
                    var obj = JSON.parse(cmd)
                    obj["ts"] = new Date().getTime()
                    t.getEcho(obj)
                } catch (e) {
                    Log(e)
                }
            }
            
            var msg = threading.mainThread().peekMessage(-1)
            if (msg) {
                try {
                    var obj = JSON.parse(msg)
                    obj["ts"] = new Date().getTime()
                    t.getEcho(obj)                
                } catch (e) {
                    Log(e)
                }
            }
        }
        Log("Waiting is over...", "#FF0000")
                
        var tbls = t.deal()
        LogStatus(_D(), "\n`" + JSON.stringify(tbls) + "`")
        Sleep(20000)
    }
}

If testing, you need to fill in the specific server IP address and the extended API KEY of the FMZ platform.

var httpUrl = "http://123.123.123.123:8088/CommandRobot"
var accessKey = "xxx"
var secretKey = "xxx"

1. The serverFunc function creates a concurrent Http service to monitor external signals. For external messages received by the extended API interface, the GetCommand function is used to monitor.

• Messages pushed by the Http service thread: monitored by var msg = threading.mainThread().peekMessage(-1).

• Interaction messages forwarded by the extended API interface: monitored by var cmd = GetCommand().

2. The signal sending and receiving processes are non-blocking. The platform optimizes the underlying multi-threaded resource recovery mechanism. For concurrent functions like Thread or exchange.Go, there is no need to explicitly wait for concurrent tasks to complete (such as join functions, wait functions, etc.). The underlying system will handle resource recovery automatically (requires the latest version of the docker to support this).

    // Extract code snippet, send signal
    tester.run = function() {
        var robotId = tester.currentRobotId

        for (var i = 0; i < 10; i++) {
            var msgByAPI = {"ts": new Date().getTime(), "id": tester.idByAPI, "way": "ByAPI"}            
            tester.arrSendMsgByAPI.push(msgByAPI)
            tester.idByAPI++
            threading.Thread(sendMsgByAPI, msgByAPI, robotId, accessKey, secretKey)   // Concurrent calls, non-blocking

            var msgByHttp = {"ts": new Date().getTime(), "id": tester.idByHttp, "way": "ByHttp"}
            tester.arrSendMsgByHttp.push(msgByHttp)
            tester.idByHttp++
            threading.Thread(sendMsgByHttp, msgByHttp, httpUrl)                       // Concurrent calls, non-blocking
        }
    }

    // Extract code snippet, receiving signal
    var cmd = GetCommand()                              // Listen for messages from the extension API, non-blocking
    var msg = threading.mainThread().peekMessage(-1)    // Listen for messages from self-built Http service, using parameter -1, non-blocking

Next, let’s look at the test process, where the information is annotated directly in the code:

function main() {
    __Serve("http://0.0.0.0:8088", serverFunc)      // Create a concurrent http service in the current strategy instance

    var t = createMsgTester(accessKey, secretKey, httpUrl)   // Create an object for test management
    while (true) {                                           // The strategy main loop starts
        Log("Test start...", "#FF0000")
        t.run()                                              // At the beginning of each loop, the run function of the test management object is called in two ways (1. Sending signals through the extended API, 2. Sending signals directly to the Http service created by the current strategy), each way sends 10 requests concurrently

        var beginTS = new Date().getTime()
        while (new Date().getTime() - beginTS < 60 * 1000) {   // Loop detection of interactive messages from extended APIs and messages from self-built Http services
            var cmd = GetCommand()
            if (cmd) {
                try {
                    var obj = JSON.parse(cmd)
                    obj["ts"] = new Date().getTime()        // Detect interactive messages, record messages, and update time to the time received
                    t.getEcho(obj)                          // Record to the corresponding array
                } catch (e) {
                    Log(e)
                }
            }
            
            var msg = threading.mainThread().peekMessage(-1)
            if (msg) {
                try {
                    var obj = JSON.parse(msg)
                    obj["ts"] = new Date().getTime()        // Detects the message received by the self-built Http service, and the update time is the receiving time
                    t.getEcho(obj)                          // ...
                } catch (e) {
                    Log(e)
                }
            }
        }
        Log("Waiting is over...", "#FF0000")
                
        var tbls = t.deal()                                  // Pair according to the recorded messages and check if there is any unpaired message. If there is, it means the signal is lost.
        LogStatus(_D(), "\n`" + JSON.stringify(tbls) + "`")
        Sleep(20000)
    }
}

Test Results

After a period of testing, it can be observed that the Http method takes a little less time on average than the API method.

The strategy has a built-in Http service to receive signals. This test method is not very rigorous, and the request should come from the outside. For the sake of simple understanding, this factor can be ignored. For the two methods of signal acquisition, the strategy’s built-in Http service reduces one link after all, and should respond faster. For signal stability, it is more important that the signal cannot be lost or missed. The test results show that the extended API of the FMZ platform is also stable, and no signal loss was observed in the test, but it cannot be ruled out that factors such as the network may cause signal problems. Using the built-in Http service to receive external signals directly is also a better solution.

This article is just a starting point, welcome to discuss, thank you for reading.