How to run your own Kinesis Watcher Node

This document is going to show you how to set up your own "watcher node."

As a refresher:

What is a Blockchain?​

At its core, a blockchain is a digital ledger or record-keeping system. But instead of being stored in one place like a traditional database, this ledger is distributed across multiple computers, known as nodes, around the world. Each of these nodes has a complete or partial copy of the blockchain, and they work together to maintain and update this ledger.

Imagine a blockchain as a series of digital "blocks" that contain lists of transactions. Each time a transaction is made, it's verified by the nodes in the network and then added to a new block. This new block is linked to the previous one, forming a chain of blocks—hence the name "blockchain."

The unique thing about blockchain is its decentralization. Unlike conventional databases managed by a central authority, like a bank or government, a blockchain is maintained collectively by the nodes. This makes the system more transparent and secure because no single entity has complete control over the entire blockchain. It's also more difficult to tamper with: to change one block, you'd need to change every block that comes after it, which would require the consensus of the majority of nodes—a practically impossible task.

One more crucial feature is that once a block has been added to the blockchain, it can't be changed or removed. This makes the blockchain a tamper-proof, "immutable" record of all transactions that have ever occurred on that network.

In summary, a blockchain is a decentralized, transparent, and immutable digital ledger, maintained by a network of nodes. It serves as the foundational technology for various applications, from cryptocurrencies like Bitcoin and Kinesis to more complex systems for supply chain management, digital voting, and beyond.

What is a Watcher Node?

Kinesis is based on the Stellar blockchain, and under that system there are a set of trusted nodes that vote on consensus. Without getting lost in the weeds: every trusted node in the system looks at each transaction that is presented, offers its opinion on whether a given transaction is valid, and if the correct majority of nodes think a transaction is valid then it is queued for inclusion in the next block. These blocks are created about once every 5 seconds.

When you are running a Watcher Node, you are maintaining a copy of the blockchain that is up-to-date at all times, but your node is not allowed to vote on the legitimacy of the transactions. It's just keeping a local copy for you, or for those you choose to share it with.

What will this look like?

You can visit https://watcher.aboutkinesis.com/ to see my node if you'd like, but it looks like this once you've entered all the details:



On my node, or on your own, this will produce an error until you point it at your local copy of the blockchain. So where it says "Set Custom Network" on the upper right you'll need enter your local URL (probably http://localhost:8000 for KAU or https://localhost:8001 for KAG), though on my node you can enter https://kau.aboutkinesis.com or https://kag.aboutkinesis.com in order to access those blockchains.

This is the Stellar Explorer that we will be installing at the end of this tutorial. It's generic in that Kinesis doesn't make it, but it understands Stellar-based blockchains like the Kinesis blockchain. If you dig through transactions you'll see references to lumens and stroops instead of KAU and KAG, but that's because the software is generic and designed for Stellar and doesn't know what KAU and KAG are - it just sees Stellar. It works well enough for our needs, and it allows us to export all blockchain transactions into Excel for further analysis which can be useful for reconciling the blockchain against the audits if you choose to do so for your own due diligence.

If you want to use an explorer that's customized for the Kinesis blockchain, the official explorer is a great place to start: https://explorer.kinesis.money/

The big advantage of using a separate explorer is that it enhances legitimacy. If Kinesis were up to no good, they could conceivably modify their explorer to hide their misdeeds. Using a third party tool that's open source that we know just reports on the blockchain removes that potential risk.

Required Skills

You need to be familiar with the command line. I will be illustrating this process using the Ubuntu command line, but you don't really need to use Linux at all. You do need to be somewhat comfortable with working from a command line however.

You also need come familiarity with Docker and Git.

Required Hardware

I run this as its own virtual machine, but you can run this locally on your Windows or Mac computer if you like using Docker. Just make sure you've got enough disk space to to hold the blockchain data - I know for sure that 100 gigs is not enough, but for now 200 gigs might be plenty. The node I'll be building here has a couple of terabytes available, but I will extend that if the data requires as Indonesia takes off.

Required Software

Git

Every blockchain node needs to be running the same software, and git is central repository that maintains this code. One command will get you a copy of the current code, and another one-line command will update the code once an update has been created.

Docker and Docker-Compose

In the olden days someone like me would need to install applications manually. So for a project like this I would need to install a database, and a web server, and all the tools required to manage the code that actually runs the blockchain. Then I would need to configure them correctly, and there would be hundreds of variables that would be different between what I was running and what the developer was running, and diagnosing problems could be difficult.

Nowadays lots of software is "containerized." So for example, the Kinesis Watcher node you'll be building will require a database as part of the core infrastructure, and the docker-compose process just sets one up for you that's been tested with the existing code. When you walk through this process you're getting the same version of the code and required applications that everyone gets. When the project is updated, the apps and containers will be updated automatically.

This is just a much simpler way of installing and managing the system. You just need to learn another tool.

(There's more than this, but this should suffice to explain what we're doing, and why.)

---

With that out of the way, let's get started:

Install Docker

You'll need to choose how to install Docker, which means you probably want to install Docker Desktop if you're installing this on a Windows or Mac. The documentation is here: https://docs.docker.com/engine/install/

I installed docker directly on Linux, which meant I needed to install docker-compose separately. If you installed Docker Desktop then this step isn't necessary; if not then the instructions here are likely useful: https://docs.docker.com/compose/install/

In Ubuntu, a simple

sudo apt install docker-compose

Got things going.

Install Git

Instructions are here: https://github.com/git-guides/install-git

On ubuntu:

sudo apt-get install git-all

Install the Watcher Itself

And now, we can start:

The code for the Kinesis watcher node is located here: https://github.com/bullioncapital/nodes

We will be performing the process as it exists in August of 2023 for this tutorial. What I'm really doing is walking through the steps on that page, as many seem to get confused. If things change in the future, just follow the directions on that page. It's well documented, but the instructions assume a certain level of competence with the command line, git, and docker. The additional value I'm creating here is that I'm walking you through things more slowly, so you don't need the same level of experience to get started.

Again, I'm walking through this using Ubuntu. If you're using another version of Linux you may not need the "sudo" statements as you may just be running as a user with full permissions. If you're running this locally on your Windows or Mac computer this likely isn't required either, but you'll be entering the commands in the Git Bash terminal window (again, without the 'sudo' command).

We will be installing the KAG blockchain in this tutorial. All the responses we see in my terminal will be from the KAG blockchain, but I will try and document both blockchain installations.

I would install one blockchain first, then redo the process for the second blockchain should you choose to run both.

0. Clone the repository



On the git repository, click the green button, then click the copy button on the popup (highlighted) to copy the address to your clipboard.

(Note that when I say "paste" in the remainder of this document, the way you past into a Linux shell or Git Bash is to right click in the window. That's it - right-click is "paste the text in my clipboard wherever the cursor is," and highlighting text and pressing the middle button is how you copy.)

Now, open your shell, and type "git clone " and paste the address after:

git clone https://github.com/bullioncapital/nodes

That will create the codebase for the watcher node in your current directory. If this is the first thing you've installed, then you should see a nodes directory directly below where you ran this command (probably your home directory). In linux, the ls command lists the contents of a directory, and cd changes directories. Entering cd by itself takes you back to your home directory.



I've already installed the stellarexplorer as well, but we'll get to that last.

Now, move into that nodes directory:

cd nodes

These are the contents of the folder that git created for you - this is the code to set up and configure your watcher node. Using the default Ubuntu shell color scheme, green represents scripts you can run, blue represents directories, and white represents other files. So the README.md file has the same directions you see on the github project page, and the docker-compose.yml file will be used by the next command to generate the docker containers we need to run the watcher.

So let's get started.

1. Starting the services

In the command prompt, to bootstrap the KAG node, type the following:

sudo ./setup.sh kag-mainnet 8001

The first time you do this you'll see more output than this - don't worry about that. As long as everything says "done" then you're fine.

If you want to create the KAU watcher instead, then you need to enter kau-mainnet and 8000 on that setup.sh command. For the remainder of the steps below, you will need to enter kau-mainnet instead of kag-mainnet. If you are installing both blockchains, then you'll go through this process once with the KAG network code, then again with the KAU network code.

2. Database initialization

sudo ./db-init.sh kag-mainnet

This will throw out a couple of screens of information, that will finish with something that looks like

Successfully applied 56 integrations

Now we double-check that by typing:

sudo ./exec.sh kag-mainnet db

This gets us into the Database container. Once there we type:

psql -h localhost --username=postgres --command="\l" | grep kag-mainnet

If the database was created correctly, it should be clear from the output:



If you do not see two lines representing Core and Horizon database installations, then the last command failed. Diagnose and fix the problem before continuing.

Now, press control-d or type exit to get out of the database docker container and back into your regular command shell.

exit

Now, if you're not sure where you are, I'd like you to look at the previous screenshot above. Shells try hard to make it clear where you are and who you are (as this affects what powers you have), so in the first line of the screenshot above the shell is telling me:

• I'm logged in as 'derek' on a machine named 'kinesiswatcher'
• My current directory is the nodes directory underneath my home directory (the ~ is the shortcut for /home/derek)
• I currently am logged in as a regular user, and don't have administrative powers. The $ prompt means I'm a user. If I were the account with admin powers (called 'root') that dollar sign would turn into a #

Now, the exec.sh command in that image causes us to enter the database container, and the shell inside that container is telling us the same information:

• I'm now logged in as 'root' on the machine 54e9b48105cf (name created by docker-compose)
• The / means we're in the top directory, but that doesn't really matter for this process. This is equivalent to being at the c:\ folder in Windows.
• The # on the command line means I have root powers. I can do everything on this machine.

The machine names and user names aren't important, but knowing which system you're currently logged in to, and what user you are, is. If you get lost and a command doesn't work as expected, check the command line prompt and make sure you're where you think you are. You may just be talking to the wrong computer.

Conceptually, docker creates separate computers that you will be controlling. We'll be hopping in and out of them throughout this tutorial via the exec.sh script, so if you get an error you likely just lost track of where you are.

Again, the way to exit from one command prompt to another is to type 'exit' or use the shortcut control-d.

3. Catching Up

This step is a bit more complicated. What we need to do here conceptually is as follows:

1. Figure out what the current block on the blockchain is. We will need to check an up-to-date blockchain node with our web browser and make note of this number.
2. Have the Kinesis Core catch up to that block in chunks
3. Configure the Kinesis Horizon server to ingest that data directly from existing nodes
4. Then tell Horizon how much to import. We will be doing the entire blockchain, as this is what's most useful if you want to do things like reconcile the blockchain against the audits. You need all the data.

So, let's start:

For the kag-mainnet, we need to open this page: https://kag-mainnet-arch-syd-node2.kinesisgroup.io/.well-known/stellar-history.json. For KAU, you can click here instead.

This is the number you're looking for:



This number will not be current when you install your node! A new block is created approximately every 5 seconds, and you want the current number. So open the page, and make a note of it.

Now, we will move into the Core docker container again and perform the catchup:

sudo ./exec.sh kag-mainnet core

Now, once you're in the core (as indicated by the change in the prompt), type the following command (replacing 27552127 with the block number you got above):

stellar-core catchup 27552127/512

(Note that that's the current blockchain number, a forward slash, and the number 512.)

In a few seconds the process will finish, and as scary as "Application destructing" and "Application destroyed" may sound, if you see the following results then the process was a success.



Remember to type "exit" or control-d to exit.

Now, this step is a bit more complex. When I do this (I've done this a few times) I tend to type this up in a text document, then copy it and right-click it in a terminal to paste it.

So, enter the Horizon instance:

sudo ./exec.sh kag-mainnet horizon

And once there, enter the following, changing it as is appropriate for your installation:

Base Command from the github documentation

export LEDGER_MAX=<LEDGER_MAX retrieved from HAS currentLedger>
export LEDGER_MIN=$((LEDGER_MAX - 512))
export ENABLE_CAPTIVE_CORE_INGESTION=true
horizon db reingest range $LEDGER_MIN $LEDGER_MAX

That's kind of ugly, right? LEDGER_MAX there is a variable that's replaced with whatever the current block number is that you found above. $LEDGER_MIN is probably 2 if you want the whole blockchain. So in my case this would be:

export LEDGER_MAX=27552127
export LEDGER_MIN=2
export ENABLE_CAPTIVE_CORE_INGESTION=true
horizon db reingest range $LEDGER_MIN $LEDGER_MAX

Now, if you wanted to you could probably just remove the variables and insert the numbers directly:

export ENABLE_CAPTIVE_CORE_INGESTION=true
horizon db reingest range 2 27552127

Whatever you want to do, go ahead and paste that into your terminal window. Then go grab a coffee or go for a run or something.

This is going to run for a long time - hours, maybe more. Be patient. At least it tells you something about where it is in the process.



As a reference, I rebuilt my KAU node yesterday, and I added --parallel-workers 4 to the command to run faster, and it took somewhere around 18 hours to complete. That's on a reasonably fast computer (I only gave that virtual machine 4 cores) and most of the time the unix load was around 11 or 12. It ran fine, but it takes a while, and if you're running it with one worker instead of 4 it will take longer still.

It's fine. Hopefully you only need to do this once.

I'll stop here and wait for the process to continue before resuming in the next post.

 

[Derek]

OK, back from my weekend out of town.



This is what things will look like once the Horizon instance has finished its ingestion. Note that this took days, but if I'd run with the --parrallel-workers command line flag it would have been much faster.

If your computer is set to go to sleep after a certain period of time this will fail. Ensure your computer will stay up through this phase or you'll cause problems.
To verify the process completed successfully, we are going to type the following just as the directions say:

That:

• Exits the Horizon container
• Enters the db container
• And runs a PostgreSQL command to verify intake. Note that you will need to modify the line in the directions to use the kag-mainnet, and insert the max block you've been using through this process, as I did here. The numbers you use will be different when you do this.

4. Going Live​

Now, we're almost done.

Exit the db container by hitting control-d or typing exit

exit

Now we'll start the core in 'live' mode. The directions tell us to type:

./exec.sh <NETWORK_CODE> core
stellar-core run --wait-for-consensus

I modified that to this:


The directions tell you to type stellar-core run --wait-for-consensus, which is the correct command. The problem is that when you do this your command will run on your console, and when you exit your console the program will stop. You options to get around this include:

• Run a different terminal window for each of the programs you'll need to manually start (that'll be 5 if you're running the Stellar Explorer and both KAU and KAG nodes) and leave them running continuously.
• Use a container management tool like Portainer where you can enter each container, issue the appropriate command, and just leave it running. This is the simplest if you're already running a lot of containers, but it's additional overhead here.
• Download and use Screen or a competitor
• Just use the & command line switch to run the program disconnected from your terminal.

The last option is what we did here, and we wrapped the command in brackets to make things run properly.

(stellar-core run --wait-for-consensus &)

Now the program is technically running in the background, but you'll still see its output as it catches up to the live blockchain transactions:



Once you start seeing things like Got consensus and Closed Ledger you know you've caught up and you can control-d or type 'exit' to exit the core container.

Now, to do the same with Horizon:

sudo ./exec.sh kag-mainnet horizon
export ENABLE_CAPTIVE_CORE_INGESTION=false
(horizon serve &)

Again, sudo is needed on Ubuntu and might not be on yours, and I'm using the ( <command> &) approach to be able to watch the service start up but have it disconnected from my terminal so I can exit the container and leave Horizon running.

This will run for a few minutes, then you'll see screens like this:



When you get to that point, Horizon is up to date and you can exit the container by typing control-d or typing 'exit' as before.

Now that this is complete, you can test your KAG node by pointing your browser at it. It runs on port 8001, so if you're running it on your local machine http://localhost:8001 will get you there. This should show you a screen very similar to what we did at the beginning to get the current blockchain number.

(Note that port 8000 is used for KAU, and 8001 for KAG).

If you want to see a neat trick that makes reconciliation of the ledger easier, go to http://localhost:8001/coin-in-circulation and you'll see something like this:

{
  "ingest_latest_ledger": 27607187,
  "history_latest_ledger": 27607187,
  "history_latest_ledger_closed_at": "2023-08-27T19:38:13Z",
  "history_elder_ledger": 2,
  "records": [
    {
      "circulation": "3408001.1746000",
      "mint": "8639557.0974000",
      "redemption": "5231555.9228000",
      "date": "2023-08-15T00:00:00Z"
    },
    {
      "circulation": "3412401.1746000",
      "mint": "8643957.0974000",
      "redemption": "5231555.9228000",
      "date": "2023-08-16T00:00:00Z"
    },
    {
      "circulation": "3416620.0496000",
      "mint": "8648175.9724000",
      "redemption": "5231555.9228000",
      "date": "2023-08-17T00:00:00Z"
    },
    {
      "circulation": "3423820.0496000",
      "mint": "8655375.9724000",
      "redemption": "5231555.9228000",
      "date": "2023-08-18T00:00:00Z"
    },
    {
      "circulation": "3424420.0496000",
      "mint": "8655975.9724000",
      "redemption": "5231555.9228000",
      "date": "2023-08-21T00:00:00Z"
    },
    {
      "circulation": "3424620.0496000",
      "mint": "8656175.9724000",
      "redemption": "5231555.9228000",
      "date": "2023-08-23T00:00:00Z"
    },
    {
      "circulation": "3425220.0496000",
      "mint": "8656775.9724000",
      "redemption": "5231555.9228000",
      "date": "2023-08-25T00:00:00Z"
    }
  ]
}

Those are the circulation numbers your blockchain node sees at the times mentioned. I performed that capture on the 27th, and the results went back to the 15th. That is the number the blockchain sees, that we should be seeing matched (or exceeded) in the audits. If you just want to be diligent about capturing those numbers you can save yourself a lot of effort in the next step: installing the open source Stellar Explorer.

 

[Derek]

Installing Stellar Explorer​

Your node is a lot more useful if you can browse it, and it gets really useful if you can download a CSV file of all transactions to date and analyze that in a spreadsheet. That's why we're installing Stellar Explorer.

Stellar Explorer is open-source software that lets you explore a Stellar ledger. This works with the Kinesis blockchain because Kinesis is derived from Stellar, but the explorer doesn't know what Kinesis is - it just presents the data as if it's Stellar.

That said, it's a great tool to analyze the Kinesis blockchain, using an open-source tool.

You can install it from here:

GitHub - chatch/stellarexplorer: Ledger Explorer for the Stellar Network 🚀

Ledger Explorer for the Stellar Network 🚀. Contribute to chatch/stellarexplorer development by creating an account on GitHub.

github.com

So go back to your home directory (that's just 'cd' by itself on Linux), then clone the software like we did above, just with this new repository:

git clone https://github.com/chatch/stellarexplorer.git

By default the explorer will only export 20,000 transactions, and that's not enough. To modify this, edit this file:

~/stellarexplorer/src/components/shared/DataFetchingAllContainer.js

(I use vi, but edit it however you like.)



20,000 was too low, so I added some zeros. Change this to whatever you think is appropriate, but choose a number larger than the max block size.

Now, go back to the stellarexplorer folder:



From here, follow the directions on the github page. A simple

sudo npm i && npm start

will get you up and running. If you want to do a more formal build, go ahead.

If you want to make this available to others, please secure it properly. Securing your installation to make it public is not covered here.

 

[Derek]

OK folks. That's all of it. I hope that's clear.

If you have questions, please ask and I'll either modify the original posts or clarify in the comments.

 

[Uchiki]

This looks like a great write up @Derek .

Will be interesting to get feedback from anyone who uses it to set up a watcher node!

 

[ZürichGnome]

First of all thanks to Chorn Sokun and Derek for great instructions.

1)
I ran this with great expectations. Started Sunday morning.

after 36 hours elapsed or so
CONTROL-C

The torture was just too painful

I have spent months of my life in machine rooms around Europe. All-nighters.

And this reminded me all too vividly of those experiences.

My colleague, Helen (from Brunei as a matter of interest for Indonesians out there) and I went into a funk when we realised the main developers had never even heard of sqlldr (SQL loader).

Those developers were also writing nice detailed messages to the screen.
Which nobody reads.
And which slow the processing down seemingly to a crawl.

There is little more mind-numbing than looking at 15 computer screens for hours at night, all diligently showing what they are loading when you know that with better software design, the whole thing could have been dealt with more reliably in batch in 20 minutes.

The equivalent for Kinesis/ Horizon/ Stellar would apparently be pgloader

1. pgloader load the data into temporary tables. Doesn't have to be main database.
2. Do the validation work in memory, in chunks.
3. Write chunks of good records out wherever you need to.
4. Write bad records to error log file.
5. Ende

2)

horizon db reingest range 2 27552127

Is thre any way of disabling the reams of info messages to stdout (aka the screen) ? I went fishing for a while in github but surely somebody has sussed this one out a long time ago ?

3)
My next steps -if I can recover my motivation- might be to pay for computer space somewhere in The Cloud.

Balaji S. Srinivasan in 2013 had a very usable tutorial on that topic. Amazon AWS.

linode/ akamai looks relatively expensive.

4)
A word of thanks also to Ketan Gulabdas, ministryofnodes for a very good tutorial on how to set up an Oracle Virtual Box to create a virtual Ubuntu machine (VB). A standard choice apparently. He appears to be based in Sydney.

His purpose here is to create a Bitcoin node.
Yet the first 10 minutes or so are excellent for our uses too.

I used an empty WD USB 2 TB hard-drive (lots of interesting fdisk and gparted challenges prior; new and important linux material to experiment on). Only after the fact did I realise that the horizon process writes to the first partition by default (?). I had set that too low at 50 GB.

I had also git-cloned nodes twice. Once to first parition (my error). Once to second (200 GB+). I carefully started the reingest from that second partition. To no avail.

df -hT

Showed disk space getting used up only on the first parition.
Slowly. Very slowly.
Like watching pain dry.

I allocated 5 G RAM of 8 G RAM to the VB.
And 2 processors of 2 (...)

This ensured that my main workhorse box was unusable for 36 hours for other work.
Perhaps I'm just impatient.

I could try again with lower numbers but I must admit my motivation is on a downer at the moment.

sudo apt-get install virtualbox
sudo ls
sudo virtualbox &

I had not planned ahead properly.

Password for my VPN service?
Other passwords.
Enable clipboard copies between main box and VB. Prior. Feasible under VB settings /options.
Make hard-disk partitions available to VB prior. I did one extra. Had to reformat as ext4 again inside the VB. It could see it . I could use it.

linux> man gnome-disks
linux> gnome-disks &

man is not available in the minimal installation of Ubuntu by VB. Or perhaps I'm getting confused by the special processes created by docker.

Installed curl and lynx inside the VB

It might be an idea to not go for kag-mainnet but kag-testnet
This might be irrelevant if the reingest procedure sucks up the whole history of the Stellar chain anyway.

Anyway. Food for thought for people more intrepid than I ...

 

[ZürichGnome]

A postscript:
Later on same day.

Had to hard reboot 15-30 times
A tad concerning.
Shall we say...............

Possibly nothing to do with the reingest process.
More likely the VB stuff (Vitual Box = virtual machine).
I don't know for sure

My linux / Ubuntu personal computer is normally highly reliable.
90-99% uptime.
Brilliant machine.

The tasks we are talking about in this thread are operations tasks. This requires operations experience. I clearly don't have enough.

I did not take photographs of my afflictions (with a smartphone). I know nobody can offer support if you don't describe the problem precisely with logs, etc...

Did a vi of /etc/fstab because I had recently wanted to work cleanly by declaring my newly formatted special 2TB external hard drive. Bad idea apparently after a hard reboot. ESC dd Deleted that line (vi standard stuff). Just to save waiting 1.5 minutes every failed boot. That's from the minimal boot command line interface. Not ubuntu. That's precisely what I was trying to load.

I just force myself to remain calm when fiddling around inside GRUB and BIOS settings. I am not sure how I got out of this pickle. Possibly I picked the first recovery option in the boot screen. Event that did not work first first time. Not sure. I am always highly befuddled by these situations. I have experienced similar about 2-3 times in 10 years. Keep calm and muddle on. Somehow.

Later when ubuntu did boot. I think I downloaded 200+ Mb of system updates. Then miraculously we were back to normal.

My instinct is thus reinforced into advising people on this forum to use resources in The Cloud away from your main workstation.

Unless you really know what you're doing.

On a more positive note:

Any knowledge about Ubuntu will be useful to better understand the Kinesis environment (which seems to run under Amazon AWS anyway. That is ubuntu as well. Yippee Yo). Any ubuntu and/ or linux knowledge gleaned will not be wasted.

ubuntu and linux are solid.

The free option at Amazon AWS would very likely be a perfect way for a beginner to practise basic linux skills. 95% of the time, trial and error really is the best and perhaps only way to learn linux.

The problem we-the-curious are facing is how to get up to speed on a system instigated by a team of people who are mainly sales and marketing driven. Or, stated more positively, people from a precious metals background. Which is indeed crucial in the case of Kinesis. Unfortunately what we need more of here in my opinion is guys with a solid software engineering background. And Ops. Historically those two are not the same skill set.

I am guessing the techies at github (bullioncapital, stellar, horizon, etc...) are either working on more important priorities at the back-end (?) or have moved on to other jobs.

Perhaps I'm wrong. I still have ZERO blockchain experience. But I was second-level support at Digital Equipment Corporation years ago (albeit in an extremely lowly position). Any techie can tell within seconds when someone is blagging when talking about tech. That just ain't work long-term.

In the end even a blockchain will be about data, volume, location, backup procedures, security, network failure, CPU power and so on. The classics buried under reams of Javascript. The GUIs are pretty and necessary for users but at the end of the day, it all boils down to the bread and butter of computing:

data

and

clean software

End of rant.