Unitalk with Shawn Douglass: The Future of Web 3.0 Data Driven Applications

16 min readDec 4, 2019

Originally posted on 26, Oct. 2018.

A long deep dive into cryptoeconomic data. 🧐

Hi Shawn. Nice having you here. Please introduce yourself and Amberdata.

Hi, this is Shawn Douglass, Co-Founder of Amberdata.io. We are building core-infrastructure to enable the adoption and usefulness of blockchain and DApps. Today we offer real-time analytics and monitoring for every smart contract, time series historical account balances and token transfers.

We provide full-text search on every transaction, log, internal message, and event. We have Application Performance Monitoring (APM), Historical Event analytics and real-time views into blockchain infrastructure and real-time event streams for every smart contract the instant they are published on-chain.

We also offer deeper security audits than anyone, for over 280k smart contracts now and growing. Lastly, we have inbound APIs which allows us to ingest any blockchain data and outbound APIs that people can build upon today. One API to learn unlocking blockchain for everyone. We would love feedback on what we have built as well as what we are missing.

It seems that Amberdata and Etherscan are kind of alike according to your introduction. How is Amberdata different from Etherscan?

Etherscan was the first blockchain explorer for Ethereum and we have a lot of respect for what they have provided for the community early on. The Amberdata team has run production software businesses at scale and recognized three things.

First, any system that is facilitating transaction of >100M USD is expected to be available, resilient and maintain its integrity even if exposed to potentially continuous attacks. Traditional platforms have infrastructure and application performance monitoring and operational analytics and a support teams monitoring these systems around the clock standing by if there is degradation of any service. This was missing and the community need it so we built it. Enterprises use products like Datadog, AppDynamics and Splunk in their operations centers to monitor the state and health of their infrastructure and applications. We took these operational use cases and built a single platform defining this category for blockchain.

Second, although there are many blockchain projects there are only a few type of blockchain architectures, UTXO, Account based and DAGs. There are really only a few types of virtual machines EVM and WASM and consensus mechanisms are also grouped into families being more evolution of traditional distributed systems than entirely new. Thus we have architected Amberdata to decompose difficult to get blockchain data from any blockchain into more easily accessible data structures, restful & websocket APIs which we built upon. So although our analytics platform has basic blockchain explorer functionality like etherscan, stellar.expert or NEO Tracker. We are not a tightly coupled monolithic architecture, our open inbound APIs allow the community to push our ingestion API data from any blockchain.

Lastly, smart contracts transfer value and often operate in tradable tokens. Therefore by instrumenting smart contract platforms like Ethereum, AIOn and Stellar we have deeper insight into on chain trading data than anyone. This positions us to provide you with value, volume, velocity, # unique users, gas price and market depth per token. Funds and traders can now monitor their portfolios, diligence potential investments and act on tokens they are watching in “My Dashboard”.

Why is it hard to capture blockchain data, can you give an example?

Blockchain is beautiful because the data structures enable transparency into an immutable record of state transfers that is verifiable. This allows us to re-delegate trust from institutions to protocols. Unlocking a creative renaissance of applications moving on chain. This does not mean however, that the data is easy to access. I’ll give a few basic examples.

First, timeseries account balances for ETH and tokens are not easy to get. Let say you want to know your daily change in value for an account you have on Ethereum from January 15, 2017. Lets operate in round numbers to keep it simple. You would need to go back to block ~ 3000000 and walk forward through every block to today and call > 3.5M blocks with your address and store the output of that into an excel file or a database. To simply chart your account balance to today. If you want to know your token account balances this is even more complex. Your account balance today is easy. That’s one call but over time, not so easy. Even an archival node run in your cloud provider is expensive to run and requires continuous care and monitoring. We make blockchain data easily accessible via dashboards and APIs.

Second, blockchain is not searchable, bloom filters are efficient at telling if a data element is a member of a set but the data is not indexed making it searchable and quickly accessible. We index every transaction, log, internal message, and contract event. This enable you drill into blockchain data to answer any question about usage, performance, security, reporting or compliance. Ultimately, you need to be able to measure KPIs to improve systems, applications and even crypto-economics to evolve.

Third, Smart contract platforms capture the end state of executed code however, the call stack and execution before that is not stored on chain. This makes sense as it is ephemeral state of the execution of a smart contract. However, if you want to know why, when, where, how something went wrong or how you can improve, its super important to be able to do queries and get insights against that data. We provide time series contract events for every contract on Ethereum today.

Lastly, smart contracts may call other smart contracts. Orchestration across discrete services is good engineering however, when your transfering value you need to know what happened where, when, how and if it was successful and efficient. Capturing this ephemeral data is only possibly by replaying each function call in the EVM and further instrumenting that so it makes sense. We make all of this easy to enable you and everyone else to build upon smart contracts and make web3.0 a reality. Without all of this we must blindly trust that things are working as expected.

Could you share some interesting learnings from monitoring Ethereum over the past year?

Ethereum is a hotbed for innovation and is not just a cryptocurrency but a globally distributed platform which enables entrepreneurs and developers to build, run and monetize unstoppable software. One year ago, only 30% of all transactions were smart contract transactions; for the last 7 days, 58% of all transactions were smart contract driven. Despite the crypto bear market, Ethereum has only gained developer interest. Only 4 months ago, an average of 600 smart contracts deployed onto Ethereum mainnet on a daily basis; over the past month that number has increased to 8,000 daily!

You have recently added support for automated smart contract security audits. Why is giving users access and visibility to security issues important?

Every server and service running on the internet is constantly being probed for known security vulnerabilities. There is a constant cat and mouse game between security professionals and those curious and or nefarious minds. This is true for simple wordpress blogs to the most secure trading systems.

Blockchain provides an attack services with a target rich environment where there is real value at risk. A famous bank robber Willie Sutton when asked why he robbed banks famously responded “Because that’s where the money is”. Smart contracts are where the money is on the internet. Blockchain is about delegating trust to consensus mechanisms and enabling frictionless exchange and finality in transaction. Trust is critical for web3.0 to work. Because smart contracts once deployed, to change are for the most part unchangeable. Security is more important than ever before.

Getting your smart contract audited for security is crazy expensive, and it is almost impossible to find and hire qualified teams to conduct them. There is also little confidence that an audit by one team will provide the same results of another. We believe that security is absolutely critical for this revolution to gain and maintain momentum and bring web3.0 to fruition. Amberdata is there for helping developers to easily identify vulnerabilities and mitigate risk as they build and maintain their applications. This also helps investors and consumers understand the security posture of a smart contract and its respective token and make informed decisions.

We hope that by providing these tools to the community, we can reduce outside dependencies and enable the community to develop faster more safely.

What are the metrics that investors should be looking for when evaluating a token?

Similar to traditional assets, I believe, price action, volume, velocity, volatility and expected move, are key technical indicators also applicable for token trading. Additionally, I think adoption and usage as measured by the number of unique users give a feeling for momentum building over time. We show all these in our platform as well as are adding NVT Signal which is similar to P/E ratios in equities giving you a measure of network over transaction volume. I also like to look at the risk adjusted return on a token over ETH or a period of time. Less than 20% of tokens out perform ETH the underlying asset. I’m looking for good valuation models, I would love hear from people if they have something they are willing to share.

A few months ago, Facebook became embroiled in one of the biggest personal information leaks in history. How do you avoid being the next big data aggregator like Google and Facebook?

Traditional data aggregators platforms like Facebook and Google captured most of the value created by their users as they aggregated content and served up targeted advertising to their users, while users happily produced their product “Information” at zero marginal costs. This implies a walled garden where data is collected, analyzed and privately and the user generated data is used to target the same people who created it. User get a free service for being the product.

Internet 3.0 is about breaking down those walls again and freeing up sovereign individuals to act and benefit more directly from their contributions. Amberdata is aggregating and making accessible blockchain and blockchain generated data to everyone to participate, in a marketplace and economy via Web3data tokens.

I believe decentralization is more about taking back control and opening up access and opportunities on a global scale than where and how data is stored.

Can you tell us about the work Amberdata is doing with Web3data foundation?

The Web3data token will be integrated into our platform tokenizing our unit economics. This will enable network effects not available without it and providing richer data for the community, while opening up and incentivizing the community to contribute and participate into an open data marketplace.

As the launch partner for the web3data token, Amberdata will accept payment for our to be released premium features in USD, ETH or Web3data tokens. If customers choose to pay in Web3data they will receive discounts, which is a model that has worked well for other token bootstrapped economies. For payments made in cash, we will convert those transactions into Web3data tokens on the backend mitigating issues others have run into.

We are excited about and looking at using ERC-948 to enables frictionless subscription to data and services governed on-chain by a smart contract. This is particularly compelling because it reduces our exposure to volatile gas prices and mitigates performance issues of being entirely on-chain today.

We believe this will enable us to operate a global platform making decentralized data and web3.0 easy for everyone to adopt, use and build upon. We aspire to be an example others will follow.

What is the landscape of blockchains, which blockchain will become No.1 or will there be an interop of blockchains?

If I knew that I would sell my house and buy those tokens before responding :-) Seriously, I believe the industry will consolidate around a few major blockchain and smart contract platforms however, most digital goods and the representation of physical assets will likely be tokenized. Tokenization can be thought of as the next step in the evolution of digital transformation and business process automation. Removing counter parties and automating settlement.

Therefore cross-chain interoperability, ILP, bridging, swaps, side-chains, sharding, plasma, shasper, etc, are critical. Without the ability to transfer value and assets across chains, each blockchain becomes a walled garden locking is participants in to an open but biased economy. People need to be able to choose the right solution for the problem and change when required.

End-to-end transparency will unlock cross-chain transfer flows and enable broader adoption of tokens.

Have you visited China and what’s your outlook on China’s role in Web 3.0 revolution?

Yes, I love China and have been many times. I’m a big fan of Sichuan hot pot, chili duck tongue, Beijing and Shanghai cuisine. I guess I like to eat LoL

I think the Web 3.0 revolution is happening in China as much maybe even more so than in Silicon Valley. China already uses WeChat for payments so Crypto is an intuitive continuation. Web3.0 also removes the need to convince traditional gating parties to resources such as capital and large companies of the merits of an entrepreneurial idea. Web 3.0 makes access frictionless and available to anyone willing to invest the time to learn. China is very well positioned and a major contributor to the adoption and success of Web 3.0.

Is there a possibility that some blockchains don’t want to partner up with you because they are afraid of being “monitored”? If so, how would you convince them?

“Monitored” being seen as a negative here I assume. Blockchain is about transparency and trust and removing required intermediaries. Even if we provide insight into the state and health of transactions. This doesn’t mean nessarily exposing the payloads of the txns. For example today most ecommerce is done over TLS/SSL and all production services have some kind of monitoring.

Blockchain itself is transparent already and Amberdata claims to create greater transparency. In wha ways does Amberdata provide greater transparency exactly?

The transparency on blockchain is not a given actually. Almost all blockchain protocol today does not have built-in audit and reporting feature. You can easly call any address for a point in time on a block and get the state at a block but doing it over a time series, making that data searchable or getting insight into the interals or even the overall health of the network is more complex.

I’m curious about your code auditing on smart contracts, have u open-sourced your security tools? If not, will you?

Actually, we talked to a bunch of smart contract security team doing audits. They all use the same open source tools. We took opensource tools that people are using today for state code, byte code and decompilers and built a rubric for scoring to quantify risk. We felt smart contract security audits are really required and shouldn’t cost 50k-1M USD and people shouldn’t have to wait weeks or months to get those done. So we are built on OpenSource and we are provding a free service for the community to mitigate some of these issues.

Can Amberdata help to identify some specified spam attacks on Dapps? i.e. we have the issue that some guy created lots of transactions on the network during the last few seconds and took the big price of Fomo3D game.

We can absolutely see it via our platform when that happened. We can view it from the transaction spike as well as the gas price spike. This is a great example of why you need infrastructure monitoring. We are using a shared service the network, all systems have constraints and in this case one of the constrained resources impacted other users.

Is it so that Amberdata only detects anomalies but you don’t offer solutions or help the smart contracts to remove the anomailies?

Our first goal is to make the basic security audit result easily accessible for everyone. We are a small team, we first want to shed light on the obvious activity that may not be as expected.

Have you open-sourced your security tools? If not, will you?

My second question is, can Amberdata help to identify some specified spam attacks on Dapps? Exp., we have the issue that some guy created lots of transactions on the network during the last few seconds and took the big price of Fomo3D game.

Can you tell us the smart contracts auditing is occurring on source code of solidty or evm code, and whether the safe auditing occurs dynamically or statically?

Awesome quesitong :-) We do audits on solidity code and byte code as well as as static analysis using decompilers. The audits are static today. We do see some potential bad behavior on some smart contracts in real time, like people paying gas to call functions on smart contracts that don’t exsist and thus failing, but haven’t built a way to present that yet. This I think is like port scanning probibly on open networked systems. people looking for an easter egg.

In my opinion, most of the safe problems come out when they are running really, so dynamic auditing is more important than static one, of cource is very hard to code to achieve, at least now.

I agree live running code is what is most important, that’s why we have an anomaly detection engine but it’s still early days for us. There is so much basic infrastructure that needs to be built and making sure people don’t have simple errors is foundational, we know how to do robust streaming analytics at scale for data, network, performance and security so there is so much to build. You should join us :-)

What does your current paying customers look like, and what kind of customers will be your main revenue source in the future?

We believe we can monitize the platform via advance features for security, compliance, audit, market places and even providng exchanges and other analytics platforms or people building deep industry vertical solutions with data. Right now we are building and defining this catagory. There is a 28B Total addressible market in US Enterprise software market for Application Performanc Monitoring, Operationa Insights and Infrastructure monitoring. We have rolled up all of those offerings into a single built for purspose blockhain focused platform. We later learned that we actually have Bloomberg like data as well because every transaction on blockchain is financial. Awesome question thank you.

What’s the main purpose of creating a network effect via tokens? Do you worry about multiple players in the blockchain data analytics sector and it’s hard to differentiate simply from technical capabilities (as data is not longer a barrier)?

Great question, actually data is still very much a barrier processing and storing data for easy query and analytics in real time across one blockchain is still hard, doing it across many blockchain much harder. We are not concerned about other Analytics platforms actually. Think about Amazon and AWS (Amazon Web Services) in the US for example. I’m sure the same could be said for Alibaba and AliCloud.

We build an analytics platform but did so in a general purpose way enough that we learned that we could expose our backend infrastructure so others could build on top of it like AWS for Amazon. The analytics platform is only the tip of the iceberg.

What do you think of oracles? How do you monitor/analyze interactions between smart contracts and the world outside?

Awesome question :-) Oracles are an interesting space. Clearly smart contracts need inputs from offchain data but there is the entire trust debate there as well. We monitor smart contract call stacks which includes the input/output of calls to other smart contracts. So for a contract we can see in the real time event stream as well as have a search history to be able to do forensics or triage issues.

For example:

as well as over time

this is cryptokitties contract, which orchestrats across several other contracts.

Somehow what Amberdata doing now makes me feel like it’s going “centralized”, because you aggregate data, transaction records and other stuff. Isn’t it against the principle of decentralized world?

That’s precisely why we are building decentralized data marketplace. “Centralization” is not just about the storage of data, but rather access to the data and the ability to build valuable from the data. The decentralized data marketplace will be powered by smart contract to enable access via ERC 948. Blockchain is not efficent for large scale data processing nor storage today. So we break complex data structures into relational, graph, indexed for search, rules engine to event on KPIs and Anomoly detection. Data comes in via open APIs and data goes out via a common open API. We are simply making blockchain data more accessable to all. It doesn’t matter IMHO that we use off chain compute and storage to do this. It is simply more efficient and performant to do this. We actually get this question alot, its a really great question.

If that is the case, my next question, have you considered using IPFS for data storage?

I wish we could use IPFS, I participated in the filecoin ICO which is built by protocol labs who built IPFS. Right now the performance, availability and cost isn’t there yet. We tried to use IPFS. Our Cloud infrastructure bills are $$$ so when there is a performant decentralized solutions for storage and compute that we can get $/IOPS performance we are all in :-)

Know more about Unitimes:

Website: www.untimes.pro

Twitter: UnitimesHQ_

[The copyright of this article belongs to Unitimes. Please contact us at editor@unitimes.io if you want to repost the article. Opinions expressed by contributors belong to themselves.]

Unitalk with Shawn Douglass: The Future of Web 3.0 Data Driven Applications

Written by Unitimes