Introducing "App Stream": Curate and serve Steem posts/comments/replies by app/author/category

Hello Steem, today I will be outdooring a new Steem utility I have been working on: App Stream.

During the development of Peer Query beta as a full fledged platform featuring content created only on it, we run into a challenge.

While we could index posts and comments made on our site, we could not index comments/replies made from other clients on posts created through Peer Query.

This problem applies not only to Peer Query, but also to all other Steem clients.

The challenge

Steem is great for content publishing, yet its open source data nature means that the same content can be served and interacted with from over several different interface.

This feature is good for open source and censorship resistance, however it poses a challenge for app/client owners who want to index all activities on posts made from their clients.

This is because while a post may be created on a one client, it would be edited, commented on, voted and resteemed from other clients.

Consider this challenge:

• Ama creates a query/post through PeerQuery.com
• John comments on it through raw Steem JS
• Jin replies to John's comment using Busy.org
• then the author Ama, edits the same post using Steemit.com
• Mary replies to Jin's reply using eSteem app
• countless more activities(resteem, votes, ...) happen on the same post from different clients

No matter how much indexing Peer Query does for activity on its site, we would definitely miss those made from other clients.

The only way to solve the problem is to stream all operation on the Steem blockchain and check if they match something in your DB or not.

App Stream

App Stream is designed for this challenging task, and is designed to be as modular and efficient as possible. App Stream is a standalone Node.js program that curates all posts as well as its comments, replies and even edits from whatever target you set in a MYSQL DB.

There are 3 types of target:

1. app
2. category
3. author

There are six Db engines:

• post-by-app
• post-by-author
• post-by-category
• post-comment-by-app
• post-comment-by-author
• post-comment-by-category
• post-comment-reply-by-app
• post-comment-reply-by-author
• post-comment-reply-by-category

Countless other features and configuration instances, see Docs for more.

How to install

1. Create empty DB, make sure MYSQL version is 5.7+ for json support
2. Create .env file in root folder and enter DB details. See sample in docs/sample.env
3. Git clone ssh:git@github.com:peerquery/app-stream.git or https://github.com/peerquery/app-stream.git
4. Install dependencies: npm install
5. For custom setup, edit: /config/config.js
6. Run system through setup manager: node setup
7. Use MYSQL Workbeanch/PHPMyAdmin to explore data in DB or visit api.localhost/api

Note:

You do not need to create any tables for the DB. App Stream runs on:

• 2 tables with all kinds of fields: json, text, timestamp, varchar, auto-increment, ...
• 1 view of the each of the two tables above
• 3 complex conditional db indexing stored procedures each with 11 IN variables and 1 OUT variable
• 10 db api stored procedures with IN and OUT variables

Manual setup would be inefficient and difficult, and yet it has to be setup successfully before the main server is initialized.

The db_setup module handles all this automatically for you, so you do not have to touch the DB at all. After all is done successfully it will automatically call the server.

Who can use App Stream?

Steem clients

Curating by app was App Stream's primary purpose, Steem clients would benefit the most from App Stream as they can use it to curate all posts made from their clients.

With the app_match option, you could curate data from only a specific version of an app, eg:   esteem/1.6.0,  esteem/1.5.1, or esteem/whatever.

With a separate App Stream server, clients could server all content directly from their DB instead of relying on fetching content directly from RPC node endpoints.

App/client owners now can index all posts, comments, replies as well as edits made to posts made from their client, no matter which thirdparty client the activity originated from.

Steem Curators

Curators would also greatly benefit from App Stream by using any of the category indexing engines:

• post-by-category
• post-comment-by-category
• post-comment-reply-by-category

We're working on Curator, a revolutionary open source interface for curating content which perfectly integrates into App Stream. Stay tuned.

Steem blogs

I don't know if this is a thing yet, but with the post-comment-reply-by-author engine you can setup this system to curate all posts by an author, as well as the comments and replies by it.

While this is not recommended because Steem API allow this function through  steem.api.getDiscussionsByAuthorBeforeDate(), with App Stream's source_app you can get specialized curation such as curating the author's post only if they are from a specific client such as Steemit.com, Busy.org, ...

Steem BI

With App Stream, you can curate all posts, comments and replies from any Steem author, category or app.

Such data would allow you to gain specialized data for specialized analysis.

Documentation

App Stream comes with a painfully long and detailed documentation. You do not need to read it all to get App Stream running, its only there to explain everything about App Stream with examples incase you need to tweak it or get lost getting something to work.

See the documentation on the Github repo or via this link: https://github.com/peerquery/app-stream/blob/master/README.md.

Hopefully the detailed Table of Contents would make it easier to navigate.

Cool addons

• Depth determiner for each post, comment and reply - will accurately add depth to a reply whose dept is 37!
• API documentation on API endpoint bases
• Subdomain support

See Docs for more info with examples.

Cool hacks

• Jailbreak to stream all Steem posts, comments and relies regardless of the source
• Host App Stream as a service and serve content via API
• Integrate with Hivemind for richer data interface
• Add a notifications feature

What's next

Curating only for the purposes of indexing in not enough. For clients such as Peer Query, we will need a full curation functionality.

I am already half way done with Curator, a powerful open source content curation interface for Steem. Curator will integrate with App Steam and allow a team(of multiple admins, moderator and curators) to curate content from Steem.

Curator comes with a full bot support which means all you have to do is supply the bot's account and posting key - it will periodically upvote all contents approved by the curators.

Like App Stream, Curator was initially being designed for Peer Query, however I realized that most of the content curators on Steem do so without a standalone app interface.

Then came the need to design Curator as a standalone app so other content curators(by tag, app or author) can also use it.

When Curator is done hopefully before this moth ends, Peer Query beta would be live; a complete move away from current query tag curation into a full platform with users, reputation, projects, curators and bots and more.

Acknowledgment

Thanks to @smooth and @howo for their recommendations on how to structure the db indexing functionality. Templates for the two streamer apps are by @almost-digital and @howo.

Thanks to you for reading, I hope you find App Stream of use.

Links

Github repo: https://github.com/peerquery/app-stream

Docs: https://github.com/peerquery/app-stream/blob/master/README.md

Author

https://github.com/Dzivenu