Analysis of non-English communities on Steem

Repository

Introduction

There are many communities on the Steem network that use a different language than English. For example, I am actively involved in the #polish community using the @jacekw account. The purpose of the following analysis is to find the most active communities of this type, compare them and observe how they have changed over time.

Outline

Scope of the analysis
Tools
Verification of initial data
Different tags for the same languages
Posts
Payouts
Average payout per post
Authors
Tags with prefixes
Conclusions
Proof of work

Scope of the analysis

The data has been downloaded from the SteemSQL database (tableComments) and refer to the first 6 months of 2018. The following script was used to download posts from the given tag.

SELECT url, total_payout_value, active_votes, json_metadata, created, body_language
FROM Comments (NOLOCK) c
WHERE depth = 0 AND
      (CONTAINS(json_metadata, 'spanish') AND json_metadata LIKE '%"spanish"%') AND
      YEAR(created) = 2018 AND
      MONTH(created) <= 6

Tools

Verification of initial data

Potential language community tags have been manually selected based on 1000 most popular tags (from a 14 day time period).

lang_tags = [
    'indonesia', 'spanish', 'aceh', 'kr', 'cervantes', 'cn', 'deutsch', 'castellano', 'venezuela',
    'tr', 'polish', 'fr', 'myanmar', 'japanese', 'ru', 'pt', 'thai', 'ua',
    'morocco', 'arab', 'pilipinas', 'steemit-austria', 'mexico', 'vn', 'rusteemteam', 'cesky', 'bangladesh',
    'russian', 'hindi', 'br', 'arabic', 'teamserbia', 'steemromania', 'teamukraine', 'filipino', 'serbia'
]

There are different conventions here:

country name e.g. #indonesia, #venezuela
name of the language used, in English e.g. #spanish, #polish
name of the language used, in the mother tongue e.g. #cervantes, #deutsch
two-letter code e.g. #kr, #cn
other name e.g. #steemit-austria, #rusteemteam

In my opinion, the name of the country is not the best choice for the language community tag, because this is the first tag that comes to mind if we want to add a post regarding given country in English.
The tags have been selected manually, so additional verification is needed, as some of them may not be relevant to language communities at all.

For this purpose, I used the column body_language from the tableComments. Below you will find charts showing the share of individual languages in given tags.

We can see that some tags are dominated by English. Such tags will be omitted from further analysis. Since we are not sure to what extent the body_language can be trusted, we will set the threshold quite low, at 30%.

.	Tag	Ratio	Lang
1	castellano	94.1	es
2	br	92.5	pt
3	cervantes	89.4	es
4	kr	87.4	ko
5	thai	87.3	th
6	spanish	87.0	es
7	teamukraine	85.5	uk
8	venezuela	84.7	es
9	pt	81.0	pt
10	polish	79.2	pl
11	myanmar	77.4	my
12	tr	68.7	tr
13	japanese	68.4	ja
14	deutsch	67.9	de
15	fr	65.1	fr
16	steemit-austria	63.1	de
17	rusteemteam	61.2	ru
18	cesky	59.7	cs
19	indonesia	53.2	id
20	cn	51.0	zh
21	arabic	50.6	ar
22	ru	48.3	ru
23	mexico	47.4	es
24	aceh	46.2	id
25	morocco	45.6	ar
26	arab	41.2	ar
27	ua	34.6	uk
28	pilipinas	33.1	tl
29	russian	32.9	ru
30	hindi	30.1	hi
-	-	-	-
31	filipino	19.7	tl
32	vn	17.8	vi
33	serbia	15.9	sr
34	bangladesh	15.4	bn
35	teamserbia	15.0	sr
36	steemromania	13.9	ro

The last 6 tags will be deleted.

Different tags for the same languages

It also appears that some communities use the same language as others. Let's see what the relations are between them, whether they have a large common part.

es : castellano, cervantes, spanish, venezuela, mexico
pt : br, pt
uk : teamukraine, ua
de : deutsch, steemit-austria
ru : rusteemteam, ru, russian
id : indonesia, aceh
ar : arabic, morocco, arab

def plot_venn(tags):
    plt.figure(figsize=(6, 6))
    fn = venn2 if len(tags) == 2 else venn3
    fn([tag_urls_dict[tag] for tag in tags], map(lambda t: '#' + t, tags))
    plt.show()
    
for lang_code, tags in same_lang_dict.items():  
    if 2 <= len(tags) <= 3:
        plot_venn(tags)
            
plot_venn(['spanish', 'cervantes', 'castellano'])
plot_venn(['spanish', 'mexico', 'venezuela'])

pt : #br, #pt

uk : #teamukraine, #ua

de : #deutsch, #steemit-austria

ru : #rusteemteam, #ru, #russian

id : #indonesia, #aceh

ar : #arabic, #morocco, #arab

es: #spanish, #cervantes, #castellano

es : #spanish, #mexico, #venezuela

The #teamukraine tag is practically completely contained in #ua, so it is going to be omitted from further analysis, especially that both of them concern the Ukrainian community. For #pt and #br the situation is similar, but #br seems to be a separate (Brazilian) community.

Posts

Let's see how many posts were added in each tag.

.	Tag	Posts
1	indonesia	300522
2	spanish	282685
3	aceh	258482
4	kr	242179
5	cervantes	158545
6	cn	74577
7	deutsch	71291
8	castellano	61437
9	venezuela	58809
10	tr	57838
11	polish	29744
12	myanmar	20385
13	fr	20350
14	japanese	19592
15	ru	18445
16	pt	16875
17	thai	16681
18	ua	14414
19	morocco	8851
20	arab	6444
21	pilipinas	6389
22	steemit-austria	6386
23	mexico	4643
24	rusteemteam	4615
25	cesky	3956
26	russian	2432
27	teamukraine	2133
28	hindi	1990
29	br	1326
30	arabic	1240

We can see a great diversity here, communities from leading places have several hundred times more posts than those from the end. The Spanish language tags are top of the list, which may indicate that Steem is well known in South America.

Author rewards

However, the number of posts is an insufficient indicator, because it is very easy to spam a given tag with low value posts, and this will not indicate the popularity of a given tag at all. Therefore, let's also look at the sum of rewards in individual tags.

.	Tag	Author rewards
1	kr	342070
2	spanish	166190
3	cn	158431
4	cervantes	105259
5	deutsch	86543
6	indonesia	81085
7	tr	63973
8	aceh	42160
9	fr	36596
10	castellano	35293
11	japanese	26803
12	pt	22386
13	myanmar	19214
14	polish	19053
15	ru	18731
16	venezuela	17815
17	steemit-austria	13182
18	ua	12459
19	thai	12041
20	morocco	9888
21	mexico	8887
22	arab	6835
23	br	4579
24	cesky	4246
25	pilipinas	3428
26	rusteemteam	2364
27	hindi	1325
28	russian	718
29	teamukraine	519
30	arabic	414

The table looks quite similar to the previous one. It is also worth to look at the average rewards in a given tag. This will allow us to find out how rich the community is.

Average payout per post

.	Tag	Average author rewards per post
1	br	3.453
2	cn	2.124
3	steemit-austria	2.064
4	mexico	1.914
5	fr	1.798
6	kr	1.412
7	japanese	1.368
8	pt	1.327
9	deutsch	1.214
10	morocco	1.117
11	tr	1.106
12	cesky	1.073
13	arab	1.061
14	ru	1.016
15	myanmar	0.943
16	ua	0.864
17	thai	0.722
18	hindi	0.666
19	cervantes	0.664
20	polish	0.641
21	spanish	0.588
22	castellano	0.574
23	pilipinas	0.537
24	rusteemteam	0.512
25	arabic	0.334
26	venezuela	0.303
27	russian	0.295
28	indonesia	0.270
29	teamukraine	0.243
30	aceh	0.163

We can see that the differences are significant. It should also be taken into account, however, that tags with a small number of posts may not have reliable results.

Authors

We can look at the popularity of a given tag from a different angle - looking at the number of authors, not posts. This indicator seems even better, because it omits situations where a given user spams a very large number of posts.

.	Tag	Authors
1	spanish	48312
2	indonesia	42307
3	aceh	40846
4	cervantes	31520
5	kr	25622
6	venezuela	18329
7	castellano	15121
8	cn	9226
9	deutsch	9109
10	tr	6931
11	polish	4785
12	japanese	3118
13	fr	2976
14	myanmar	2851
15	ru	2083
16	thai	2060
17	mexico	1998
18	pt	1829
19	arab	1058
20	morocco	1039
21	steemit-austria	880
22	ua	854
23	pilipinas	846
24	russian	796
25	hindi	638
26	rusteemteam	458
27	arabic	441
28	cesky	369
29	br	291
30	teamukraine	81

Now we see how small some communities are! And if a community does not exceed a certain threshold, its members may prefer to post in English, which makes the growth of such a community more difficult.

Tags with prefixes

A common problem in language communities (I say this as someone who is actively involved in the #polish community) is how to use other tags. If I make a post in Polish and tag it: #polish #bitcoin #cryptocurrency, these last two tags will make the post reach also English-speaking audiences, so it is not a good solution.

Another idea seems to be to use tags in the mother tongue, but then you may find tags that function as a word in both languages, e.g. #film. This shows that it is not a good solution either. Therefore, the Polish community decided (based on the convention adopted by the Korean community) on the convention of tags with the prefix pl-. This makes it possible to separate posts in Polish from the English-speaking audience.

Let us therefore look at which other communities use such a convention. The table below shows how many unique tags are with the specified prefix.

.	Prefix	Count
1	kr	2302
2	pl	815
3	cn	221
4	ru	168
5	jp	64
6	de	31
7	tr	24
8	pt	12
9	fr	8
10	ua	5
11	es	5

We can see that the Korean community has the largest number of such unique tags, followed by the Polish and Chinese communities. As far as the Polish community is concerned we can see a tree of these tags on the website: https://steemweb.pl/categories (by @rafalski)

Let's see what are the most popular tags of this type.

.	Prefix-tag	Count
1	#kr-newbie	83291
2	#kr-life	23164
3	#kr-writing	19077
4	#kr-event	13164
5	#cn-reader	13133
6	#kr-daily	12263
7	#kr-art	8748
8	#kr-travel	7077
9	#kr-food	6037
10	#pl-artykuly	5987
11	#kr-pen	5678
12	#kr-coin	5494
13	#kr-join	5484
14	#kr-news	5011
15	#jp-newbie	4171
16	#kr-overseas	4055
17	#kr-diary	3812
18	#kr-gazua	3752
19	#kr-series	2980
20	#kr-funfun	2738
21	#kr-dev	2665
22	#kr-book	2601
23	#kr-youth	2563
24	#kr-story	2355
25	#cn-malaysia	1957
26	#kr-economy	1777
27	#kr-hobby	1708
28	#kr-steemit	1706
29	#kr-game	1693
30	#pl-fotografia	1612

As we can see, most of the tags are Korean.

Conclusions

We have managed to find 30 language communities that do not use English. There are probably more, but the process of finding them is not a simple one, because of the different tagging conventions that have been adopted. All these communities are probably waiting for the appearance of the functionality Communities, because this will make it easier for them to function.

It also appears that the size of language communities is not always correlated with the number of people using a given language / country size. An example of this is tag #kr, which is high in all lists and has a relatively small population compared to e.g. the Chinese community.

Some communities use almost only their mother tongue, while others have chosen to use English (e.g. #serbia). It is also worth remembering that the Russian community decided at some point to move to its own blockchain: https://golos.io, but on the other hand, this did not make it completely disappear from the Steem network. Interestingly, there are also quite a big countries that do not have a language community.

There are many other issues that could be explored here, such as how the community formation process looks like, who the pioneers are, who the leaders are (if any).

Proof of work

Scripts used in this work (as Jupyter Notebook)