[deepamehta-devel] Twitter Research Plugin Released

[Malte Reißig]

Hi dear DeepaMehta Friends and Developers!

Background:
In preparation for the latest developer meeting at the TU Berlin a
couple of days ago I wanted to connect a popular web-service to
DeepaMehta 4. I spontaneously chose twitter as an exemplary platform to
connect, mainly because I am planning to implement a City-Map fueled by
the #kiezatlas-hashtag for the upcoming 10th Anniversary-Party of the
Kiezatlas Project and I needed to get to know the Public Twitter-API anyway.

Also I wanted to dive into the GUI Framework of the current DeepaMehta
Webclient and explore it's current limitations by a working example. So
I just released the DeepaMehta 4 Twitter Research Plugin 1.0 (Source
Code [1]) and you can try it out, it is working fine with (a fresh)
DeepaMehta 4.1 installation. To install this plugin, you currently first
have to download the project and then build it from source [2].

Usage & Installation:
For users: This plugin allows users of DeepaMehta to query [3] Twitters
Public Search API [4] and start qualitative research around events based
on the messages sent by users of twitter.com. The query basically wraps
all meaningful options of the API (e.g. Language, Location and
Query-Input). The "Twitter Search"-Topic enables to "Load More Tweets"
(Paging) and to "Refresh Search" loading new tweets (since the last
search).

Please note when installing: The plugin just works as expected (a
"Twitter Search" shows up in your "Create"-Menu) if you install the
plugin during the initialization of a _new/blank/fresh_ DeepaMehta 4.1.
Do not install this plugin into an existing DeepaMehta installation (one
which already has content in it). To do so, you need to copy the
"twitter-research.jar" into the "bundle" directory of your DeepaMehta
installation and then start a fresh DeepaMehta installation for the very
first time. This is due to an issue I just discovered and which remains
to be investigated (see https://trac.deepamehta.de/ticket/465).

The Public Twitter API makes available most (not all) tweeted contents
from within the last 6-9 days. Once queried all tweets can be accessed
through DeepaMehtas fulltext search (and usage of the lucene-search
commands).

The plugin also extracts "Twitter Users" (uniquely) and thus associates
all tweets found across various queries with the correct twitter-user.
Around "Tweets" this plugin makes avaible some of the information
provided by twitter through their API which seems invisible in their
web-interface, namely ("Tweet Withheld in Countries" and the "Copyright
Withheld"-Flag).

Further Development:
This plugin is part of the deepamehta webclient and utilizes much of the
webclients own GUI Framework, thus providing working examples for Simple
Renderer, Multi Renderer and Commands.

Noteworthy is that the Twitter Public Search API enables us to easily
extract to URLs and Hashtags from Tweets (via the "Entities"-Field),
thus making a potential tighter integration of this Twitter Search
Plugin with the DeepaMehta Webbrowser or the DeepaMehta Tags Module (at
some time point in the future) appear seamless for end-users and other
applications that build on "Tags" and "Web Resources".

I would be happy to hear from other developers who are interested in
improving this plugin because currently I have no time on significantly
improving the featureset of this. Nonetheless I am happy to get bug
reports and fix possible issue.

Deployment:
I would be happy if someone could introduce this plugin to our friend
Mr. Jenkins, so a binary release of the plugin becomes available for
end-users at http://download.deepamehta.de.

Issues:
There are a couple of itches when bringing this plugin but for now I'll
leave it up to you and your feedback to point out some of the issues you
perceive.

Overall this plugin should provide a good starting point if one simply
wants to start querying "Web Resources" from "social bookmarking"
services or stuff alike.

I hope somebody is digging it.

Cheers!


[1] https://github.com/mukil/twitter-research

[2] https://trac.deepamehta.de/wiki/PluginDevelopmentGuide#Buildtheplugin

[3] https://dev.twitter.com/docs/using-search

[4] https://dev.twitter.com/docs/api/1/get/search