O'Reilly logo

Getting Started with Fluidinfo by Nicholas H. Tollervey, Nicholas J. Radcliffe

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 1. What Is Fluidinfo?

Fluidinfo is an online storage system in which there is a place for information about everything—everything that exists, everything that could exist, and everything that can be imagined. It allows anyone to store any information, about anything, in any digital form. And Fluidinfo makes it easy to find information in the system, and to extract it, using an unusually simple query language.

Fluidinfo is a first-class Internet citizen, exposing all its functionality through HTTP, the core protocol that underpins the World Wide Web. Programmers can take advantage of its RESTful API, which makes it easy to integrate with other applications.

Finally, Fluidinfo is social: users can exercise fine control over who can read their data and can even enable other chosen users and applications to write data on their behalf.

The Openly Writable World

One of the key aspects of Web 2.0 was the increasing writability of the Web. Although its inventor, Tim Berners-Lee, always envisaged the Web as openly writable, in its early years there were many more readers than writers. The advent of blogging tools, social networks, user product reviews, and online data services have all contributed to a significant reduction in the asymmetry between Internet authors and consumers. Yet even today, the extent of the Web’s writability remains strictly limited.

The motivating concept behind Fluidinfo is to provide a shared data space where anyone can attach granular data of any kind to anything on the Web or anything in the world. Of course, physical objects like the Eiffel Tower don’t exist on the Web, or in Fluidinfo, so digital data can’t literally be attached to them. But the open structure of Fluidinfo and the emerging conventions for how to refer to anything—digital or physical, real or imaginary—offer a very real sense in which Fluidinfo has a nominated container ready to accept information about anything. Any Fluidinfo user can add data to any such container.

Fluidinfo seeks not only to provide a place for arbitrary data, and mechanisms for reading and writing it, but also to leave the creator of that data firmly in control of it. We are all familiar with web services that encourage us to create data inside them, and to import data to them, but which make it much harder to get that data out, still less to allow other applications and services to access it. Fluidinfo takes a different approach: it not only provides excellent mechanisms for extracting and exporting data, but it includes facilities designed to encourage users and applications to share, remix, and reuse data in ways that will often not have been anticipated when it was created. Fundamentally, Fluidinfo leaves the data creator (or owner) in full control, not only in terms of who can read and write that data, but also which applications can use, modify, and extend it.

Fluidinfo’s radical reimagining of the way data can be created, shared, and controlled online marks a significant departure, which we believe more fully realizes Tim Berners-Lee’s original vision of the Web as a read-write medium. This book is a practical guide to several ways to access and use Fluidinfo; we hope it will help you to embrace the writable Web.

Key Concepts

Some systems gain their power through complexity, others through simplicity. Fluidinfo is among the latter. Five core concepts cover most of it:

  • Objects represent things (real or imaginary).

  • Tags attach information to objects.

  • Users use their own tags to attach data to shared objects.

  • The permissions system controls who can read and write each tag. (Objects do not have owners, so anyone can tag any object.)

  • Queries select objects by specifying properties of their tags and values. The selected objects can be read or tagged.

Objects

Fluidinfo is a collection of objects that function as shared data containers. Data is stored by attaching it to the relevant object using a tag—a kind of digital Post-it note.

Most objects are used to store data about something specific and the object is said to be about that thing. It is possible to tell what an object is about in two ways:

  • By checking the value of the special, globally unique about tag (fluiddb/about) attached to the object.[2]

  • From the context given by arbitrary tags attached to the object. The about tag is the only tag that the system guarantees to be unique and immutable. If you query for objects on the basis of other tags, you may see the tags, and therefore the objects retrieved, change over time.

In some sense Fluidinfo has an object for everything. This is reminiscent of Jorge Luis Borges’s 1941 story The Library of Babel, in which he describes a library containing all the books that could be written. Unfortunately the set of all books that could be written contains far more books full of lies, nonsense, and errors than it does accurate or useful books. The Library of Babel urgently needed a way to find the valuable material.

Most Fluidinfo users are well intentioned, so the quality of information is generally rather higher than in the Library of Babel. Fluidinfo also provides a mechanism to help judge information’s trustworthiness by starting every tag with its owner’s name. In this way, the system facilitates the establishment of networks of trust.

Tags

Tags attach information to objects.

Fluidinfo users can add data to any object by tagging it. A tag has a name, and its mere presence on an object can carry meaning: Alice might tag books with an alice/has-read tag to indicate that she has read them, while Bert might tag things he wishes to acquire with a bert/wants tag. Even the about tag is owned by a user—the Fluidinfo superuser, fluiddb.[3] This is why the full name for the about tag is fluiddb/about.

Fluidinfo tags can also store data in the form of a value, which can be a piece of text, a number, an image, or any other digital information. For example, if Alice likes the book Alice in Wonderland, she might add tags with values such as alice/rating=10 or alice/comment="I ♥ Lewis Carroll", as well as the valueless alice/has-read, to Fluidinfo’s Alice in Wonderland object (Figure 1-1).

The Alice in Wonderland object, with Alice’s tags
Figure 1-1. The Alice in Wonderland object, with Alice’s tags

Tags whose name starts with alice are owned and controlled by Alice (username alice). By default, only she can tag objects using her tags, while anyone can read them, but Alice can choose who can read and write her tags on a tag-by-tag basis. Users can assess the trustworthiness of information by examining its provenance and who controls it, which is visible in every tag’s name.

The alice at the start of all Alice’s tags is an example of a namespace, which groups tags together in much the same way that a folder groups files together in a file system. Users can create further sub-namespaces if they wish.

Users

Anyone can read public data from Fluidinfo without even logging in, but in order to write data you need to authenticate with a valid username and password. Since all Fluidinfo tags are identifiable as being owned by a named user, the system is ideally suited to storing opinions and other personalized data. Because Fluidinfo users can also be applications or organizations (usually using their website’s address as a username), the system can also comfortably accommodate application data and institutional data. The permissions system can be used to enable groups of users, or even the entire user base, to have shared write access to tags, allowing data to be formed collaboratively, and in the most extreme cases, in a wiki-like manner.

Permissions

Permissions control who can read and write each tag. They do not apply to objects, leaving Fluidinfo openly writable; in other words, any user can tag any object.

Fluidinfo permissions are expressed in two parts: a policy that is either open or closed and a list of users who are exceptions to the policy.

For example, the write permission on the alice/rating tag has a closed policy with one exception—alice herself. As a result, only Alice can tag objects with alice/rating, and only she can remove such tags. In contrast, the read permission for the same tag has an open policy with no exceptions, which means that anyone can read Alice’s ratings.

Tag permissions can be set programmatically, or using tools such as Fish (Chapter 2) or libraries (see Chapters 4, 5, and 6).

Queries

Fluidinfo has a simple query language that makes it easy to find and retrieve information. Many queries look very similar to their description in “natural” English.

Here are some examples of some of the more common types of queries used within Fluidinfo.

Tag presence

We can select all the objects that have a particular tag with a query like this:

has alice/rating

meaning

all the objects that have an alice/rating tag attached to them.

Comparisons

We can specify numeric values with ordinary equality and inequality tests:

alice/rating >= 7

meaning

all the objects that have an alice/rating of seven or more.

Text search

We can specify words that tag values should contain using the matches operator:

alice/comment matches "fantastic"

meaning

all the objects that have an alice/comment containing the word “fantastic.”

Multiple conditions

We can combine any of the previous queries using and, or, and except, adding parentheses as necessary:

(alice/rating >=7 or alice/comment matches "fantastic") except has alice/owns

meaning

all the objects that have an alice/rating of seven or more or an alice/comment that contains “fantastic,” other than those with an alice/owns tag.

Users can use any tags they can read in their queries.

A single Fluidinfo query can refer to tags from multiple users. For example, the following query matches all the objects for books that Alice rates highly (6 or more) but that Bert hasn’t read:

(has alice/books/title and alice/rating >=6) except has bert/has-read

(The first clause, has alice/books/title is used here to narrow the set of objects to books Alice knows about, as opposed to all the other things she might rate.)

Organizational Metaphor

As ever more of the world is digitized, the task of managing our data becomes increasingly complex. The overlapping challenges we face include:

  • Finding a particular piece of information within our own data

  • Keeping data in sync across multiple devices and the Web

  • Extracting or integrating data we have stored using different applications and services that may offer little or no support for such export

  • Organizing and preserving our data

  • Annotating both our own data and information we may not own but can access

  • Giving other people and applications appropriate access to our data without losing control of that data

There are many partial solutions to these problems, most of which reduce complexity for the user in exchange for migrating all data to a single place, sometimes with some support for satellite copies. This usually comes at the price of reduced control and flexibility. There are lots of online services that will manage many classes of data for you, and offer good import mechanisms, but once you have exported your digital life to one of these, and perhaps used the tools to annotate and organize the data better, you may find it difficult to reclaim your data.

Fluidinfo grew out of a different vision of how data should be stored and managed. It is not a panacea—the issues involved are probably too complex for any complete technical “solution”—but it does represent a coherent reimagining of the information landscape that speaks to each of these issues with a distinctive voice.

Like Wikipedia for Structured Data

An obvious reference point for Fluidinfo is Wikipedia. In a sense, both Wikipedia and Fluidinfo have a (potential) place for information on any topic. In Wikipedia’s case, the repository for a topic is a Wikipedia article, identified by a URL fragment; in the case of Fluidinfo, the repository is an object, and the about tag, fluiddb/about, acts as the identifier.

The two most fundamental divergences between Wikipedia and Fluidinfo concern structure and point of view.

Structure

A Wikipedia page is largely unstructured in the technical sense that the information it contains is mostly free-form text, often augmented with images, tables, and other elements designed to be read by a human being. In contrast, the Fluidinfo object is structured, typically consisting of a large number of named pieces of information, each having a type. Fluidinfo’s query language and API are also relevant differences, but are natural given the more structured nature of Fluidinfo data.

Point of view

An explicitly articulated goal of Wikipedia is to achieve a “neutral point of view,” which is developed through processes of cooperation, collaboration, negotiation, mediation, and (in some cases) edit wars, moderation, and locking. Although a history of changes is maintained, the live Wikipedia page on a topic presents a single version of the truth.

In contrast, all data in Fluidinfo is clearly associated with an owner and different users can all attach their own data to the same shared item. Fluidinfo is very suitable for opinions and other personal data, in a way that Wikipedia is certainly not. Fluidinfo’s permissions system provides extra flexibility around data ownership and control.

Because of this key difference, there are no edit wars in Fluidinfo.

Like Delicious for Bookmarking Anything

Another reference point for Fluidinfo comes from social bookmarking sites such as Delicious and Pinboard. These sites provide online storage for bookmarks to web pages, and allow users to organize their bookmarks by attaching any number of simple tags to them. In this context, a tag is just a word, and users normally attach a number of tags to each URL bookmarked to make finding pages on particular subjects easy.

The sense in which these sites are social is that, by default, a user’s bookmarks are publicly visible. An interesting consequence of this is that searching for pages that many users have tagged with particular terms tends to produce highly relevant results for that term; the degree of coherence of the emergent taxonomic structure (sometimes known as a tagsonomy) is remarkable.

Fluidinfo can be viewed as a social bookmarking site in which the bookmarks need not point to web pages but can represent anything (for example, books, songs, films, locations, stocks, Twitter users, people, cars, buildings, taxis, words, concepts, equations, truths, falsehoods, colors, or similes). Fluidinfo further differentiates itself by allowing its tags to have typed values,[4] by providing a query mechanism, by allowing more sophisticated and finer-grained permissions, and by providing a correspondingly more powerful API.

The About Tag

As we have seen, where Wikipedia is a collection of collaboratively produced encyclopedia articles, and Delicious is a collection of tagged and annotated URLs, Fluidinfo is a collection of shared objects, each of which represents something.

When an object is created in Fluidinfo, the creator can specify any text as the value of the special tag fluiddb/about, which is normally referred to simply as the about tag. If an object already exists with the nominated about tag, rather than creating a new one, the system reuses the existing object; if the about tag doesn’t exist, a new object is created. Once created, objects are never destroyed and about tags are never changed. As a result, about tags act as permanent, unique identifiers for objects, and they are the primary way of specifying objects directly.

Fluidinfo itself attaches no significance to the value of the about tag, so the choice of which object to use to store data is left to users and is a matter of convention. In simple cases, such as URLs, the choice is simple and natural—people invariably use the full URL directly as the about tag (for example, fluiddb/about="http://fluidinfo.com"), though there is slight ambiguity around whether to include a trailing slash (most people do not). In more complex cases, it is important to adopt sensible conventions if you want your data to end up on the same objects as other people’s. Such conventions, and tools for helping to construct the conventional about tags, are discussed in more detail in Chapter 9.

Signing Up for a Fluidinfo Account

If you want to be able to write data in Fluidinfo, you will need to sign up for an account by visiting Fluidinfo’s sign-up page. There are two ways to do this. If you have a Twitter account and would like to use the same username in Fluidinfo, just click the “Sign in with Twitter” button and authorize Fluidinfo on Twitter’s site; a new Fluidinfo account will be created for you with your Twitter username (if available). If you don’t have a Twitter account, or would prefer not to link your accounts, simply supply your name, a unique username (which will identify all of your tags), and a working email address.[5]

Once you’ve agreed to the terms of use and proved yourself to be human by answering the CAPTCHA[6] Fluidinfo will email you a verification link.

After you have clicked the verification link, your account will be activated.

The next few chapters introduce three different ways of interacting with Fluidinfo. In Chapter 2, we introduce the Fluidinfo Shell, Fish, which provides a mechanism for sending commands directly to Fluidinfo. Then, in Chapter 3, we survey various web applications that use Fluidinfo to store their data. Finally, we show how Fluidinfo can be used programmatically, first from Python (Chapters 4 and 5), then with JavaScript (Chapter 6), and finally directly using the HTTP API (Chapter 7).

Note

As this book goes to press, Fluidinfo (the company) is just releasing an experimental new interface to Fluidinfo (the storage system) at http://fluidinfo.com. This experimental interface is not described in this book at present, because it is in a constant state of flux, often receiving more than one update in a day. We plan to add a chapter describing it when the interface stabilizes.



[2] The selection and meaning of about tags are determined by emerging conventions discussed in Chapter 9.

[3] Fluidinfo was originally called FluidDB, and the name of the superuser has not been changed.

[4] The note that can be attached to a bookmark in Delicious and Pinboard can be seen as a single example of a special tag with a value in these systems.

[5] You can have more than one account associated with a single email address.

[6] A type of challenge/response test used to ensure that a response is generated by a person.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required