杏吧原创

It鈥檚 all being logged? (Image: Chris Ratcliffe/Bloomberg via Getty Images)

I TWEET. Quite a bit. A quick scroll through my feed will show messages that run the gamut: a link to an article about Wi-Fi on Mount Fuji (see 鈥�Web posts help to spot drugs鈥� side effects鈥�), a joke with my colleagues about the Victorian origins of the word 鈥渟ext鈥�, a GIF of an ostrich chasing a giraffe.

All those thousands of tweets, and billions more by others, will be saved for all time in the hallowed halls of the Library of Congress in Washington DC. One of the largest libraries in the world, it promised in 2010 that it would organise and preserve every tweet ever sent.

Five years later, the archive still isn鈥檛 available. What鈥檚 taking so long? And, some might wonder, why even bother in the first place?

Twitter is an important piece of our cultural legacy, says , a library technologist in Vermont. 鈥淎 library doesn鈥檛 just tell the one story you read in history books, but all of the interwoven multicultural threads and stories that are going on,鈥� she says.

Twitter helps to tell the story of some of the biggest political and social movements of the last decade, including the Arab Spring, Occupy Wall Street, and ISIS. 鈥淚t keeps track of who we are and what we鈥檝e been,鈥� West says.

Some tweets that seem boring today could be much more interesting in the future. Someone currently tweeting in obscurity could one day become prime minister. Even anonymous mundane musings about the weather or what someone ate for lunch might be valuable to researchers. Aggregated, the messages capture how people use language and communicate with one another.

鈥淭weets that seem boring today will be interesting in the future if that person becomes prime minister鈥�

But saving all of Twitter poses problems of daunting magnitude. Michael Zimmer, a privacy and internet ethics specialist at the University of Wisconsin-Milwaukee, has tracked the library鈥檚 progress, and outlined some of its greatest obstacles (First Monday, ).

One problem, of course, is the sheer number of tweets. When the library started the project, there was already a four-year backlog of of about 21 billion tweets. Now, half a billion tweets are sent daily.

Another problem is that a tweet isn鈥檛 just a string of 140 characters. Each comes with a packet of metadata, including when and where the tweet was sent, who sent it, and how many people marked it as a favourite, shared it or responded. There are over 100 information fields in all.

For help managing all this data, the Library of Congress turned to Gnip, a social media aggregation company in Boulder, Colorado, that has since been bought by Twitter. It slices the Twitter stream into hour-long segments for the library to download. The library checks each chunk for corruption, notes down some statistics and then, as a security measure, saves copies of the file in two different locations.

But just storing tweets isn鈥檛 enough. In the library鈥檚 latest update on the project, it said that a single search of that initial batch sent before 2010 could take 24 hours 鈥� far too slow to be usable.

Figuring out how to make the search function work needs to be a priority, says West. 鈥淧eople don鈥檛 just want to know that the data is being handled. They want access to the content, tools to interact with it.鈥�

Meanwhile, many people don鈥檛 realise that their tweets, even public ones, are being recorded, says Zimmer, who is concerned that the library has not announced plans to give people control over their own archives. He thinks the site should consider giving people an opt-out, or even the ability to delete tweets that they regret.

鈥淔or a lot of people, it was never in their frame of understanding that their tweet would be kept on a historical record for all eternity,鈥� he says.

Leader: 鈥�Tweet everythings: why we should keep them all鈥�

This article appeared in print under the headline 鈥淗istory in the tweeting鈥�