Structured database "Panama archive" laid out in the open access

The biggest leak of documents in the history of the Internet: 2, 6 terabytes




The International Consortium of Investigative Journalists (ICIJ) laid out in free access "Panama Archive": the largest database of offshore companies from computers Panamanian law firm Mossack Fonseca unknown persons.

Anonymous (John Doe) gave reporters the German newspaper Süeddeustche Zeitung 2, 6 terabytes of files - spreadsheets, letters, text, PDF, TIFF and other formats, including ancient and has unused. Understanding the scale of the work, they were asked to ICJI organize an international joint project.

Millions graphics drove through the Tesseract OCR program 40 time servers in the Amazon cloud. For text indexing using Apache Solr, to process documents in different formats -. Apache Tika

During the year, the archive studied 370 reporters from 80 countries. To facilitate their work journalists, developers of the ICIJ hooked GUI library program from Project Blacklight. To display the information in graphical form and show relationships between objects, it was necessary to use a proprietary program Linkurious, but by the contents of Talend tool relational SQL database Mossack Fonseca transferred to Neo4j format.



Now the result of developer productivity is invited to evaluate all comers through the Internet.

ICIJ database in a structured form is available at:
https://www.occrp.org/en/panamapapers/database.html

You can download a copy on your computer (35, 7 MB in archive).

• data-csv.zip
• torrent file
   
  
  The database contains information on almost 214,000 offshore companies in 21 offshore jurisdictions.
  
  Interactive map of offshore companies customers
  40,911,357
  
  
  
  At the base of 11,516 companies that belong to the 6285 Russian citizens. Among them - the friends and relatives of senior officials. Such large-scale leak of documents may lead to a number of high-profile resignations and criminal cases, although offshore companies often operate in a gray area legally, without violating the law.
  
  Published only part of the information from the 11, 5 million files that were obtained from computers of law firm Mossack Fonseca, one of the world's largest generators of one-day firms.
  
  The International Consortium of Investigative Journalists do not publish all the information available, there is no original documents or a large database with personal information, bank accounts are not listed companies, the contents of e-mails and financial transactions of companies. This is done so as not to light the personal data of many people who are not involved in financial crimes.
  
  Disclosed only names, their jurisdiction, postal addresses and the names of the leaders of the offshore companies. The data cover the period from 1977 to 2015.
  
  Base "Panama archive" data is published under a free license Creative Commons Attribution-ShareAlike. The International Consortium of Investigative Journalists invites the entire community to focus efforts on the study and classification of the information published.
  
  UPD. The first finding Geektimes community
  
  
  UPD2. The second discovery Geektimes community
  
  
  
  
  Source: geektimes.ru/post/275558/