About OpenFreedom.info robot
The OpenFreedom.info robot/crawler is a server that indexes news feeds and sites that relate to Article 19 of the Universal Decleration of Human Rights. In short: the freedom of collecting and sharing information. All these articles are analysed and further processed to get an idea of what is happening on our planet and also provide a database for future reference.
Origin
Any traffic claiming to originate from OpenFreedom.info will ALWAYS come from either this IP-address:
109.72.93.115 or
194.145.200.120, which also resolve back to this site. Any other IP-addresses can be considered false unless it is mentioned on this page! In such case, please contact OF.i using the secure
Contact form. When the crawler adapts one or more new IP-addresses, they will be announced on the
Twitter status reports.
Bandwidth
The crawler attempts to limit the amount of traffic from your site as much as possible. You can help reduce this by enabling GZIP-compression on your site. If you want to change the rate at which the crawler visits your site or feed, please use the
Contact Form.
Access
The only area(s) on your site that are being downloaded by the crawler are public
news indexes and
articles that are linked from there. However, in case you really want certain content from your site to be removed from OpenFreedom.info, please use the
Contact Form. Your request will be reviewed as soon as possible. Removal of content only happens in special occasions. For example, when your article contains your personal identifying information, login details, and the like. Other kinds of requests are considered
censoring and therefor your site will get a '
Censored' flag, meaning your content is unreliable.
Files like
robots.txt are ignored as they are often invalid or even prevent access to public news areas.
Publishing Dates
Currently only RSS, ATOM and RDF feeds are supported. When a site does not supply a syndication feed, custom code is written to retrieve the articles and their titles, links and publishing date. In some cases the date at which your article was published can not be properly detected. For this the system relies on either the RSS <pubDate> field, <dc:date> field, <created>, or the <published> field. Please, make sure your feed supports one of these methods for proper indexing. Sitemaps (sitemap.xml) files are not yet supported.
In other cases the provided publishing date in the feed is too far ahead in the future. Those dates are reset to the exact moment the new articles are downloaded. Currently, only timestamps based on the Gregorian calendar are supported. Other calendars such as Hijri are for now displayed incorrectly. This will be resolved later.