Common Crawl Technology Stack

Cloud based storage systems by AWS.

Amazon Web Services (AWS) is a comprehensive, evolving cloud computing platform provided by Amazon that includes a mixture of infrastructure as a service (IaaS), platform as a service (PaaS) and packaged software as a service (SaaS) offerings.

EMC Atmos is a cloud storage services platform developed by EMC Corporation. Atmos can be deployed as either a hardware appliance or as software in a virtual environment. The Atmos technology uses an object storage architecture designed to manage petabytes of information and billions of objects across multiple geographic locations as a single system.

EMC Avamar: Fast, efficient data backup and recovery through a complete software and hardware solution. Equipped with integrated variable-length deduplication technology, Avamar facilitates fast, daily full backups for virtual environments, remote offices, enterprise applications, network-attached storage (NAS) servers, and desktops/laptops.

EMC Elastic Cloud Storage (ECS), formerly Project Nile, is an object storage software product marketed by EMC Corporation. ECS was designed to adhere to several tenets of object storage, including scalability, data resiliency and to take advantage of existing or new commodity server hardware in order to manage costs. It is marketed as software-defined storage.

EMC SourceOne Archiving Family: Advanced archiving software that improves user productivity, providing seamless access to archived email, files, and Microsoft SharePoint content. Proactive information management helps with litigation readiness and a centralized archive accelerates high-volume discovery searches and enables secure legal holds.

Linux is a family of open source Unix-like operating systems based on the Linux kernel.

Bootstrap is a free and open-source front-end framework for designing websites and web applications. It contains HTML and CSS-based design templates for typography, forms, buttons, navigation and other interface components, as well as optional JavaScript extensions.

DataTables is a table enhancing plug-in for the jQuery Javascript library, adding sorting, paging and filtering abilities to plain HTML tables with minimal effort. If client prefer to use a package manager such as NPM or Bower, distribution repositories are available with software.

Hypertext Markup Language (HTML) is the standard markup language for documents designed to be displayed in a web browser. It can be assisted by technologies such as Cascading Style Sheets (CSS) and scripting languages such as JavaScript.

JavaScript is the programming language of HTML and the Web.

Nginx is a web server which can also be used as a reverse proxy, load balancer, mail proxy and HTTP cache.

PHP: Hypertext Preprocessor is a server-side scripting language designed for Web development, but also used as a general-purpose programming language. PHP code may be embedded into HTML code, or it can be used in combination with various web template systems, web content management systems, and web frameworks.

pip python is a package-management system written in Python used to install and manage software packages.

Bottlenose helps businesses to understand their influence- anticipate and identify trends with real-time trend visualization, tracking and analysis tools. 

Klout helps people who want to be great at social media. Join today to start sharing original content and measuring your online impact.

OnCrawl is a data-driven SEO crawler and log analyzer for enterprise SEO audits.

Cloudflare mitigates threats from website scraping to application level attacks e.g. SQL Injection and DDOS Protection without any additional hardware.

Cloudflare Content Delivery Network (CDN) is a geographically distributed group of servers that ensure fast delivery of Internet content, including HTML pages, JavaScript files, stylesheets, and images.

GitHub is a place to share code with friends, co-workers, classmates, and complete strangers, helping individuals and teams to write faster, better code.

Medium is a blog-publishing platform.

YouTube is a video sharing service where users can watch, like, share, comment and upload their own videos. The video service can be accessed on PCs, laptops, tablets and via mobile phones.

Cloudflare protects and accelerates any Internet application online without adding hardware, installing software, or changing a line of code. Internet properties powered by Cloudflare have all traffic routed through its intelligent global network, which gets smarter with each new site added.

Gmail is a free, advertising-supported email service offered as part of Google's G Suite, with 15GB of storage, color-coded inbox filters and unsend button.

Google Analytics allows users to measure sales & conversions, plus gain fresh insights into how visitors use sites, & how they arrived on site.

Kaggle offers innovative business results and solutions to companies.

Mobilize is an all-in-one membership management platform.

PayPal is a global eCommerce platform for buyers & sellers. PayPal allows payments & money transfers to be made securely via email, phone, text message or Skype.