Requirements
1. Threshold of 1 million qps for 1 hour
2. For example, https://github.com/changpil/pywork
Limiting for framework API when users access to https://github.com/changpil/gowork
3. Public API, no username and password required. (No token needed)
Key-Value Database
Used for storing key-value pairs in a distributed manner.
Amazon DynamoDB (DAX caching for DynamoDB), Redis(Redis provides a different range of persistence options), Aerospike
Common Uses:
some of the most common use cases of key-value databases are to record sessions in applications that require logins.
a shopping cart where e-commerce websites can record data pertaining to individual shopping sessions.
storing clickingstream data
Web Cache for Radis and Aerospike
Pros:
Simplicity, Speed, Scalability
Cons:
Simplicity, No query Language
Document database
Used to storing JSON documents
MongoDB, CouchDB, CosmosDB
Common Uses:
User profile
Book Database
content management (Catalogs)
Patient Data
Pros: Schemaless
Cons: Consistency, Atomicity
it would be possible to search for books from a non-existent author.
a change involving two collections will require you to run two separate queries (per collection).
Graph Database
A graph is composed of two elements: a node and a relationship. Emphasized connections between data elements, storing related “nodes” in graphs to accelerate querying
Amazon Neptune, Neo4j, Tiger DB
common Uses:
Real-time recommendation engines
Fraud Detection
Identity and access management (IAM)
Ledger Database
A ledger database is log-centric. Store using an append only record journal. Here, the term “log” means a storage abstraction that is “an append-only, totally-ordered sequence of records ordered by time”.
Amazon QLDB
Common Uses:
Store financial transactions
Maintain claims history
Centralize digital records
Distributed Data processing System
Apache Hadoop, Spark
DBMS (Relational Database-SQL)
With ACID for consistency with fixed schema
mysql, postgresql, Oracle
Common Uses (write and update heavy):
Relational databases are better to use with payment transaction records
A columnar database stores data in columns rather than the rows used by traditional databases. Columnar databases are designed to read data more efficiently and return queries with greater speed. it store data across tables that can have very large number of columns. If data has large amount of scale such as Google Search “Pyhton”:[23, 45, 67, 89, 123, 234,789,2344,3345,5543,6545,7545,7955,8099,12344,14566,14767,14890,14891,14892,… ]. This is a good candidate for columnar database. It works well with Scatter and Gather strategy for faster throghput.
Cassandra (noSQl, Columar), Apache HBase, Amazon Redshift, PostgreSQL, MariaDB
Common Uses (read heavy):
internet search
other large-scale web applications
Pros:
A columnar database is preferred for analytical applications because it allows for fast retrieval of columns of data. Columnar databases are designed for data warehousing and big data processing because they scale using distributed clusters of low-cost hardware to increase throughput.
Queries that involve only a few columns.
Cons:
Incremental data loading
Queries against only a few rows
How to Choose
hash(term)%r gives random server for term. If I change the hash((ord(term[0])-ord(‘a’))//3), starting a, b, c terms storing in server 1, def storing server 2. Totally, you needs to have 26//3 +1 = 9 servers. The benefit is this is globally sorted from server 1 to server 9 by terms.